search

achillean / shodan-python

The official Python library for Shodan
https://developer.shodan.io
Other
2.48k stars 559 forks source link

Refactor shodan convert command to use Pandas #200

Closed rmhowe425 closed 3 months ago

rmhowe425 commented 10 months ago

Requesting a refactor of the underlying code for the shodan convert command so that the Pandas library is used for IO operations for the JSON, CSV, and Excel file types. Would be more than happy to submit a PR for this.

Benefits of this refactor:

Edit: Integrating pandas into the Shodan CLI could also have additional benefits as well. Pandas could be used to better (and more easily) structure and output data pulled from the Shodan API via storing the raw JSON in data frames. We could give users the opportunity to control what fields they want to see in output data by including / dropping DataFrame columns accordingly, which would minimize much of the custom looping and parsing of raw JSON data.

rmhowe425 commented 10 months ago

@achillean do you think you would be interested in a PR for this issue?

achillean commented 10 months ago

There are some challenges:

  1. The shodan package currently works across a lot of Python versions, including Python 2.7.
  2. The dependencies for Pandas would substantially increase the size of the shodan package.

The current CLI was written before a lot of the data science packages became prominent and it would be nice to have a way to efficiently filter by columns, manage schemas etc. but I don't want to break existing scripts. I don't know when I'm ready to kill Python2.7 support but we still get support tickets from people that use it.

I think it would be better to split breaking changes into click plugins or otherwise offer it as an optional component. The current CLI is already setup to allow 3rd-parties to create subcommands as outlined on: https://github.com/click-contrib/click-plugins

rmhowe425 commented 10 months ago

@achillean I would definitely be interested in creating plugins for the Shodan CLI. Looking at the repo that you linked, it wouldn't be too much of an effort.

If I were to create plugins that supported my proposed functionality, would you be willing to incorporate them into the Shodan CLI? It sounds like the answer would be "yes"? 😄