search

achillean / shodan-python

The official Python library for Shodan
https://developer.shodan.io
Other
2.47k stars 552 forks source link

Loading data is incomplete #186

Closed studentAlanMorgan closed 1 year ago

studentAlanMorgan commented 1 year ago

when downloading, it shows one result, it says that 100% downloaded, but saves a completely different result. download code works for python. script code:

!/usr/bin/env python

import shodan import requests from subprocess import call

from requests.packages.urllib3.exceptions import InsecureRequestWarning requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

def download(PATH, count, SEARCH_FOR): s = f'{PATH} download --limit {count} shodan_data.json.gz {SEARCH_FOR} ' print(s) call(s, shell = True)

Configuration

PATH = "shodan" API_KEY = "SHODAN API" SEARCH_FOR = 'DORK'

session = requests.Session()

try:

Setup the api

api = shodan.Shodan(API_KEY)

    # Perform the search

count = api.count(SEARCH_FOR)['total'] print(count) download('shodan', count, SEARCH_FOR)

except KeyboardInterrupt: print ("Ctrl-c pressed ...")

shodan

rmhowe425 commented 1 year ago

@studentAlanMorgan I believe that this is a known issue and is being caused by an ongoing data migration.

achillean commented 1 year ago

The answer from @rmhowe425 is correct. The problem has to do with banners not getting loaded fast enough from the backend database so the API skips them which can lead to early exits of the download command. The issue resolves itself once the database is fully operational again and should actually already be resolved at this point.