search

dwilliamhouston / Calishot-2.0

3 stars 3 forks source link

Make the Shodan Scan Multi-Threaded #8

Open dwilliamhouston opened 10 months ago

dwilliamhouston commented 10 months ago

Enhance the scan so its when it scans Shodan each country is its own thread. This should make the scan a lot quicker.

Krazybug commented 10 months ago

Although you can easily set this up with gevent as I did for the indexation, I'm not sure it's a good idea. There API is rate limited and it wont take too much time to get all their result but i can't tell you as i don't have a premium account

dwilliamhouston commented 10 months ago

I have a premium account and I the longest time is the Shodan search and then also the book search of each server. If we could make each of those tasks a separate thread I think it will be more efficient and also stop causing blockers if one book server is slow.

Krazybug commented 10 months ago

The indexation of each server and the check is already asynchronous. But i say that you call the seq version of index_site which is only here for debugging purpose. Also keep in mind that the calibre server are often selfhosted and slow and I voluntarily make small request of 1000 books to not load them too much. But I launch around 100 requests in parallel in a pool against different servers.

Nice that you have a premium account but how mech time does it take to get the complete list with 700 hundred of results ?

As I proposed you it could be nice if we could discuss about some aspect by chat as I've sent an invite to you.

dwilliamhouston commented 10 months ago

Not going to do anything until we discuss. Have family here right now so its a little difficult to talk during the day.

On Fri, Dec 22, 2023 at 1:44 PM Krazy Bug @.***> wrote:

The indexation of each server and the check is already asynchronous. But i say that you call the seq version of index_site which is only here for debugging purpose. Also keep in mind that the calibre server are often selfhosted and slow and I voluntarily make small request of 1000 books to not load them too much. But I launch around 100 requests in parallel in a pool against different servers.

Nice that you have a premium account but how mech time does it take to get the complete list with 700 hundred of results ?

As I proposed you it could be nice if we could discuss about some aspect by chat as I've sent an invite to you.

— Reply to this email directly, view it on GitHub https://github.com/dwilliamhouston/ShodanTest/issues/8#issuecomment-1868021397, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7LTQQ7MQJSIPFNFZ7USQDYKXPJLAVCNFSM6AAAAABA7DXW22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGAZDCMZZG4 . You are receiving this because you authored the thread.Message ID: @.***>

Krazybug commented 10 months ago

No worry. It was just to avoid long discussion in a ticket and define a way to cooperate before ... Enjoy your time with your family And I guess that we're not in the same time zone

dwilliamhouston commented 10 months ago

I live in Houston, TX and am in CST which is -6 hours from GMT/London.

On Fri, Dec 22, 2023 at 2:00 PM Krazy Bug @.***> wrote:

No worry. It was just to avoid long discussion in a ticket and define a way to cooperate before ... Enjoy your time with your family And I guess that we're not in the same time zone

— Reply to this email directly, view it on GitHub https://github.com/dwilliamhouston/ShodanTest/issues/8#issuecomment-1868031241, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7LTQUQKWJJYEQ4YAUDUG3YKXRG5AVCNFSM6AAAAABA7DXW22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGAZTCMRUGE . You are receiving this because you authored the thread.Message ID: @.***>