search

laserson / ftptree

Crawl an FTP site and visualize file size-weighted directory tree
22 stars 5 forks source link

FTP concurrent requests problem #5

Open draganpodvezanec opened 6 years ago

draganpodvezanec commented 6 years ago

Hi.

I have put

CONCURRENT_REQUESTS = 1 CONCURRENT_REQUESTS_PER_IP = 1 CONCURRENT_REQUESTS_PER_DOMAIN = 1 CONCURRENT_ITEMS = 1

in ftptree_crawler/settings.py

Anyway, soon after starting crawler, I get an error:

ConnectionLost: ('FTP connection lost', <twisted.python.failure.Failure twisted.protocols.ftp.CommandFailed: ['530 Sorry, the maximum number of clients (6) from your host are already connected.']>)

FTP server is limited to 6 concurrent connections.

Does FTPTREE obey these CONCURRENT settings while crawling?

laserson commented 6 years ago

Thanks for submitting the issue. I don't have time to maintain this, though. Feel free to submit a PR if you figure it out.