search

lavv17 / lftp

sophisticated command line file transfer program (ftp, http, sftp, fish, torrent)
http://lftp.yar.ru
GNU General Public License v3.0
1.08k stars 159 forks source link

Not an issue, a question about robots.txt #670

Closed ghost closed 1 year ago

ghost commented 1 year ago

Is there any equivalent to 'wget -e robots=off' for LFTP? LFTP is extremely quick to mirror a website it would have been good if this existed, I have not found anything in the wiki or faq.

lavv17 commented 1 year ago

AFAIK, lftp does not take robots.txt into account at all

вт, 5 июл. 2022 г. в 02:26, wuseman @.***>:

Is there any equivalent to 'wget -e robots=off' for LFTP? LFTP is extremely quick to mirror a website it would have been good if this existed, I have not found anything in the wiki or faq.

— Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/670, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHLWXB5IGMXBTT5L2MOBV3VSNXJFANCNFSM52UNCR7Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Alexander.

ghost commented 1 year ago

Alright @lavv17 thanks for your reply, I will try wget2 instead for this purpose.

Regards