hartator / wayback-machine-downloader

Download an entire website from the Wayback Machine.
Other
5.36k stars 713 forks source link

error 503 stuff #287

Open BelleButCode opened 7 months ago

BelleButCode commented 7 months ago

semi new to this so not sure if i set params right but i believe i complied to the robots list alongside rate limiting (with this fork https://github.com/ShiftaDeband/wayback-machine-downloader

wayback_machine_downloader https://web.archive.org/web/20160918235500/http://wrongsideoftheart.com/gallery/a/ --only "/.(jpg)$/i" --exclude /control/ --exclude /report/ --exclude /thumb/ -l

Getting snapshot pages./usr/lib/ruby/3.0.0/open-uri.rb:364:in open_http': 503 Service Unavailable (OpenURI::HTTPError) from /usr/lib/ruby/3.0.0/open-uri.rb:741:inbuffer_open' from /usr/lib/ruby/3.0.0/open-uri.rb:212:in block in open_loop' from /usr/lib/ruby/3.0.0/open-uri.rb:210:incatch' from /usr/lib/ruby/3.0.0/open-uri.rb:210:in open_loop' from /usr/lib/ruby/3.0.0/open-uri.rb:151:inopen_uri' from /usr/lib/ruby/3.0.0/open-uri.rb:721:in open' from /var/lib/gems/3.0.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader/archive_api.rb:13:inget_raw_list_from_api' from /var/lib/gems/3.0.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:92:in block in get_all_snapshots_to_consider' from /var/lib/gems/3.0.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:91:intimes' from /var/lib/gems/3.0.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:91:in get_all_snapshots_to_consider' from /var/lib/gems/3.0.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:105:inget_file_list_curated' from /var/lib/gems/3.0.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:164:in get_file_list_by_timestamp' from /var/lib/gems/3.0.0/gems/wayback_machine_downloader-2.3.1/lib/wayback_machine_downloader.rb:177:inlist_files' from /var/lib/gems/3.0.0/gems/wayback_machine_downloader-2.3.1/bin/wayback_machine_downloader:70:in <top (required)>' from /usr/local/bin/wayback_machine_downloader:25:inload' from /usr/local/bin/wayback_machine_downloader:25:in `

'