search

Skallwar / suckit

Suck the InTernet
Apache License 2.0
733 stars 38 forks source link

os error 22 thread '<unnamed>' panicked when i attempt a crawl #104

Open The-Stellaris-Effect opened 3 years ago

The-Stellaris-Effect commented 3 years ago

seemed to work for a while but then was suddenly hit with this error. -c switch doesn't seem to help [ERROR] Couldn't create /run/media/path/to/website/"f.txt": Invalid argument (os error 22) thread '' panicked at 'Couldn't create /run/media/path/to/website/"f.txt": Invalid argument (os error 22)', src/logger.rs:42:9 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace thread 'main' panicked at 'called Result::unwrap() on an Err value: Any', src/scraper.rs:208:10

CohenArthur commented 3 years ago

@The-Stellaris-Effect which website was it that you tried to download?

CohenArthur commented 3 years ago

Also it would be nice to have the exact details on the error. This error means that the OS couldn't create a file, because the path input is not valid. Is the file you're trying to download really named "f.txt"?

The-Stellaris-Effect commented 3 years ago

I was attempting to crawl a site known as phys.org, it's a scientific news site essentially. also yes the file was actually called "f.txt" according to the error idk what exactly the issue was however i'm either suspecting it's an env difference of the server and my device where "" is allowed server side but not client side. or it was because i was adding to a directory a program previously wrote to it. i have been transitioning away from httrack due to issues regarding crashes and just general instability

Skallwar commented 3 years ago

Ok. Could you provide how you invoked SuckIT? If possible, tune the URL in order for us to get the error the fastest way possible.

I have been transitioning away from httrack due to issues regarding crashes and just general instability.

Suckit might not be the way to go as of now, a lot of website configurations are not handled yet and you might crash often