Re-run Flyscrape with cache: file option from the previous successful scraping
Abort (power failure, system failure)
Now the cache isn't closed properly (amazon.cache, amazon.cache-shm, amazon.cache-wal)
Re-run Flyscrape with unclosed cache file
CPU peaks at 100% per core. htop -d 0 reveals a constant 395% CPU usage (meaning 95-100% per core) and 1.23 GB memory usage. Memory ever exceeds 1.5 GB usage.
Worth a look for better CPU management in case of open cache file?
A Few Moments Later: I don't think the CPU stuff above is a problem. I will do a test again and update the info. The memory info below is correct, so that's worth a look.
Memory
When Flyscrape is re-running with a cache: file from a previous successful scraping session with a closed cache file, the CPU is 100% for 1 core, and others are switching. Memory usage never exceeds 1.5 GB. Tested with cache files from less than 1 GB to 10 GB.
Upside: Flyscrape can run on older hardware where memory is limited or less memory required to buy.
Downside: newer hardware resource remains unused.
CPU
Steps taken:
cache: fileoptioncache: fileoption from the previous successful scrapinghtop -d 0reveals a constant 395% CPU usage (meaning 95-100% per core) and 1.23 GB memory usage. Memory ever exceeds 1.5 GB usage.Worth a look for better CPU management in case of open cache file?
A Few Moments Later: I don't think the CPU stuff above is a problem. I will do a test again and update the info. The memory info below is correct, so that's worth a look.
Memory
When Flyscrape is re-running with a
cache: filefrom a previous successful scraping session with a closed cache file, the CPU is 100% for 1 core, and others are switching. Memory usage never exceeds 1.5 GB. Tested with cache files from less than 1 GB to 10 GB.Upside: Flyscrape can run on older hardware where memory is limited or less memory required to buy. Downside: newer hardware resource remains unused.