search

nathell / soupscraper

dej, mam umierajoncom zupe
20 stars 3 forks source link

Exception in thread "async-thread-macro-1" #7

Open dragon99919 opened 4 years ago

dragon99919 commented 4 years ago

After successful page fetch and assets download, while generating local copy, program started to take a lot of memory at once and then it threw an exception shown below:

obraz

Looking at the assets folder, it appears that only 7320 from above 70k were saved, and no index was generated.

Logs also don't tell a thing, last entry is about downloading stuff.

nathell commented 4 years ago

@dragon99919: This is an out-of-memory error. Try aborting and then re-running with the -Xmx1G switch, as in:

java -Xmx1G -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -jar soupscraper-0.1.0.jar yoursoup
dragon99919 commented 4 years ago

Just tried, same thing happened, exactly same error with all the lines related and such...

schlingel commented 4 years ago

@dragon99919 It's about the memory. Give the process as much memory as you can.

If your machine has 8GB, try 4gb. If it has more, try more.

dragon99919 commented 4 years ago

Gave 10GB and it worked overnight! Thanks for the help! : )