Open NewMember12345 opened 3 years ago
I never did such big scraping session. But i can image 2 main reasons:
To rule out the first one can you do the following checks:
In the terminal you start type the command line in, you should see something like: Heap: X GB Free mem: Y GB Can you give those values ?
Also can you try adding -Xms2048m -Xmx4096m to your command line. like java -Xms2048m -Xmx4096m -jar JAVMovieScraper.jar.
If this does not help, i will profile application's memory usage when i got some time
@Wizell Unforunately I use windows and do not code. So i have no way of knowing how to do those you mentioned.
How do you start JavMovieScraper jar file ?
I have Java runtime environment 8 installed. On Windows, I just double click on the .jar. It default to launch with JRE
If i try to run it cmd in windows, I get this big box error:
D:\torrents\later\tmp>java -Xms2048m -Xmx4096m -jar JAVMovieScraper-v0.3.7.jar Java(TM) SE Runtime Environment 1.8.0_261-b12(Oracle Corporation) -- 1.8(Oracle Corporation) Heap: 3 GB Free mem: 1 GB Creating settings.xml since it was not found... Initializing default jav preferences Aug 23, 2020 5:08:35 PM moviescraper.doctord.scraper.DitzyHeadlessBrowser
INFO: Build browser with U: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0 java.lang.NullPointerException at java.io.File. (Unknown Source) at moviescraper.doctord.scraper.DitzyHeadlessBrowser.configure(DitzyHeadlessBrowser.java:35) at moviescraper.doctord.controller.siteparsingprofile.specific.JavLibraryParsingProfile.browserConfigure(JavLibraryParsingProfile.java:102) at moviescraper.doctord.controller.siteparsingprofile.specific.JavLibraryParsingProfile. (JavLibraryParsingProfile.java:93) at moviescraper.doctord.controller.amalgamation.AllAmalgamationOrderingPreferences.initializeJAVCensoredGroupDefaultPreferences(AllAmalgamationOrderingPreferences.java:129) at moviescraper.doctord.controller.amalgamation.AllAmalgamationOrderingPreferences.initializeDefaultPreferences(AllAmalgamationOrderingPreferences.java:103) at moviescraper.doctord.controller.amalgamation.AllAmalgamationOrderingPreferences.initializeValuesFromPreferenceFile(AllAmalgamationOrderingPreferences.java:69) at moviescraper.doctord.view.GUIMain.reinitializeAmalgamationPreferencesFromFile(GUIMain.java:157) at moviescraper.doctord.view.GUIMain.initialize(GUIMain.java:169) at moviescraper.doctord.view.GUIMain. (GUIMain.java:142) at moviescraper.doctord.view.GUIMain$1.run(GUIMain.java:126) at java.awt.event.InvocationEvent.dispatch(Unknown Source) at java.awt.EventQueue.dispatchEventImpl(Unknown Source) at java.awt.EventQueue.access$500(Unknown Source) at java.awt.EventQueue$3.run(Unknown Source) at java.awt.EventQueue$3.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source) at java.awt.EventQueue.dispatchEvent(Unknown Source) at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source) at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source) at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source) at java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.awt.EventDispatchThread.run(Unknown Source)
You got the command line right, the error is a mistake in the settings of the application. I fixed it few hours ago. Please retry with the next release that should happen during this week.
Thanks @Wizell
I was able to run the command on a VM that runs my primary scraping. I could only max it to 1gb heap and free mem. I'll try to see if it can do more than 50 titles. I typically scrape 100 titles every 3-4 days.
@Wizell Do you have a quick idea how to compile your files from github into a working .jar? Maybe for future users because we see code fixes but unable to use them until a release.
In the terminal you start type the command line in, you should see something like: Heap: X GB Free mem: Y GB Can you give those values ?
java -jar JAVMovieScraper-v0.3.7.jar
Java(TM) SE Runtime Environment 1.8.0_231-b11(Oracle Corporation) -- 1.8(Oracle Corporation)
Heap: 247 MB
Free mem: 13 MB
Aug 23, 2020 5:38:06 PM moviescraper.doctord.scraper.DitzyHeadlessBrowser
Issue: When I use JAVMovieScraper to scrape for anything more than 50 titles, sometimes less, i get an error with java heap space out and it wont write any more data like poster, fanart, nfo.
This occurs as well if I scrape for 30-ish titles, write them to disk, then select the next 30. Seems like this java temp space can't handle that many jpgs. I often just close out the jar and open up another instance.
This did not occur in really old iterations of this project. But it has happened since v3.6 or so. I never thought to bring it up but I'd like to see if zuko or wizell can shed some light?