Flameish / Novel-Grabber

Novel-Grabber can download novels from pretty much any webnovel and lightnovel site.
MIT License
468 stars 62 forks source link

[BUG] Suddenly stopping the download near the end. #193

Closed PasteLord closed 3 years ago

PasteLord commented 3 years ago

Found this a few days ago, cannot thank you enough for making this program!

Issue I tried to download all of this novel but it stops roughly around chapter 520. https://foxaholic.com/novel/i-was-caught-up-in-a-hero-summoning-but-that-world-is-at-peace/

I've tried downloading small portions of chapters, which work and get completed, but every time I go all chapters it stops at around 520. It would also say it couldn't read the image at chapter 322.

Expected behaviour Completed novel chapters.

Screenshots Screenshot 2021-05-11 091435

Platform Windows 10 x64 bit Java 16

Logs [INFO]Saved chapter: 523 - First Anniversary Extra Chapter ~~ Isekai Trip for Two Part 1 ?: Amusement Park ~~ [INFO]Saved image: 12238937529540ba8a5d21156b0894f8.jpg [ERROR]Could not read image: Server returned HTTP response code: 403 for URL: https://foxaholic.com/art/2020/05/??????-790x592-1.jpg java.io.IOException: Server returned HTTP response code: 403 for URL: https://foxaholic.com/art/2020/05/??????-790x592-1.jpg at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1985) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1577) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224) at grabber.GrabberUtils.getImage(GrabberUtils.java:47) at grabber.Chapter.getImages(Chapter.java:99) at grabber.Chapter.saveChapter(Chapter.java:49) at grabber.Novel.downloadChapters(Novel.java:118) at gui.GUI.lambda$null$2(GUI.java:342) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at java.base/java.lang.Thread.run(Thread.java:831) [ERROR]Could not save image: ??????-790x592-1.jpg [INFO]Saved image: kofi3.png [INFO]Saved chapter: 524 - First Anniversary Extra Chapter ~~ Isekai Trip for Two Part 1 ?: Amusement Park ~~ [INFO]Saved image: sky-cycle-pedal-powered-rolloer-coaster-japan-3.jpg Exception in thread "pool-4-thread-1" java.lang.OutOfMemoryError: Java heap space at java.desktop/java.awt.image.DataBufferByte.(DataBufferByte.java:92) at java.desktop/java.awt.image.ComponentSampleModel.createDataBuffer(ComponentSampleModel.java:439) at java.desktop/java.awt.image.Raster.createWritableRaster(Raster.java:1005) at java.desktop/javax.imageio.ImageTypeSpecifier.createBufferedImage(ImageTypeSpecifier.java:1074) at java.desktop/javax.imageio.ImageReader.getDestination(ImageReader.java:2877) at java.desktop/com.sun.imageio.plugins.jpeg.JPEGImageReader.readInternal(JPEGImageReader.java:1154) at java.desktop/com.sun.imageio.plugins.jpeg.JPEGImageReader.read(JPEGImageReader.java:1122) at java.desktop/javax.imageio.ImageIO.read(ImageIO.java:1468) at java.desktop/javax.imageio.ImageIO.read(ImageIO.java:1363) at grabber.GrabberUtils.getImage(GrabberUtils.java:47) at grabber.Chapter.getImages(Chapter.java:99) at grabber.Chapter.saveChapter(Chapter.java:49) at grabber.Novel.downloadChapters(Novel.java:118) at gui.GUI.lambda$null$2(GUI.java:342) at gui.GUI$$Lambda$241/0x0000000800edc430.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at java.base/java.lang.Thread.run(Thread.java:831)

[INFO]Saved chapter: 321 - Valentine's Day Extra Chapter ~~ Lunamaria & Noir ~~ [INFO]Saved image: Sachertorte_DSC03027_retouched-1024x768.jpg [INFO]Saved image: kofi3.png [INFO]Saved chapter: 322 - Valentine's Day Extra Chapter ~~ Alice & Fate ~~ [ERROR]Could not read image: https://foxaholic.com/art/2020/05/2850b1f9e0afd270d61c893f77b209f2_f1447.jpg java.io.FileNotFoundException: https://foxaholic.com/art/2020/05/2850b1f9e0afd270d61c893f77b209f2_f1447.jpg at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1981) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1577) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224) at grabber.GrabberUtils.getImage(GrabberUtils.java:47) at grabber.Chapter.getImages(Chapter.java:99) at grabber.Chapter.saveChapter(Chapter.java:49) at grabber.Novel.downloadChapters(Novel.java:118) at gui.GUI.lambda$null$2(GUI.java:342) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at java.base/java.lang.Thread.run(Thread.java:831) [ERROR]Could not save image: 2850b1f9e0afd270d61c893f77b209f2_f1447.jpg [INFO]Saved chapter: 323 - Valentine's Day Extra Chapter ~~ Lilia & Sieglinde ~~

Flameish commented 3 years ago

java.lang.OutOfMemoryError: Java heap space was the problem and not the images, they don't matter that much if one fails.

NG stores images in buffer which can cause the heap space to go pretty high if the novel has many. I tried the novel you linked and it downloaded everything without problem, the memory usage was at around 1GB afterwards. Make sure that you have the 64bit version of java installed, it got a higher max heap space setting or you can try starting the Novel-Grabber.jar with the command argument -Xmx3072m (3GB RAM) for example from cmd/terminal.

PasteLord commented 3 years ago

Ok boss thanks! 🙆‍♂️

Did the stuff you told me, worked like a charm. Cheers~