Memory Leak?

[cstegmann]

Hi there,

I'm using the docker image ghcr.io/xonshiz/comic-dl:latest and when using it to grab more than one chapter at a time the memory usage keeps increasing until the host machine either runs out of memory and crashes or it exits. I set a 2000 MB limit on the container now using the --memory flag but that is reached after about half an hour.

Here's the docker command:

docker run -d --name comic-dl --memory=2000m -v /c/Users/Username/ComicDL:/directory:rw -w /directory ghcr.io/xonshiz/comic-dl:latest_linux_amd64 comic_dl -dd /directory --auto

and here's the config.json I'm using:

{
    "download_directory": "comics",
    "sorting_order": "ascending",
    "conversion": "cbz",
    "keep": "False",
    "cookie": "None",
    "image_quality": "Best",
    "comics": {
        "https://fanfox.net/manga/peerless_dad/": {
            "url": "https://fanfox.net/manga/peerless_dad/",
            "next": 1,
            "last": "None",
            "username": "None",
            "password": "None",
            "comic_language": "0"
        }
}

I had to add the "cookie":"None", in there myself or it would refuse to run at all.

Thanks for the help, even with the memory leak I am slowly but steadily getting things done.

EDIT: Added a screenshot

[Xonshiz]

That is an interesting find. I'm not quite sure where exactly it might be doing this, but do you see this issue for just "fanfox" or is it happening with other websites too? My first guess was that it's trying to spawn multiple threads, but I think we had a limit of max concurrent threads somewhere 🤔 .

[Xonshiz]

Oh and that's while downloading the actual images only and not while scraping the page.

[cstegmann]

That is an interesting find. I'm not quite sure where exactly it might be doing this, but do you see this issue for just "fanfox" or is it happening with other websites too? My first guess was that it's trying to spawn multiple threads, but I think we had a limit of max concurrent threads somewhere 🤔 .

I tried it with Mangadex as well, same problem. I also updated to the 2023.01.08.2 image, I was using 2023.01.08 when I wrote the issue.

I don't know if this will help, but I recorded it when I tested it with mangadex.

https://imgur.com/a/mvrl0qj

[Xonshiz]

Thanks for sharing this. I'll try this out on my machine this weekend. 👍🏽

[cstegmann]

Did you have a chance to take a look at this yet?

[Xonshiz]

No, really packed with professional and personal life. Super difficult to find out time for this. But, I'm planning on re-writing this entire script soon. I'll try to find some time soon 🙏🏽