search

Girbons / comics-downloader

tool to download comics and manga in pdf/epub/cbr/cbz from a website
MIT License
453 stars 48 forks source link

Prevent out of memory #98

Closed ChristofferGreen closed 2 years ago

ChristofferGreen commented 2 years ago

When downloading a large set of comics (for example Spawn) on a raspberry pi comics-downloader would crash due to running out of memory. Adding a call to garbage collection in the loop for downloading images prevents this from happening.

ChristofferGreen commented 2 years ago

Let me know if you feel there is a better way to do this.

Girbons commented 2 years ago

When downloading a large set of comics (for example Spawn) on a raspberry pi comics-downloader would crash due to running out of memory.

@ChristofferGreen could you please share more info? How are you running the comics-downloader? How many comics you are trying to download and the raspberry spec?

Calling the Garbace collector like that might not the best way to solve the issue, I mean we should start looking to other things to improve the performance and memory consumption.

ChristofferGreen commented 2 years ago

Hi!

Yes, this might not be a good way to solve it, it may just be a solution that works for my needs.

Sharing some more info:

Running on a raspberry pi 4b 4gb with about 32gb of swap. I think the process runs out of 32-bit ram rather than the machine. Os: Raspbian GNU/Linux 10 (buster)

go version go1.15.15 linux/arm

Output log: ~/src/comics-downloader:master$ ./comics-downloader -url https://www.comicextra.com/comic/spawn -all
INFO[0000] Downloading...
100% |████████████████████████████████████████| [2s:0s]INFO[0083] PDF file correctly saved
100% |████████████████████████████████████████| [3s:0s]INFO[0087] PDF file correctly saved
100% |████████████████████████████████████████| [4s:0s]INFO[0092] PDF file correctly saved
100% |████████████████████████████████████████| [3s:0s]INFO[0095] PDF file correctly saved
100% |████████████████████████████████████████| [4s:0s]INFO[0100] PDF file correctly saved
100% |████████████████████████████████████████| [4s:0s]INFO[0105] PDF file correctly saved
100% |████████████████████████████████████████| [5s:0s]INFO[0110] PDF file correctly saved
100% |████████████████████████████████████████| [3s:0s]INFO[0114] PDF file correctly saved
100% |████████████████████████████████████████| [3s:0s]INFO[0118] PDF file correctly saved
100% |████████████████████████████████████████| [3s:0s]INFO[0122] PDF file correctly saved
100% |████████████████████████████████████████| [4s:0s]INFO[0127] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0130] PDF file correctly saved
100% |████████████████████████████████████████| [0s:0s]INFO[0130] PDF file correctly saved
100% |████████████████████████████████████████| [0s:0s]INFO[0131] PDF file correctly saved
100% |████████████████████████████████████████| [0s:0s]INFO[0131] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0134] PDF file correctly saved
100% |████████████████████████████████████████| [3s:0s]INFO[0138] PDF file correctly saved
100% |████████████████████████████████████████| [3s:0s]INFO[0141] PDF file correctly saved
100% |████████████████████████████████████████| [3s:0s]INFO[0145] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0148] PDF file correctly saved 100% |████████████████████████████████████████| [3s:0s]INFO[0152] PDF file correctly saved 100% |████████████████████████████████████████| [5s:0s]INFO[0158] PDF file correctly saved 100% |████████████████████████████████████████| [7s:0s]INFO[0166] PDF file correctly saved 100% |████████████████████████████████████████| [4s:0s]INFO[0171] PDF file correctly saved 100% |████████████████████████████████████████| [3s:0s]INFO[0174] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0177] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0180] PDF file correctly saved 100% |████████████████████████████████████████| [17s:0s]INFO[0200] PDF file correctly saved 100% |████████████████████████████████████████| [17s:0s]INFO[0219] PDF file correctly saved 100% |████████████████████████████████████████| [22s:0s]INFO[0244] PDF file correctly saved 100% |████████████████████████████████████████| [16s:0s]INFO[0263] PDF file correctly saved 100% |████████████████████████████████████████| [16s:0s]INFO[0281] PDF file correctly saved 100% |████████████████████████████████████████| [16s:0s]INFO[0299] PDF file correctly saved 100% |████████████████████████████████████████| [16s:0s]INFO[0318] PDF file correctly saved 100% |████████████████████████████████████████| [15s:0s]INFO[0335] PDF file correctly saved 100% |████████████████████████████████████████| [16s:0s]INFO[0353] PDF file correctly saved 100% |████████████████████████████████████████| [27s:0s]INFO[0383] PDF file correctly saved 100% |████████████████████████████████████████| [16s:0s]INFO[0401] PDF file correctly saved 100% |████████████████████████████████████████| [17s:0s]INFO[0421] PDF file correctly saved 100% |████████████████████████████████████████| [18s:0s]INFO[0441] PDF file correctly saved 100% |████████████████████████████████████████| [16s:0s]INFO[0460] PDF file correctly saved 100% |████████████████████████████████████████| [19s:0s]INFO[0481] PDF file correctly saved 100% |████████████████████████████████████████| [18s:0s]INFO[0507] PDF file correctly saved 100% |████████████████████████████████████████| [15s:0s]INFO[0525] PDF file correctly saved 100% |████████████████████████████████████████| [15s:0s]INFO[0542] PDF file correctly saved 100% |████████████████████████████████████████| [14s:0s]INFO[0558] PDF file correctly saved 100% |████████████████████████████████████████| [16s:0s]INFO[0577] PDF file correctly saved 100% |████████████████████████████████████████| [3s:0s]INFO[0580] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0583] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0586] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0589] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0592] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0595] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0597] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0600] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0603] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0605] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0608] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0611] PDF file correctly saved
100% |████████████████████████████████████████| [2s:0s]INFO[0614] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0616] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0619] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0622] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0625] PDF file correctly saved 100% |████████████████████████████████████████| [3s:0s]INFO[0628] PDF file correctly saved 100% |████████████████████████████████████████| [3s:0s]INFO[0632] PDF file correctly saved 100% |████████████████████████████████████████| [0s:0s]INFO[0632] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0635] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0637] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0640] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0643] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0645] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0648] PDF file correctly saved 100% |████████████████████████████████████████| [3s:0s]INFO[0651] PDF file correctly saved 100% |████████████████████████████████████████| [2s:0s]INFO[0654] PDF file correctly saved 100% |████████████████████████████████████████| [3s:0s]INFO[0657] PDF file correctly saved 100% |████████████████████████████████████████| [3s:0s]INFO[0661] PDF file correctly saved 100% |████████████████████████████████████████| [3s:0s]INFO[0665] PDF file correctly saved 100% |████████████████████████████████████████| [3s:0s]INFO[0669] PDF file correctly saved 100% |████████████████████████████████████████| [5s:0s]INFO[0674] PDF file correctly saved 100% |████████████████████████████████████████| [4s:0s]INFO[0679] PDF file correctly saved 100% |████████████████████████████████████████| [3s:0s]INFO[0683] PDF file correctly saved 100% |████████████████████████████████████████| [1m36s:0s]runtime: out of memory: cannot allocate 4194304-byte block (2476539904 in use) fatal error: out of memory

runtime stack: runtime.throw(0x3f89db, 0xd) /usr/local/go/src/runtime/panic.go:1116 +0x5c runtime.largeAlloc(0xa0488, 0x70001, 0xb2c000) /usr/local/go/src/runtime/malloc.go:1179 +0x174 runtime.mallocgc.func1() /usr/local/go/src/runtime/malloc.go:1071 +0x38 runtime.systemstack(0x152) /usr/local/go/src/runtime/asm_arm.s:347 +0x84 runtime.mstart() /usr/local/go/src/runtime/proc.go:1116

goroutine 1 [running]: runtime.systemstack_switch() /usr/local/go/src/runtime/asm_arm.s:294 +0x4 fp=0x16849e8 sp=0x16849e4 pc=0x74bf0 runtime.mallocgc(0xa0488, 0x3aa868, 0x1, 0x4737c4) /usr/local/go/src/runtime/malloc.go:1070 +0x8f8 fp=0x1684a4c sp=0x16849e8 pc=0x1ddd4 runtime.newobject(0x3aa868, 0x1) /usr/local/go/src/runtime/malloc.go:1195 +0x2c fp=0x1684a60 sp=0x1684a4c pc=0x1e0f4 compress/flate.NewWriter(0x47c678, 0x16505b0, 0x1, 0x40, 0x27f2280, 0x2c) /usr/local/go/src/compress/flate/deflate.go:670 +0x1c fp=0x1684a80 sp=0x1684a60 pc=0x1e20e0 compress/flate.NewWriterDict(0x47c630, 0x33c0ac0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x40, 0x3cc738) /usr/local/go/src/compress/flate/deflate.go:685 +0x58 fp=0x1684abc sp=0x1684a80 pc=0x1e21b4 compress/zlib.(Writer).writeHeader(0xb7a4f40, 0x3cc738, 0xb7a4f40) /usr/local/go/src/compress/zlib/writer.go:132 +0x144 fp=0x1684ae8 sp=0x1684abc pc=0x25db4c compress/zlib.(Writer).Write(0xb7a4f40, 0x155e420, 0x7e, 0xa8, 0x0, 0x0, 0xb7a4f40) /usr/local/go/src/compress/zlib/writer.go:146 +0x13c fp=0x1684b14 sp=0x1684ae8 pc=0x25de5c github.com/jung-kurt/gofpdf.sliceCompress(0x155e420, 0x7e, 0xa8, 0x1684c48, 0x1, 0x1) /home/pi/go/pkg/mod/github.com/jung-kurt/gofpdf@v1.16.2/util.go:88 +0x78 fp=0x1684b44 sp=0x1684b14 pc=0x2df268 github.com/jung-kurt/gofpdf.(Fpdf).putpages(0x14bc000) /home/pi/go/pkg/mod/github.com/jung-kurt/gofpdf@v1.16.2/fpdf.go:3944 +0x330 fp=0x1684d58 sp=0x1684b44 pc=0x2cb0cc github.com/jung-kurt/gofpdf.(Fpdf).enddoc(0x14bc000) /home/pi/go/pkg/mod/github.com/jung-kurt/gofpdf@v1.16.2/fpdf.go:4787 +0xd4 fp=0x1684d8c sp=0x1684d58 pc=0x2d31b4 github.com/jung-kurt/gofpdf.(Fpdf).Close(0x14bc000) /home/pi/go/pkg/mod/github.com/jung-kurt/gofpdf@v1.16.2/fpdf.go:692 +0x90 fp=0x1684dac sp=0x1684d8c pc=0x2c5d44 github.com/jung-kurt/gofpdf.(Fpdf).Output(0x14bc000, 0x47cc90, 0x1650580, 0x1b6, 0x1650580) /home/pi/go/pkg/mod/github.com/jung-kurt/gofpdf@v1.16.2/fpdf.go:3469 +0xa8 fp=0x1684dcc sp=0x1684dac pc=0x2c914c github.com/jung-kurt/gofpdf.(Fpdf).OutputFileAndClose(0x14bc000, 0x27df5c0, 0x53, 0x5, 0x1619ee1) /home/pi/go/pkg/mod/github.com/jung-kurt/gofpdf@v1.16.2/fpdf.go:3450 +0xb8 fp=0x1684df0 sp=0x1684dcc pc=0x2c9060 github.com/Girbons/comics-downloader/pkg/core.(Comic).makePDF(0x147b0e0, 0x1500f00, 0x0, 0x0) /home/pi/src/comics-downloader/pkg/core/core.go:177 +0x614 fp=0x1684ef0 sp=0x1684df0 pc=0x304190 github.com/Girbons/comics-downloader/pkg/core.(*Comic).MakeComic(0x147b0e0, 0x1500f00, 0x0, 0x0) /home/pi/src/comics-downloader/pkg/core/core.go:334 +0x9c fp=0x1684f04 sp=0x1684ef0 pc=0x3052a4 github.com/Girbons/comics-downloader/cmd/app.download(0x1500f00) /home/pi/src/comics-downloader/cmd/app/downloader.go:87 +0x2bc fp=0x1684f64 sp=0x1684f04 pc=0x35e7dc github.com/Girbons/comics-downloader/cmd/app.Run(0x1500f00) /home/pi/src/comics-downloader/cmd/app/downloader.go:123 +0x178 fp=0x1684f84 sp=0x1684f64 pc=0x35eb7c main.main() /home/pi/src/comics-downloader/cmd/downloader/main.go:93 +0x210 fp=0x1684fb8 sp=0x1684f84 pc=0x35f6e4 runtime.main() /usr/local/go/src/runtime/proc.go:204 +0x208 fp=0x1684fe4 sp=0x1684fb8 pc=0x47c60 runtime.goexit() /usr/local/go/src/runtime/asm_arm.s:857 +0x4 fp=0x1684fe4 sp=0x1684fe4 pc=0x76a1c

This also happens when saving as cbz, so not pdf specific.

Girbons commented 2 years ago

Thank you for the details @ChristofferGreen, I'll try to look into it and let you know what we can do to prevent that

Girbons commented 2 years ago

@ChristofferGreen what about trying a worker pool here taking as example https://pkg.go.dev/golang.org/x/sync/semaphore, it would be something like that, could you do a test to see if it solves the issue?

    maxWorkers := int64(runtime.NumCPU())
    sem := semaphore.NewWeighted(maxWorkers)
    ctx := context.Background()

    defer sem.Acquire(ctx, maxWorkers)
    for i, link := range comic.Links {
        link, i := link, i
        sem.Acquire(ctx, 1)

        if link == "" {
            continue
        }

        g.Go(func() error {
            defer sem.Release(1)
            rsp, err := options.Client.Get(link)
            if err != nil {
                return err
            }
            defer rsp.Body.Close()

            imgFile, err := os.Create(fmt.Sprintf("%04d-image.%s", i, format))
            if err != nil {
                return err
            }
            defer imgFile.Close()

            err = util.SaveImage(imgFile, rsp.Body, format)
            if err != nil {
                return err
            }

            if barErr := bar.Add(1); barErr != nil {
                options.Logger.Error(barErr.Error())
            }
            return nil

        })
    }
Girbons commented 2 years ago

Should be solved in v0.33.0