get-got / discord-downloader-go

A Discord bot program to download and otherwise handle files sent in Discord channels with extensive configuration. Can be used as a genuine Discord Bot or user account / self-bot
MIT License
271 stars 30 forks source link

Out of memory #110

Open HASTJI opened 7 months ago

HASTJI commented 7 months ago

Hello.

Running program as Administrator, disk C and other disks has more then 20+ gb of free space, also have 7 of 16gb free ram, BUT when trying to download several .mov's( each is 40-50mb) in a row from discord channel - it said that there was an error and throw me a bunch of debug info:

runtime: out of memory: cannot allocate 301989888-byte block (540606464 in use)
fatal error: out of memory

goroutine 256 [running]:
runtime.throw({0x1183d5f, 0xd})
        C:/Program Files/Go/src/runtime/panic.go:1047 +0x4d fp=0xb4bf4f0 sp=0xb4bf4dc pc=0xab6cad
runtime.(*mcache).allocLarge(0x25609c8, 0x11edc000, 0x1)
        C:/Program Files/Go/src/runtime/mcache.go:236 +0x1be fp=0xb4bf518 sp=0xb4bf4f0 pc=0xa93f7e
runtime.mallocgc(0x11edc000, 0x0, 0x0)
        C:/Program Files/Go/src/runtime/malloc.go:1053 +0x3eb fp=0xb4bf550 sp=0xb4bf518 pc=0xa8b85b
runtime.growslice(0xbb1c000, 0xe57c001, 0xe57c000, 0x1, 0x10b2d40)
        C:/Program Files/Go/src/runtime/slice.go:266 +0x3ec fp=0xb4bf57c sp=0xb4bf550 pc=0xacb78c
io.ReadAll({0x53f58690, 0xb0b4510})
        C:/Program Files/Go/src/io/io.go:699 +0x8f fp=0xb4bf5bc sp=0xb4bf57c pc=0xaf139f
main.downloadRequestStruct.tryDownload({{0xb22ed20, 0x66}, {0xb5570f8, 0x17}, {0x0, 0x0}, {0xb026900, 0x2f}, 0xb6d2750, {0x3750280, ...}, ...})
        P:/Projects/Go/src/github.com/get-got/discord-downloader-go/downloads.go:727 +0xf66 fp=0xb4c0138 sp=0xb4bf5bc pc=0x1027e86
main.downloadRequestStruct.handleDownload({{0xb22ed20, 0x66}, {0xb5570f8, 0x17}, {0x0, 0x0}, {0xb23c270, 0x2e}, 0xb6d2750, {0x3750280, ...}, ...})
        P:/Projects/Go/src/github.com/get-got/discord-downloader-go/downloads.go:458 +0x10a fp=0xb4c058c sp=0xb4c0138 pc=0x102551a
main.handleMessage(0xb6d2750, 0xb4c1b60, 0x0, 0x1)
        P:/Projects/Go/src/github.com/get-got/discord-downloader-go/handlers.go:374 +0x2796 fp=0xb4c0bac sp=0xb4c058c pc=0x1031c56
main.handleHistory(0xb609860, {0xb0b6b47, 0x13}, {0x0, 0x0}, {0x0, 0x0})
        P:/Projects/Go/src/github.com/get-got/discord-downloader-go/history.go:538 +0x62d4 fp=0xb4c1fcc sp=0xb4c0bac pc=0x10394c4
main.main.func2.1()
        P:/Projects/Go/src/github.com/get-got/discord-downloader-go/main.go:191 +0x51 fp=0xb4c1ff0 sp=0xb4c1fcc pc=0x1049cd1
runtime.goexit()
        C:/Program Files/Go/src/runtime/asm_386.s:1326 +0x1 fp=0xb4c1ff4 sp=0xb4c1ff0 pc=0xae4e41
created by main.main.func2
        P:/Projects/Go/src/github.com/get-got/discord-downloader-go/main.go:191 +0x4a3

goroutine 1 [chan receive, 7 minutes]:
runtime.gopark(0x11be018, 0xb0371b0, 0xe, 0x17, 0x2)
        C:/Program Files/Go/src/runtime/proc.go:381 +0xff fp=0xb05da48 sp=0xb05da34 pc=0xab9bdf
runtime.chanrecv(0xb037180, 0x0, 0x1)
        C:/Program Files/Go/src/runtime/chan.go:583 +0x3f4 fp=0xb05da90 sp=0xb05da48 pc=0xa85be4
runtime.chanrecv1(0xb037180, 0x0)
        C:/Program Files/Go/src/runtime/chan.go:442 +0x1c fp=0xb05daa4 sp=0xb05da90 pc=0xa857ec
main.main()
        P:/Projects/Go/src/github.com/get-got/discord-downloader-go/main.go:484 +0x19b7 fp=0xb05dfc4 sp=0xb05daa4 pc=0x103ccd7
runtime.main()
        C:/Program Files/Go/src/runtime/proc.go:250 +0x22e fp=0xb05dff0 sp=0xb05dfc4 pc=0xab97ae
runtime.goexit()
        C:/Program Files/Go/src/runtime/asm_386.s:1326 +0x1 fp=0xb05dff4 sp=0xb05dff0 pc=0xae4e41

goroutine 2 [force gc (idle), 7 minutes]:
runtime.gopark(0x11be104, 0x167bc78, 0x11, 0x14, 0x1)
        C:/Program Files/Go/src/runtime/proc.go:381 +0xff fp=0xb057fdc sp=0xb057fc8 pc=0xab9bdf
runtime.goparkunlock(...)
        C:/Program Files/Go/src/runtime/proc.go:387
runtime.forcegchelper()
        C:/Program Files/Go/src/runtime/proc.go:305 +0xcf fp=0xb057ff0 sp=0xb057fdc pc=0xab9a0f
runtime.goexit()
        C:/Program Files/Go/src/runtime/asm_386.s:1326 +0x1 fp=0xb057ff4 sp=0xb057ff0 pc=0xae4e41
created by runtime.init.5
        C:/Program Files/Go/src/runtime/proc.go:293 +0x23

and other 34+ goroutine. Is there any way to fix this?

get-got commented 7 months ago

This is happening consistently and not just once?

HASTJI commented 7 months ago

Tried 3 channels - every channel with videos and images have that error. Error can appear after 40+(randomly) downloaded images+videos or after start of first file download - I didn't found any viable pattern. But every time its "out of memory" with random numbers of memory allocation errors.

Now tried run it again on channel, which I downloaded some time ago, but got same error on new files downloading: изображение

Nyaster commented 6 months ago

Same issue here, with same error. Found only in the Windows version. Solved where used WSL and compiled own from source.

get-got commented 6 months ago

Same issue here, with same error. Found only in the Windows version. Solved where used WSL and compiled own from source.

Interesting, thanks for the info.

I'm not sure how to resolve this as I haven't been able to recreate it. I'll keep note, I'll see if soon I can recreate it on a different Windows machine.