Downloading a new version of a large folder uses all physical memory available?

[AOOSG]

Is it expected that the longtail binary uses all available physical memory when updating (i.e. downloading) a large folder (like getting an updated UE5 version).

Longtail was using ~42GB of memory or so if I remember correctly.

It did not cause any issues, but it seems like a lot of memory usage.

[DanEngelbrecht]

Yes, it uses a lot, 48G seems a bit aggressive. Try —mem-trace or —mem-trace-detailed to see some details. The more cores your machine have the more memory it will use. You can limit the number of worker threads with —worker-count to possibly affect this.

[AOOSG]

Thanks, good to know. Feel free to close this issue then. I'll run with those flags next time to take a closer look at the memory usage!

[AOOSG]

Here's the memory details from using --mem-trace with 32 threads, 25GB it seems!

[MEM] total_mem:     90.88 gb (97576943845)
[MEM] current_mem:   0
[MEM] peak_mem:      25.21 gb (27071095268)
[MEM] total_count:   1.33 mb (1398005)
[MEM] current_count: 0
[MEM] peak_count:    177.94 kb (182206)

[DanEngelbrecht]

Yeah, that do seem a bit excessive 😀 I don’t have a simple solution to limit the memory usage but will have to investigate. Where is the store data saved? On GCS or S3 or something else?

[DanEngelbrecht]

Could you try https://github.com/DanEngelbrecht/golongtail/releases/tag/v0.3.6-pre1 with --mem-trace and --mem-trace-detailed and post the results?

[DanEngelbrecht]

Fixed with https://github.com/DanEngelbrecht/golongtail/pull/227