Improper Memory Allocation When Compressing JPEGs and 24-bit Images?

[nopeynope60]

So something I've noticed during my tests is that the overall memory allocated to paq8pxd does not match up with what is being stated at the end when it finished.

Two examples compressed individually: one JPEG and one uncompressed 24-bit image.

Baseline JPEG (-s9): Task Manager and Process Explorer reports 2.8GiB allocated. Reported memory usage by paq8pxd is: 3712 MB (3893306956 bytes)

Uncompressed 24-bit Image (-s9): Task Manager and Process Explorer reports 1.2GiB allocated. Reported memory usage by paq8pxd is: 3764 MB (3947871882 bytes)

It gets even wilder at (-s15) where I can manage to trigger and "Out of Memory" error at the very end of the compression if I start doing other things on my system while it's running. This is why I'm thinking it's some kind of memory allocation issue.

[kaitz]

This is not a bug. Reported memory is total maximum. JPEG itself uses below 1G but compressing headers and all other uses default model witch uses more memory. You can see it int Task Manger column Commit size.

[nopeynope60]

It's dynamically allocating when it should be a complete static allocation based on the max memory throughout the entire process. For what reason would you ever want dynamic allocation of memory in this instance? Yeah, it's wasteful for a full static allocation, but you're less likely to run into "Out of Memory" issues as a result.

I should really emphasize just how easy it is for me to trigger out of memory errors. I'm using paq8pxd while I'm also using my system. This dynamic allocation assumes you are not doing anything else on your system while paq8pxd is running.

[kaitz]

https://github.com/kaitz/paq8pxd/blob/ff16113b90aff59d320f75ad8e2166fed8ed2208/paq8pxd.cpp#L22295 https://github.com/kaitz/paq8pxd/blob/ff16113b90aff59d320f75ad8e2166fed8ed2208/paq8pxd.cpp#L22552 At these points default model is used, here comes total maximum memory usage. Segment data compression uses same amount of memory as header compression. Its static per stream model.

[kaitz]

Closed