search

nats-io / stan.net

The official NATS .NET C# Streaming Client
Apache License 2.0
138 stars 41 forks source link

issue out of memory: cannot allocate 122880-byte block (1756626944 in use) #169

Closed minhhungit closed 4 years ago

minhhungit commented 4 years ago

When I publish about 10.000 messages to nats stream server I get out of memory issue while RAM is still available

1 message ~ 100.000 bytes

image

I also attached reproduce video and code for StanPub

StanPub.zip nats-stream-out-of-memory.zip

Full exception detail:

exception detail.txt

minhhungit commented 4 years ago

But it's good with docker image, here is video when I publish with docker

nats-stream-ok-with-docker-image.zip

minhhungit commented 4 years ago

nats-streaming-server-v0.16.2-windows-amd64.zip work quite well

nats-streaming-server-v0.16.2-windows-386.zip does not

ColinSullivan1 commented 4 years ago

This error seems reasonable. On windows, without special switches and being compiled to be "large address space aware", a 32 bit process can only access a maximum of 2 GB of user space memory.

You're approaching that with 1.75 GB allocated, and I suspect at that point the heap manager can't find any pages available that satisfy the 122880 byte allocation request.

The size of your payloads (aggregate) are 1GB, and there is additional overhead the server uses to manage that.

There are flags so that the streaming server can access more memory, but if you have access to a 64 bit OS, we highly recommend you run the 64 bit streaming server. Along with more user space memory (4GB) you'll see better performance. CC @kozlovic

kozlovic commented 4 years ago

@ColinSullivan1 Nothing to add to what you have mentioned. The 32bit will indeed be limited in addressable memory space, so using the MEMORY store is then not the right choice. The user can try with file store or simply move to the 64bit version.

minhhungit commented 4 years ago

Yeah, thank you for your explain guys. I understood. I will use 64bit version.