search

quantifyearth / shark

3 stars 1 forks source link

Default shared memory limits too restrictive #65

Open mdales opened 3 months ago

mdales commented 3 months ago

I've hit this with docker (explained to me by @avsm) that the default limits on shared memory are quite restrictive in terms of the datasets we work with. I've been chasing some random sigbus errors in shark (process exits with signal 135) and I believe this is the same root cause.

This ticket is to remind me to look at making this configurable or at least have run use a higher bound. For now I can work around it by disabling parallelism in yirgacheffe

patricoferris commented 3 months ago

Potentially relevant is where we set this in the obuilder (runc) world -- currently to 65536k which is probably the same as docker: https://github.com/quantifyearth/obuilder/blob/850bd8bfd35370b6bdea4de9058804aed3c8b4e6/lib/sandbox.runc.ml#L202-L211

mdales commented 3 months ago

Yeah, I believe docker is the same, but 64MB is too small in a geospatial context. I added parallel_save to which shares data via shared memory, and really I think in this context we should set it to a large %age of actual memory. But if Obuilder is merged we can easily make that change for our use case.

mdales commented 3 months ago

Created this PR to raise it to 64GB for now. https://github.com/quantifyearth/obuilder/pull/2