search

kriszyp / lmdb-js

Simple, efficient, ultra-fast, scalable data store wrapper for LMDB
Other
486 stars 39 forks source link

Segfault #40

Closed vladar closed 3 years ago

vladar commented 3 years ago

Not sure if it's possible to investigate this but I've got a segfault in a combination of very heavy JS heap memory growth and massive writes to LMDB.

PID 424125 received SIGSEGV for address: 0x7fc63ab680d0
/spike/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x3785)[0x7fc3d0429785]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fc3d476b3c0]
/spike/node_modules/lmdb-store/build/Release/lmdb-store.node(+0x19b57)[0x7fc3d1d11b57]
/spike/node_modules/lmdb-store/build/Release/lmdb-store.node(+0x8fda)[0x7fc3d1d00fda]
/spike/node_modules/lmdb-store/build/Release/lmdb-store.node(+0x15df0)[0x7fc3d1d0ddf0]
/spike/node_modules/lmdb-store/build/Release/lmdb-store.node(+0x17d69)[0x7fc3d1d0fd69]
/spike/node_modules/lmdb-store/build/Release/lmdb-store.node(+0x3ac93)[0x7fc3d1d32c93]
/root/.nvm/versions/node/v14.15.4/bin/node[0x1376de4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7fc3d475f609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fc3d4686293]

Before the segfault there were several warnings about automatic commits:

warn Performing synchronous commit in database nodes because over 200000000 bytes were included in one transaction,
should run transactions over separate event turns to avoid this or increase syncBatchThreshold
kriszyp commented 3 years ago

I think the only time I have seen this is when running out of disk space, which can cause malloc's to fail (no more virtual memory). I presume that isn't the case here. Also, have you tried lowering syncBatchThreshold (to force sync commits without using so much memory)? Or, as the error suggests, is it possible run your updates across multiple event turns (periodically wait for a setImmediate to resume writes)?

kriszyp commented 3 years ago

Another possible thing to do is trying rebuilding lmdb-store with debug symbols explicitly enabled, which I think is node-gyp --debug configure rebuild, which might give more informative stack trace if it crashes.

vladar commented 3 years ago

I only saw this once and couldn't reproduce it again. Curious if automatic resizing could have potentially caused this?

P.S. BTW, the project is very solid. I am impressed. Thanks for building it!

vladar commented 3 years ago

Haven't seen it anymore and there were several new version released since then, so closing. I Will re-open it if I see it again. Thanks for the suggestions!