aionbot
commented
5 years ago Comment by jeff-aion (on Friday Oct 26, 2018 at 20:36 GMT)
Looking further into MappedByteBuffer, there are some problems with it which may make it not appropriate for this usage (which is odd, since large files is documented as its point - they just can't be too large or numerous):
- No explicit
unmapsupport. While the reasons to avoid this are somewhat understandable, mapped file regions always require some amount of "be careful", so it would be nice to see. As a consequence, it is possible for the mapped files to clutter the address space even when they aren't used (keeping disk resources alive and potentially blocking deletes, on Windows). - There are limits around the mapping such that only 2 GiB mappings (no greater than
Integer.MAX_VALUE) are possible, which would require that we build a paging system to handle how to address fragments of large files (unless we convince ourselves that it would be unreasonably expensive for a single DApp to ever require 2 GiB of data, even including all accessible versions)
An alternative may be to use JNA to access the LibC mmap/unmap calls, directly (some discussion of that idea found here https://stackoverflow.com/questions/15901832/how-to-memory-map-mmap-a-linux-block-device-e-g-dev-sdb-in-java ). The problem is that this substantially limits our portability.
Issue created by jeff-aion (on Friday Sep 28, 2018 at 16:06 GMT)
Create an implementation of
IObjectGraphStorewhich acts as a native graph store (not built on a key-value store). The complexity of this is in the multi-version support (which will also require the definition of that relationship withIObjectGraphStore). Otherwise, it should be relatively straight-forward to implement this on top ofMappedByteBuffer.