Investigate memory-mapped files on the JVM

[oubiwann]

This would be useful for the map service:

• Allow for a greater number of map layers without impacting memory consumption by Java process
• Allow for maps with large image resolutions (and thus provide players with data in a tighter physical locale, in-world)
• Support run-time modifications of the world
• Support faster access to image files
• Obviate the need to store image data in a database

[oubiwann]

The clj-mmap project provides a wrapper around Java NIO's support for memory-mapped files. Looking at the source code, access is typical for files: get n bytes at location i. This isn't exactly what we need; perhaps a layer on top of that, for accessing x, y coordinates of an image?

[oubiwann]

Here's an example of someone creating NIO Memory-mapped file images:

• https://stackoverflow.com/a/41765469

[oubiwann]

What about creating a RoaringBitmap for each layer?

• r, g, b would each get their own rbmap
• biome images could be processed to generate a rbmap whose values would be the hexgram30/map indices in the biome lookup data structure
• temperature and precipitation images could be processed to correspond to the index of ranges (thus mapping to a color in the legend)
• overlays could just be zeros and ones (e.g., rivers, land, sea, ice, lakes, seas, etc.)

There doesn't seem to be a Clojure wrapper for RoaringBitmap, but here's an example of memory-mapped RoaringBitmap usage.

[oubiwann]

Time to run some experiments:

1. Create a sample Clojure project with some sample images that are loaded into memory
2. Create an API for accessing the data of interest in these images
3. Start up a service with no images loaded; measure Java process/threads memory consumption (might need to enable JMX for this)
4. Load one image/layer at a time, perform an API query against the images, and measure impact on memory consumption
5. Create a new implementation for loading these images as Java NIO memory-mapped files
6. Repeat the process of system startup, initial memory measurement, and subsequent image loading and impact on memory
7. Create a new implementation for RoaringBitmap support, converting the image data as necessary
8. Repeat the process of system startup, initial memory measurement, and subsequent image loading and impact on memory
9. Create a new implementation for memory-mapped RoaringBitmap support
10. Repeat the process of system startup, initial memory measurement, and subsequent image loading and impact on memory

[oubiwann]

Some additional considerations, before I put too much time in the testing approach above:

1. Roaringbitmap specifically entails the storage of 1-dimensional data; there is a discussion here and here about supporting matrices and matrix operations (very clever stuff).
2. However, all we need is storage and retrieval (i.e., we won't be doing any linear algebra), so we could actually use SFCs (I'm thinking Hilbert curve). However, we'll loose the compressability of data with that ... there will no longer be long series of the same bool or integer; instead there will be a large array of essentially hashed 2-dimensional coordinates.
3. Even if Roaringbitmap is still a good fit for Hilbert curve data, we've got an access problem: the mapping service will allow multiple readers; potentially many readers.
4. Furthermore, games/stories that support a mutating world will need to write to the 2-D coords ...

So I don't think we can get away from database storage, here. I also think the project will be best-served by a file-based database, not an in-memory database.

[oubiwann]

I've convinced myself that we don't want to manage RGB layers, precipitation, temperature, hightmap, rivers, oceans, land, ice, etc., from a single JVM process. So I'm going to close this ticket and focus on the service-based approach investigation ticket here: #30

In that ticket I'll mention consideration for array databases and see if there are any databases that use Roadringbitmap as an underlying store ... in case that's still a good choice.