Support for binary data

[sullivan-]

Add support for binary data. This should be relatively easy with the current implementation. Just add some kind of binary type (e.g. an Array[Byte]) to basicTypes. This would in turn need to be stored as base64 in the JSON.

We would probably want to inhibit producing properties for this new basic type, as indexing or querying with <, ==, and > might be a little odd.

Any implementation would probably be affected by migration to shapeless. In particular, we should investigate how circe. I don't see any harm in adding this feature before the migration to shapeless, so long as we convince ourselves that circe will be able to produce the proper JSON first.

[sullivan-]

I'll note that concerns may come up about the size of the binary data being incorporated here. E.g., MongoDB has a 16MB size limit on an individual record. But we work with limitations of underlying database all the time.

[mardo]

Mongo's GridFS supports data streaming; this can help overcome the 16MB limit, plus being able to store huge files and, again, stream it, so we don't need all the bytes to fit in memory. Maybe worth checking it out :D

Here's one crazy idea: Other databases, let's say cassandra or SQLite, do not support storing files (at least not big files and steaming features) so, we could have the FS backend separate.

In the same manner longevity provides an abstraction layer for your data, it could provide one for your binaries. FS backends would be something like:

• OS filesystem
• Mongo's GridFS
• AWS S3

[sullivan-]

That sounds like a good idea to me (GridFS, etc). It would involve some work.

Just to note this could be done in parallel with an in-record implementation when people know the binary data is small.