Previously, variable database was a flat JSON file. This was limiting in a couple of ways:
Updates mean re-writing the whole file every time
The entire file must be loaded into memory
Because of the previous limitation, it was difficult to paginate responses in the REST API, which led to corrupt responses (see #37)
This implements a simple binary key-value database format which is more scalable from a storage perspective.
Presently, all upserts and queries involve flash operations. This may prove too costly. If this ends up being the case, it should be simple to put an in-memory cache+buffer in front of it.
Transient variables (presently this only includes the timestamp) are never persisted.
Integration tests
Mostly unrelatedly, a suite of integration tests written in rspec is located in ./test/remote. These can be run with:
bundle install && bundle exec rspec
The tests are entirely aimed at the REST API. Future coverage may include MQTT and websockets.
Two major changes are included:
Variable database rewrite
Previously, variable database was a flat JSON file. This was limiting in a couple of ways:
This implements a simple binary key-value database format which is more scalable from a storage perspective.
Presently, all upserts and queries involve flash operations. This may prove too costly. If this ends up being the case, it should be simple to put an in-memory cache+buffer in front of it.
Transient variables (presently this only includes the timestamp) are never persisted.
Integration tests
Mostly unrelatedly, a suite of integration tests written in rspec is located in
./test/remote. These can be run with:The tests are entirely aimed at the REST API. Future coverage may include MQTT and websockets.