Give Field Modules their own store in IDB to persist data

[jgaehring]

I'm slowly moving towards removing the modules metadata from logs with the new implementation of farmLog (see #358). With that, it may become desirable to provide each field module with its own sandboxed IDB store, or perhaps an entirely separate database, for persisting data of their chosing, like a list of localID's that they want to keep track of and load, even if they don't match other criteria.

I'd like to avoid doing so until we have a clear use case that can't be met by any other means, but I thought I'd start the discussion here. Curious what your thoughts are, @paul121.

[paul121]

Oh yes this would be a great feature! Two major use cases come to mind: keeping user "state", and Field Module "config":

In the Sensors Field Module, for example, it would be nice if the module remembered the last sensor the user was viewing. So when they leave the page and come back it can be auto-loaded. Similarly remembering the past X sensors viewed, or remembering "frequently accessed" sensors and displaying those at the top of a drop down. I think this could be accomplished by being able to save asset IDs to the store under different "buckets" eg: last accessed and frequently accessed.

Along the same lines, I think there would be reason to save more "custom" objects in addition to just farmOS record IDs. The "sensor value" would be another bit of context that could be saved to the store, and would likely need to be associated with the asset ID as well. So... lastAccessed: [{ assetId: 5, sensorValue: "temp"}, {assetId: 10, sensorValue: "wind"}]

Saving the last accessed timestamp might be neat, too - then we could have a button to "Display new data" :D

Overall I see the use case for "state" as needing to be quite flexible regarding the types of data that can be stored.

---

For Field Module "config" perhaps this could be a bit more structured in the store (and even the UI). Idea being that Field Modules could have a config that can be modified within FK (or potentially a default config could be provided from the farmOS server). For the Sensors FM...

• Configure only certain types of Sensors to display on your device. Within the same farm, different users might have different needs.
• Configure the "default date range" to show past 24 hours, 3 days, 7 days, etc. Again customizable by the user.
• Configure how graphs are displayed (time scale? colors? etc)

This "config" store may need to be a bit more structured so that the farmOS server could have a standardized way of providing default config to any module (configured via an optional settings form each module could provide in its farmOS module). Additionally, I'm curious about the UI for configuring settings within FK: each module could provide their own custom config page, but perhaps this could be included in a FK core "config/settings" page? With different routes for each module that wants to provide a config?

I guess if the store for "config" was this structured it wouldn't need to be held within a FM's sandboxed IDB store, but could be saved in the FK core store. Or maybe it could be saved with the FM, and FK core is the only other one with access to it. But... should a FM be able to provide config for another FM? .... (as we've chatted about, makes me think about a FM that might not provide a UI, but just provide config. One example: providing farmOS-map behaviors. Could a "farmOS-map" module have a config for available behaviors, where other FM could provide behaviors, which are then loaded into any or a subset of map views?)

[paul121]

Oh and of course, it would be great if the Sensor FM could use this to save sensor data for offline viewing. Data model would be something like:

sensorData: [{ assetId: 5, timestamp: 123456678, valueName: 25, otherValueName: 1.5}]

[jgaehring]

All great ideas! Yea, I think there's a lot to parse out in terms of what parts of the data you described should be controlled by FK core and what should be left to the individual module.

I think some of the use cases you described might be well-suited for just using localStorage, like the timestamp you described, or anything that's a simple primitive value. But there should probably be some standards for how localStorage is used by FM's to avoid collision, since it is not nearly as structured and all the keys there are essentially "global". A timestamp key especially seems prone to collisions.

Additionally, I'm curious about the UI for configuring settings within FK: each module could provide their own custom config page, but perhaps this could be included in a FK core "config/settings" page? With different routes for each module that wants to provide a config?

Oh interesting. I really like this idea of having a standardized configuration UI for each module. It seems like it would lead to a more consistent UX. Perhaps we just have the modules supply that config as JSON and let core construct the UI with standard components, to make it all the more consistent.

Really cool, we'll have to keep this convo going!

[paul121]

The timestamp might be a bit more complicated since that would need to be associated with a sensor asset & sensor value (so potentially many timestamps, or maybe only the X most recent that were viewed).. but still, do you think the X most recent sensors might be a candidate for localstorage rather than a vuex store? And use the store for things like the sensor data? Curious where to draw that line! :thinking:

And yea, config is a separate issue. But a simple version of that could be implemented by a module if it had it's own store!

[jgaehring]

do you think the X most recent sensors might be a candidate for localstorage rather than a vuex store? And use the store for things like the sensor data? Curious where to draw that line!

Yea, lots to consider when choosing between IDB and localStorage. Biggest factor I think is synchronicity vs asynchronicity, but most of the factors end up favoring IDB for large amounts of data, localStorage for smaller stuff, as a general rule of thumb. Recent sensors could go either way, imo, so maybe better to err to the side of IDB? Ultimately, though, I'd like to leave that up to the module author.

Also I'd be careful not to conflate the issue with Vuex store, which I think is a separate concern. That's more about global state, rather than local persistence. I kind of feel that there shouldn't be much need for field modules to have their own piece of the Vuex store. Component state should be sufficient for holding what amounts to "global" for the field module. And anything shared between modules, eg, logs, assets, etc, should be controlled by core. But if you show me the use case where it would be necessary or really helpful for a FM to have its own Vuex module, I'm all ears.

[paul121]

Also I'd be careful not to conflate the issue with Vuex store, which I think is a separate concern. That's more about global state, rather than local persistence.

Oh! Maybe I misunderstood what you meant by a "sandboxed IDB store" - this would be separate from the vuex store? I think this makes sense... thinking again that IDB = disk, vuex = memory. So the FM would load from IDB into its own state, that seems fine.

Is the current limitation that a FM cannot create its own IDB store since FK core has already claimed it? Thus FK core needs to make it available?

[jgaehring]

Ack, sorry! It's confusing when they're both referred to as "stores", I should have been more clear.

Is the current limitation that a FM cannot create its own IDB store since FK core has already claimed it? Thus FK core needs to make it available?

Good question. And the answer is absolutely not, which is worth remembering, that FK core doesn't have to concern itself with that, if a FM wants to create its own database there's nothing stopping it, and perhaps core should stay out of that. I guess the question is, can core help manage that productively somehow, like by offering tools for interacting with IDB, which has a bad rep for its API, or coordinating between modules somehow. I guess that's the kind of thing I had in mind, but again, always good to have a specific use case in mind when developing something like that, and to hold off until you do.