search

farmOS / field-kit

A modular, offline-first companion app to farmOS.
https://farmOS.org
GNU General Public License v3.0
60 stars 39 forks source link

Server-side configuration of field modules #287

Closed mstenta closed 2 years ago

mstenta commented 4 years ago

We've identified some use-cases where it will be necessary to have server-side configuration for field modules, so that farm admins can configure how field modules work in field kit. The configuration forms can be built in Drupal using Form API and Drupal variables/tables, served as JSON to Field Kit, and stored in Field Kit for offline reference.

jgaehring commented 4 years ago

Another consideration Mike and I discussed today is that the configuration for each field module will need to be retrieved from the server, cached in IndexedDB, and updated periodically. Ideally this would not require a lot of consideration from the module developer but would be handled by Field Kit Core.

The best solution I can come up with is to provide some sort of getter that any module can call to get its own settings. Behind the scenes, Field Kit Core will check the server for updates whenever this getter is fired; if it can be reached successfully over the network, it will return these values, and also cache the config in a special IDB store, just for that module, so that if later the server cannot be reached, it has a local copy of the last good settings.

The configuration for each field module would live at a unique endpoint, and would only contain that modules config. Field Kit Core would get these endpoints from the /farm.json endpoint, the same way it gets the URL for the module's script. This way, modules could update their own config w/o necessarily fetching all other modules' confgs as well, which might become quite large. It still might be desirable to do such large batch fetches from time to time as well, but at least this way, with separate endpoints, we have the choice.

mstenta commented 4 years ago

Yes! Great summary.

And I want to emphasize this:

It still might be desirable to do such large batch fetches from time to time as well

We discussed firing the getter function when the field module is loaded. Also firing it periodically for all field modules will ensure that settings are available even if you haven't loaded the module until you are offline.

paul121 commented 4 years ago

Another consideration Mike and I discussed today is that the configuration for each field module will need to be retrieved from the server, cached in IndexedDB, and updated periodically. Ideally this would not require a lot of consideration from the module developer but would be handled by Field Kit Core.

This is a great consideration!!

I'm thinking.. obviously we want to minimize the time between a configuration change on the server and when that config is known by the client. Could the server have some kind of a state for the Field Module config? The state could be a simple timestamp or generated string of characters. When a configuration is changed, the state would change. The client could be notified of this two ways...

It still might be desirable to do such large batch fetches from time to time as well

Regardless how the state is implemented, I think this could decrease how often large batch fetches would be made. Rather than doing a batch fetch "every hour" (as an example), a request to farm.json could be made every hour, and perform the batch update if a change has been made.

I think this could also be used as a way of notifying when new field modules are installed/updated. If FK is open when a FM is added to the server, the new (or modified) FM could be pulled in "on the fly" (not sure if this is possible?) OR at the minimum, a notification of some sort could be shown saying "New field modules are available. Logout/login to refresh"

At first I was thinking the event that FK is open when a new FM is added would be unlikely, but because the PWA could be running in a minimized browser or offline for a few days, this could happen fairly often. Also, if a farmer adds the FM to farmOS, seeing a notification to "restart" FK would be a good indication of it's success (if the FM isn't just added automatically!)

jgaehring commented 4 years ago

To really minimize time FK configuration is "out of sync" the state could be returned with every API request.

Hm, so literally every API request, whether it was for a log, or a taxonomy term, or for farm.json, would have somewhere in the JSON response this state value? What exactly would we encode in that value then? Just the last time that any field module has changed, or would it be possible to encode that a specific field module has changed? I guess I'm a little fuzzy on what all would be included in this state, if it's some sort of string/timestamp/hash, or if it's an object, how it's structured.

Also, I'm not sure if this is strictly relevant, but you might find this talk about "logical clocks" interesting: https://www.dotconferences.com/2019/12/james-long-crdts-for-mortals

paul121 commented 4 years ago

@jgaehring woah thanks for sharing that video! Connected more of the dots surrounding CRDTs for me especially concerning the actual implementations. The "logical clocks" makes a lot of sense in that context!!

What I'm thinking of is a bit more simple, although it's certainly related because we are working local-first :smiley: Because we won't be modifying the config for a field module locally (as I understand) there shouldn't be a need to handle conflicts. In terms on the local app, we just want to know when there is a change on the server regarding field module configuration (or possibly changes to the field module itself & knowledge of new field module installation). Consider this a sub-set of local-first problem where the config for the local app is only modified on the server, but needs to be kept up to date locally.

Yea, state could simply be a timestamp generated on the server anytime a change to a field module config is made. The first time the client starts it would save this timestamp. Then, rather than fetching the configs for each field module every hour, it could simply load farm.json and check the last_updated timestamp key under the field_modules object. If the timestamp is greater than the one saved locally, we know a change has been made, and all of the field module configs could be fetched.

would it be possible to encode that a specific field module has changed?

I had been thinking of this as a "global" state, but each field module could have it's own state too. This would be most useful for pulling in changes to a field module itself, which would probably be best handled by versioning the field module code? I think the "global" state would be sufficient for tracking field module configuration because it would likely be a small request to pull in the configuration for all field modules (but each field module could have a state tracking it's configuration if we wanted that granularity)

Hm, so literally every API request, whether it was for a log, or a taxonomy term, or for farm.json, would have somewhere in the JSON response this state value?

This is the most "extreme" - the alternative would be to return the state in a request to farm.json that could be run on a periodic schedule

jgaehring commented 4 years ago

Cool! I like this idea. I wonder if it's worth considering other application state as well, and modelling that as a whole. I could see this being useful for a lot of the data we're storing on Field Kit. Sort of like a global pulse for the whole system that keeps track on a high level of when certain entities have changed, and if so, directs any client to where the change has been made so it can update itself.

jgaehring commented 4 years ago

Just found another good article on logical clocks, which refers to the above talk:

https://jaredforsyth.com/posts/hybrid-logical-clocks/

jgaehring commented 4 years ago

In an effort to test script caching (#310), I ran a full web build of the client and served it up at http://localhost:8887 via Chrome Web Server. I did this rather than using the Webpack devServer so the necessary service worker would be included, unlike in the regular dev environment.

This created a bug that wasn't caught before, because of Webpack's proxy server prevented it from becoming an issue. Here's the error from the console:

Access to script at 'http://localhost/sites/all/modules/farm_precipitation/dist/FieldModule/Precipitation/index.js' from origin 'http://localhost:8887' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Basically the scripts aren't getting the necessary CORS headers that are being applied to other http responses, like the JSON we're getting via restws. @mstenta, any idea why the JS files aren't getting the same headers?

mstenta commented 4 years ago

Copying this from chat with @jgaehring:

the farm_access module adds those headers to any requests that DRUPAL serves But in the case of the JS files, Apache is serving them directly

So there are two options:

  1. We add headers via Apache
  2. We serve the JS files through Drupal

I think 2 is the best choice, because it doesn't add any requirements on the server/hosting layer

In the farm_client module, we need to provide a route (eg: client/js/*) which is a PHP page callback function that calls hook_farm_client_module_info to figure out the actual location of the JS file, and then output the content of it So... on the bight side... this would also abstract that "contract" of where the JS files live By always serving them through the same path pattern in Drupal... but in the actual module code files they could be anywhere (as long as the module declares that somehow so farm_client can find it)

I can take a quick stab at that...

mstenta commented 4 years ago

Done! I set it up to serve the field module JS file at /farm/client/js/[module-name]/index.js

This is deployed to test.farmos.net now with the farm_precipitation module, resulting in: https://test.farmos.net/farm/client/js/precipitation/index.js

jgaehring commented 4 years ago

Woohoo!! Thanks, @mstenta!!

:rocket:

mstenta commented 4 years ago

FYI @jgaehring I merged part of the farm_client branch into farmOS (this part: https://github.com/farmOS/farmOS-client/issues/311#issuecomment-626810786), but not the "field module" specific stuff yet.

I moved that to a new field_modules branch in my fork: https://github.com/farmOS/farmOS/compare/7.x-1.x...mstenta:field_modules

So update your local dev server to use that branch moving forward.

jgaehring commented 2 years ago

I'm going to close this, because it was initially opened when we were implementing field modules in 1.x, so a lot of the details discussed above aren't really relevant to what remains ahead for field modules in 2.x. I was leaving it open b/c I think we still need to keep server configuration in mind as we move closer to the release of Field Modules, which I would like to be simultaneous with the FK 2.0.0 at this point. Perhaps we can do a quick assessment of remaining issues while addressing #468.