Models/services infrastructure planning

[mhauru]

We had a chat on 2024-01-11 about what the user experience would be like for configuring models. It also seemed like running ingress, and maybe some other things as well, would benefit from similar user actions being available, so we are more broadly calling these things services, as Pat suggested already months ago.

The simplest version we would go and implement would look like this:

• There's a page with all services. E.g. "Rothamsted data ingress", "GES", "Arima".
• Each service has a page where one can enter a schedule on which this service will be run, and hit schedule.
• Alternatively/in addition on that page there is a button for "Run now". Start by implementing either scheduling or "Run now", whichever seems easiest.
• That page also has a text box where the user can write JSON, that will be passed to the service as an argument. This is used for e.g. setting model parameters. Maybe not in the first iteration, but eventually, this JSON will be validated against a schema for each service.
• There's also a list of current schedules, and an option to edit their arguments.

In general, a service is a URL to which the backend sends a request to do a thing, together with arguments. The service then either returns something useful over HTTP, or in turn calls some backend end points to e.g. insert data.

In the first instance all our services will probably be Azure functions, and the scheduling mechanism can start out being bespoke for them.

In the future we might also directly display the output of a service run on the service page. In the first version you'll have to navigate elsewhere on the site to see the outcome, e.g. new data or model results. We might also later add a prettier version to set service arguments, prettier than a pure JSON block.

For the backend side of this, I think each service would need to have in the database at least

• name
• url
• JSON schema for arguments
• current schedules

I'm a bit unclear still on how the backend will make sure that the services actually get run on time. Do we run something like cron internally? For now might be good enough to set some parameters on Azure function apps.

[EdwinB12]

184 - This issue has a more detailed discussion of what the backend does to allow these features.

[mhauru]

The minimal version of this now exists: On the front end you can create a service, give it a JSON payload, and click "Run". One can also see a history of runs and the responses to their HTTP queries.

Missing are:

• Scheduling. See #193 and #194.
• JSON schemas for service parameters.
• Handling responses from services in a meaningful way. Currently we just log that the service returns a status code and a message. Any actual data that the service wants to send us has to go via the service making backend API calls. BaseService makes doing those calls easy though. Ideally we would have for instance ways of seeing whether a service has finished running, and having the outputs data in the database linked to the service run.
• Preprocessing data in the backend before sending it to the service, and sending it as part of the payload. Currently payload parameters are fixed rather than read from the database, and if the service needs data from the backend it needs to get it through an API call. BaseService makes doing those calls easyish though.