API Design

[MarkKoz]

Here are my notes for what we need for the API. Sorry this is a bit messy. It's hard to visualise the design at such an early state, so this is more of a rough draft to give us an idea of the direction to go in. I think the appropriate designs will make themselves evident once we start implementing the API. We could try to go all out and figure it out now, since now is the easiest time to make changes. However, it's also more difficult to plan without anything done in front of us.

1. Get registered devices of a user
   • Contains the device name and its ID.
2. Get parameter/parameters for a registered device, returning the following data for each parameter:
   • Name
   • Description (optional)
   • Value
   • Unit name
   • Manual/auto flag
3. Get historic values for a parameter of a particular registered device.
   • Specify a datetime range for the query.
   • Only return data. Other parameter info (name, units, etc.) should be retrieved via № 2.
   • It makes more sense to request data for each parameter separately, as each will get its own plot.
   • Can also be used to show the delta for each parameter under "recent parameters".
4. Register a device for a user.
   • It needs to take a device ID and the its name.
   • Might need some more input, but need to figure out details of AWS IoT integration first.
5. Add a new parameter to a registered device, which needs the following info:
   • Parameter Name
   • Description (optional)
   • Unit name
   • Maybe a data type, so some validation can be performed e.g. only allow numeric inputs, non-negative, etc.
   • Unclear how automatic parameters and tests will be added.
6. Edit parameter of a device
   • Change the name, description, unit name, and associated tests.
   • Can't change data type, if we choose to support that, since migrating historic data may be ambiguous.
   • Maybe only allow editing of manually added parameters?
7. Get test/tests for a device
   • Test name
   • Description (optional)
   • Test category? (mockup shows a "sensor" and a "nutrient colorimeter" category) Would need to also figure out how to add categories then.
   • Most recent status (running, failed, succeeded)
     • Might want to use SSE or WebSocket to update the status on the frontend as a test is running. Alternative is to just poll this endpoint.
   • Date & time of next scheduled run
   • Scheduling interval (e.g. every day, every 3 hours, etc.)
   • Manual/auto flag
   • Notification settings (e.g. notify via email, via push, both, never, etc.)
   • Number representing its order for running.
   • Associated parameters
8. Create a test for a device
   • Test name
   • Description (optional)
   • Test category?
   • Scheduling interval
   • Notification settings
   • Order number
     • Unclear if these should be unique, or if allowing the same order for multiple tests could mean that they run simultaneously.
9. Run test, given its ID.
   • Should tests be unique to a device, or should multiple devices be able to share the same tests?
10. Get test history
    • Return date & time of a test run along with its status
11. Get/Set tasks
    • Schedule feature is unclear.
    • Tests could be implemented as a specialised task.
    • Should have
      • Name
      • Description (optional)
      • Date & time of next scheduled run
      • Scheduling interval (e.g. every day, every 3 hours, etc.)
      • Number representing its order for execution.
      • Most recent status (in-progress/running, failed, succeeded)
12. Get/set global notification settings for user
    • Set defaults (e.g. default to notify via email)
    • Global toggle switch for all notifications
13. Remove parameter
14. Remove device
15. Remove test
16. Remove task

I'm not accounting for notifications too well. The feature isn't fleshed out yet and it may prove to be complex, thus making the way notifications are handled above not ideal. For example, there needs to be a way to set thresholds for notifications (e.g. param value too high or low).

Also, the way tests are associated with parameters needs to be cleared up. I imagine it would likely make the most sense for there to be a one-to-one relationship between a test and a parameter.

[MarkKoz]

How will the backend know how to trigger automatic tests? It will need to communicate with the hardware to trigger it, and the hardware will need to understand what it's being asked to do when e.g. the backend requests for "test #3" to run. But how will the hardware know to recognise these tests? One idea may be for the backend to be able to request supported actions from the hardware, and present those to the user so that a test can be created around a specific action (or maybe even more than one action, depending on how fine-grained it is).

Should the backend assume that all systems registered will have identical sensors, or could it vary among systems? If the hardware can report which actions it supports as mentioned above, then the backend wouldn't have to make any assumptions.

That being said, it's not clear if the hardware will be able to also listen for requests or if it will only be able to send requests to our backend.

---

Should it be possible to run different tests in parallel, or should they always run in series? Running in parallel might have speed advantages, but it may not be significant depending on what a realistic test interval is. For example, if the lowest reasonable interval we foresee is 1 hour between test runs, then it doesn't really matter if tests take 2 minutes vs. 6 minutes.

The disadvantage of parallel execution is that some tests might interfere with each other if they run at the same time. We'd need to either elect to allow users to configure things in a bad way, or implement a system which specifies which test combinations shouldn't run in parallel.

---

Do we need to support the scenario, if it's even possible, where a test can be triggered automatically, but the user has to enter some or all of the resulting data manually?

---

Should we support data validation for manual parameters? This would involve specifying a type for the parameter e.g. integer, float, string and maybe even a valid range or list of choices. Not exactly sure yet how this would be added to the DB schema. If we don't enforce data type homogeneity, then we'll probably run into trouble when generating plots of historic data for parameters.

---

Right now, I am planning to support multiple parameters per test. Is this necessary, or should it be simplified to only 1 parameter per test?

[PurplHays]

I'll just be addressing my thoughts on the second comment right now.

Triggering tests can be done a number of ways. The arduino can receive messages and run functions if listening for specific keywords. Like the temp sensor can be sent 'config' (I believe was the keyword) and will run the function tied to that. I'll have to learn the capabilities and syntax exactly but doable. So we can have a function that will respond with the capabilities of the device. For our project the systems will have set sensors with no changes, but scalability is usually best practice so calling for capability would be the correct way I feel.

Since the arduino has a single processor it has to run the tests in series but the tests only take seconds for retrieving like 40 values and averaging them out for more stable readings. There may be some latency but speed shouldn't be much of a problem, we more so have to protect the arduino to complete a test before receiving another request I feel. Something like a queue to do one after the other if multiple requests are made. At what layer that is implemented, I'm not too sure, seems like preference.

Manual entry and automatic tests should be completely separate. The users should just be able to remove bad data from automatic tests in case the sensors are outputting bad readings (eg. sensor wasn't submerged correctly). The automatic tests should just get a single reading from the hardware and we generate timestamps for them in the backend (I'd assume) because the hardware runs on it's own clock.

Data validation for manual parameters is probably for the best, especially as you mentioned for generating plots. We'll probably need to have the user select from parameter type options when creating a new manual parameter. I'm not sure if we need data types outside of floats though, other than for user convenience but I can't think of examples that couldn't be done with a number and labeled parameter.

As I mentioned above, the hardware should only respond with an averaged value (or binary in the case of the water level). We would generate a timestamp to be able to compare between tests, not sure if other values would be needed for features of the website but I think those two are it.

[MarkKoz]

but scalability is usually best practice so calling for capability would be the correct way I feel.

Yeah my concern is that it may end up being a situation where we have to design some secondary API, albeit a very minimal one. But it would be a nicer user-experience for the device to be able to self-report what it supports upon being registered. That way the backend is decoupled from the hardware and can support any action the device can in a generic way.

Since the arduino has a single processor it has to run the tests in series but the tests only take seconds for retrieving like 40 values and averaging them out for more stable readings.

Aren't most tasks I/O bound though? It could request multiple sensors to start and wait for them to finish asynchronously.

Something like a queue to do one after the other if multiple requests are made. At what layer that is implemented, I'm not too sure, seems like preference.

I think it'd make more sense for the hardware to manage that. If the backend does then it seems like it'd be too tightly coupled.

I'm not sure if we need data types outside of floats though

You don't think there'll be a situation where a word i.e. a string is more appropriate? I'm not familiar with what kind of data the sensors deal with. If it's all numeric then we can make all fields floats and not worry about having to store data types in the DB.