Data Extraction: Data archiving processes

[aislinnpearson]

As described in the document on https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/70#issue-1093250757:

"There needs to be a process for checking the data entered before archiving. This could be done once a year, but ideally more regularly where possible. This process needs to be thought through."

This also relies on the data being entered correctly in the first place, and it is worth considering how FarmOS could help to standardise this process (perhaps by adding crop specific GANTT charts to plans?)

Duplicated for #275

[paul121]

One thing to identify here is what "checking the data" means - there is lots of different data here! Logs, assets, plan meta data, etc.

One idea that comes to mind is adding additional state(s) to assets and/or logs to facilitate this review process. An asset could go from "Active -> Needs review -> Reviewed -> Archived".

[aislinnpearson]

One idea that comes to mind is adding additional state(s) to assets and/or logs to facilitate this review process. An asset could go from "Active -> Needs review -> Reviewed -> Archived".

This is kind of what I was thinking with the "Job Status" question on the quick forms (#39 #40 and #41 all have reference to this issue I think). Open to ideas on better ways of implementing it though...

[paul121]

(below is copied from @aislinnpearson)

As per #40 (comment) the review process requires records to be tagged depending on where they are in the review process. This tag should also be set initially via the quick form.

In the quick forms the tags are specified as:

Pending -> Scheduled -> Field Operation Partially Complete -> Field Operation Complete (Awaiting Data Entry) -> Task Complete -> Record Approved

However, this workflow relies on the concept of "editable" quick forms (#95) so until we have funding for Phase II, I think the suggestion from @paul121 is better (with a few minor edits for clarity).

Active Job -> For review -> Reviewed -> Archived".

As an aside, this then needs to be integrated with the Farm workflow, with the farm and trials managers needing to keep on top of the records which need to be reviewed.

[aislinnpearson]

For information:

This issue is also discussed in #79 based on the user feedback in #70. As described in the document attached to #70 (comment):

"When the scientists are told a job has been done but the records haven't yet been entered into FarmOS, they get anxious and end up relying on phone calls and e-mails with the farm staff in an attempt to understand what was done and when. This is obviously only a problem where farm staff are relying on paper records and don’t have time to do the data entry, but there are valid reasons why both systems need to be taken into consideration. Another important point is the restrictions: if the spraying data is entered a day or two after the field application, then the restricted areas function is useless. This is common – especially with spraying."