Queued Issues: Experiment Module Meta Data (Implementation Testing)

[aislinnpearson]

Following the FarmOS milestone update on 20 July 2023 with Ian and Helen the following were requested. I understand that the Statistics team also have features that need review.

MAJOR CHANGES

Must do

• (1) https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/373 Currently we only have four roles associated with the experiments: Sponsors, Research Viewers and Research Editors. Both the scientists and FFEC have asked that we change this, including ensuring that people (especially those external to Rothamsted) can only see their own experiments. Needs specifying by Aislinn before passing to Paul to build. Also includes #382. Marked as must do as this is core functionality
• (2) https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/506 When the proposal is approved, the sponsor gets an email with instructions on what to do next (possibly the data is pushed through at this point as requires setting up experiments/ designs/ plans?). See also #395 (duplicate issue)

Should do

• (1) https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/501 Move Reviewers data field to the bottom of the comments so that it is easy to ticked/ check once the proposal has been read. This should only be visible/ accessible to reviewers, who can only add their own name (requires creating a Reviewer role - see #373 - and ensuring Reviewers are set up as researchers). Marked as Should Do as this can be done manually this season, but programmatically next season.
• (2) https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/500 Proposals submitted in the Harpenden instance are occasionally moved to Brooms Barn. Ideally we would do this programmatically, maintaining a data trail to show that the proposal has been moved between FarmOS instances/ locations. Marked as Should Do as this can be done manually this season, but programmatically next season.
• (Internal) https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/394 Along with Stats, RDE and FFEC we need to review the way that experiments, experiment designs and study years are coded. The current system works well for classical and long term experiments, but not for annual trials such as WGIN which repeat every year but get a new code every year (meaning there is no way to link all the years associated with an experiment together). Marked as Should Do as there is already a working solution (#429)

Nice to have:

• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/502 Add a % reviewed bar to show how many people have reviewed a proposal, and visualise on submitted tab. Marked as Nice To Have as this has already been completed this season, and really it is just a nice visualisation (as opposed to core functionality that restricts or improves use of the system)

MODERATE CHANGES:

Must do:

• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/503 ~Add tabs for experiment status to proposal entity. Marked as Must Do because this is essential for trials planning.~
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/453 The system was set up so that the proposal could reference experiments, designs and plans, making it easy to see where the proposal was in development. This is useful when submitting a proposal, but people do not always go back to the proposal and reference an experiment after setting up a new one. Equally, these would ideally be reviewed every year. Marked as Must Do because of data integrity issue and time saving. Aislinn to discuss with Paul and Mike before specifying. See also #505
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/538 As for issue #453, but as visualisation relates to the plots
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/539 As for issue #453, but as visualisation relates to the experiment boundaries
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/507 ~All research module entities: filter by experiment category. Marked as Must Do because of the GM trials (see GE and GM classifications) - THIS IS DONE, REMOVE~
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/524 Requested by Jonah to make it easier to find proposals and move them through the process, as this is currently quite confusing and is resulting in mistakes and prolonged discussions on which form to fill in and how to use the software. Needs further discussion with FFEC/ Mike and Paul

Should do:

• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/485 ~If a user has left Rothamsted, the email alerts sent by FarmOS will bounce. Too many of these, and the FarmOS web address becomes black listed for security reasons. Ironically these bounced emails are useful (either for people who have left or email addresses with spelling mistakes). We agreed this could be partly resolved by allowing people to 'opt in' to alerts so the email field is no longer mandatory. Marked as Should Do because we have a workable solution, but this is the first issue that should be addressed after Must Do items for those data security reasons (and because the fix is not ideal)~

MINOR EDITS

Must do:

• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/504 ~Please can we add new status levels to help with the review and planning process~ resolved in 2.15.x~
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/508 ~Proposal entity: add Genetically Modified and Genetically Edited to the experiment categories~ resolved in 2.15.x
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/509 ~As above but for the experiment entity~ resolved in 2.15.x
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/490 ~Add drill spacing to the management options on the experiment design entity~ resolved in 2.15.x
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/510 ~Add drill spacing to the management options on the proposal entity~
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/491 ~Add organic amendments to the management tab on the design entity, under Nutrition~
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/511 ~As above but for the Proposal entity~
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/513 ~Proposal entity: Clarify the description for previous cropping~ resolved in 2.15.x
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/512 ~Proposal entity: Move previous cropping to Restrictions tab~ resolved in 2.15.x
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/519 ~Update the list of statistical design options, as it has become clear through user testing not all scenarios were covered by the initial specification.~ resolved in 2.15.x
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/520 ~Update the list of additional blocking constraints, as it has become clear through user testing not all scenarios were covered by the initial specification.~ resolved in 2.15.x~
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/523 ~On the design entity please can we rename 'Number of plots per block' as 'Number of main plots per block'. Requested by Suzanne in order to standardise the naming conventions. Marked as must do for that reason.~ resolved in 2.15.x~

Should do:

• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/454 Names and descriptions for several statistical questions need editing for clarity to make the system easier to use for people with only basic level stats training. Suggestions to come from Stats. Marked as Should Do as Stats are generally helping to fill in these data fields at the moment.

BUGS

• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/518 ~Bug reported by Mike~ resolved in 2.15.x
• • https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/527 ~Bug reported by Mike~
• Bug: Experiment Programs visualised multiple times #536 I came across a visualisation bug yesterday where the research programs seem to be repeated multiple times (see below) - possibly something to do with the number or experiment or proposal entities they are referenced by? From what I can tell it is only the Research Programs that has this problem.

For discussion/ review:

• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/514 For almost all the experiments the Farm will collect data at some point. This includes raw harvest data from the plot combine, moisture data from the Jenkinson or even calculated yields from Statistics. Helen has requested that this data does not need to be in FarmOS, but that it should be easy to find from FarmOS. Being able to put the data in FarmOS would also be very beneficial for commercial clients, who might not have access to our internal electronic research notebooks system.
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/516 The experiment module developed for FarmOS is unique in that all the variables and the plot level data is fully annotated, making it compatible with similar efforts to provide FAIR agricultural data such as Grassroots, BrAPI and GLTEN. However, the process of annotating the data could be simpler and more user friendly. This could come in phase III of the experiment module development.
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/372 it is increasingly becoming evident that we need a better process for documenting these types of errors.
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/526 Discuss what the plans should be called, update the name fields in FarmOS and determine what information goes on the plans and on the designs.

Quick forms:

• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/439 Currently the nutrition quick form doesn't work well across both sites/ all use cases resulting in some incorrect data being entered and/or not being entered. Needs to be re-assessed, with Paul suggesting this is a good example of configurable quick forms, but that is likely to be a larger issue (but perhaps worth discussing).
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/521 Request from North Wyke to pre-populate the fertiliser quick form, saving them 3-4 hours a week of data entry in busy times of the year. If useful, I also said to Josh we could look at doing this for the spraying quick form and improve the harvest (trailer and bale weights) quick form.
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/525 Requests from multiple farm teams to improve the harvest quick forms

Monitor:

General monitoring issues:

• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/448 We may want to restrict the types of locations that are displayed under 'Requested Field Locations' and 'Unsuitable Field Locations' so that only locations with the type 'Field' or 'Experiment Boundary' are selectable. Marked as for monitoring so we can see if this becomes an issue after a period of user testing.
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/324 The current experiment module is designed so that experiments have one of five possible status options: planning, active, complete, terminated, archived. Which stage the experiment is currently in should determine which data fields which must be provided at each stage - for example, an experiment which has been terminated must include a reason for failure. Discussing this with @paul121 there are Drupal modules which can implement this kind of logic but the initial user testing suggests this might not be worthwhile as it reduces flexibility (e.g. when the data isn't available for valid reasons, the experiment can't move through the pipeline and that results in a build up of paper records which don't get entered, risking the loss of data). Possibly better managed through internal processes.
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/540 Once we have resolved #277 and #377 we should consider how multiple related plots and experiment boundaries are viewed/ visualised so you can see the whole history (logs) associated with the plot and potentially a list of previous plots (although this list might get very long). Needs further consideration so marked as for monitoring/ review.

Monitoring issues that might be resolved (or partially resolved) by Templates:

Nice to have:

• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/71 Recording treatment applications can be difficult for experiments with a complex factor levels. For example, the variety trials can have several hundred different varieties drilled at the same time and this would require several hundred drilling quick forms. From the farm team’s perspective the easiest solution would be to create a single log where the activity (drilling, input, etc) is listed "as per plan" and then note any deviations at a block or plot level if mistakes occur. The current work around is to have 'experiment' listed as a variety, which solves the problem for variety trials, but not more complicated tasks such as the nutrient applications on Park Grass. Some elements such as LSRE management may be resolved with Templates, hence why a holding issue.
• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/489 For some of the long-term experiments, management aspects change (e.g. varieties). We were going to add multiple text boxes to document these, but it's a relatively big piece of development which might not be needed any more once the new templates are in place.

Aislinn to document:

• https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/324 We previously suggested that we could have different levels of required data that must be provided based on the status of the experiment. In reality this often holds up the trials team, especially when the scientists don't enter the data in time. Instead we agreed to keep this simple, perhaps only enforcing additional required data fields before archiving, and perhaps scope quick forms that will make this process easier to implement. Needs specifying by Aislinn before passing to Paul to build. Marked as must do as this is core functionality
• Paul also suggested building a dashboard view for which data pieces are missing. This could go in the data extraction module

Additional requests (not for this milestone but which came up during testing):

major development

• (3) https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/277 The current data model creates new plot assets for every cropping year. We need to address this issue so that it is clear we are always referencing the same plot year after year where the land area is the same (e.g. Broadbalk), but with enough flexibility that plots can move location for experiments which are not always drilled in the same (e.g. WGIN). Also relates to the ERN work on IGSNs (see https://www.igsn.org/)
• (4) https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/377 Enable researchers to duplicate existing entities, to save having to repeat information. Needs specifying by Aislinn before passing to Paul to build. Marked as should do as this only comes into effect in April/ May next year (but is also a core specification)
• (5) https://github.com/Rothamsted-Ecoinformatics/farm_rothamsted/issues/505 Once approved, push relevant information through to Experiment/ Design/ Plan. Needs discussing about feasibility before specifying by Aislinn and passing to Paul to build. See also #453

[aislinnpearson]

Mike working on CSV importers as part of the USDA grant - logs and assets, remove columns not using, etc.