search

Rothamsted-Ecoinformatics / farm_rothamsted

Custom farmOS features for Rothamsted Research.
GNU General Public License v2.0
5 stars 1 forks source link

data extraction: standardise column names for log exports #649

Open aislinnpearson opened 4 months ago

aislinnpearson commented 4 months ago

When I downloading the logs from the classical experiments so Tony could compile the yield books, I noticed that where we (Rothamsted) have extended the underlying data model for the logs, the naming of the columns is inconsistent.

The data fields I noticed it for were mainly PPE, COSHH and seed dressings where all three of these things would be amalgamated into the same two columns. There were also examples of additional rows for lot number, method and purchase source in some exports but not others, which were always empty columns and which also I wasn’t sure about. I’ve attached two examples here to demonstrate what I mean.

paul121 commented 4 months ago

Drat. This is a known issue I had forgotten about. https://github.com/farmOS/farmOS/issues/805

Headers are generated for the first entity in the export list. When exporting many assets/logs of different bundles this means some entities may have different fields than others. It is indeed a bug that the resulting export may have incorrect column names for some of the row data.

For example, Harvest logs have a "Storage location" field, while other Input logs have "COSH" and "PPE" fields. When exporting a CSV of both Harvest and Input logs, these data fields/headers are not consistent with row data in the CSV exports.