search

ImperialCollegeLondon / paricia

Hydroclimatic data collection and information system
https://imperialcollegelondon.github.io/paricia/
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Make the data import process more explicit #373

Open dalonsoa opened 3 days ago

dalonsoa commented 3 days ago

At the moment, when importing data the user selects a Station and a Format. From the user perspective, this only selects some file related settings (like extension and delimiter) and how to deal with the date and time columns. Implicitly, however, the user is picking all the variables that are related to the chosen format via a Classification object, but there's no way for them to know what these variables are or what columns they are using except by walking through all of the Classifications.

It will make way more sense to have the Classification and the Format related via a ManyToMany field in the Format object, rather than a ForeignKey to Format in the Classification object. This way, when a user opens a format, they will see exactly what variables will be pulled from the data file and from where.

This requires some changes to the models, obviously, and to the code to parse the data file, but, specially, the main complication comes with the views and templates used to display the Format objects, which will need to be more complicated as they will need to show a list of all the Classifications related to a Format. Nothing that we have not done in the past - see for example the DeviceSpecification model in Liionsden - but nevertheless, a not straight forward process. I think it will be worth the effort from the point of view of the user experience.

dalonsoa commented 2 days ago

@ICHydro , @tsmbland , I've open his issue to discuss/tackle the data ingestion process, which is not really such a good user experience. Any thoughts are most welcomed.