Implement dataset upload

[eboileau]

Aims/objectives.

Dataset can now be imported via add-dataset. We need an upload form that, on submission, instantiates a DataService.

This form is ultimately only accessible after login, but I think it is a good idea to implement it as soon as possible, and handle route guards in a smart way before #43 is completed. This will help us to revisit services EUFImporter, ProjectService, and DataService (and maintenance scripts).

A clear and concise description of todo items.

• [ ] Sort #31, #35 (see also #41)
• [ ] Add upload form and validation (VueJS) - mock submit
• [ ] Handle POST request (create new dataset) - test via cURL
• [ ] Set-up routes and route guards
• [ ] Submit and create dataset via form

---

• [ ] Exception handling during dataset creation/import is not consistent (logging, warning, error, etc.).
• [x] Data import: standardize chrom field (chr), validate itemRgb, strand +/-, refBase ACGTN. (or none, see below), and what about coverage/frequency? Cast types.
• [x] How does header reading works if e.g. there is a blank line, or a line change \n in some field e.g. experiment, more or less header fields, or if the order is not the same as expected (see below)?
• [x] How to consistently and robustly handle suspected discrepancies at upload (form fields vs. header, form fields vs. data records) to avoid DB transactions?

[eboileau]

We need to deal with "all uploads" consistently, and eventually data cleaning, cf. Compare View - data upload. For the upload, we opted for a quick solution: ad hoc importer for BED6/11/12, read in file for each query, return as records. In the longer-term, we need something more robust (harmonize upload classes).

For the Compare View upload, MIME types are not sufficient, we need to add .bed,.bedrmod, but nevertheless it is recognized in sever-side as application/vnd.realvnc.bed. Does this matter?

[eboileau]

After receiving data for #46, I reckon that EUFImporter has to be modified to allow minor variations against the specs:

• [x] Allow for optional 12th column, unless bedRMod specs are updated (in which case we still have to generalize the importer).
• [x] Handle extra header lines and, possibly, empty header column description (chrom, etc.)
• [x] Allow None, Na, NaN, ., as string, variations thereof, etc instead of strictly "empty" in optional header fields, then cast to None.
• [x] Although annotation_source and annotation_version are required in the specs, we don't care, so also allow None, etc.

[eboileau]

2023-12-15 now EUF v1.7, see #49 .

As for handling "optional header fields" and annotation_source and annotation_version, I don't know. In principle we don't care about what value they have, if any, so maybe we don't need to bother whether we cast them to None or just leave them as they were given, even if this is None, Na, etc. When writing to EUF, we should overwrite annotation_source and annotation_version with that from the DB.

chrom field standardization is there, subject to change.

TODO: revisit services/importer.py (EUFImporter, BEDImporter)