search

cidgoh / DataHarmonizer

A standardized browser-based spreadsheet editor and validator that can be run offline and locally, and which includes templates for SARS-CoV-2 and Monkeypox sampling data. This project, created by the Centre for Infectious Disease Genomics and One Health (CIDGOH), at Simon Fraser University, is now an open-source collaboration with contributions from the National Microbiome Data Collaborative (NMDC), the LinkML development team, and others.
MIT License
94 stars 26 forks source link

"Dates" collected by DataHarmonizer are datetime objects #389

Closed turbomam closed 1 year ago

turbomam commented 1 year ago

see

turbomam commented 1 year ago

And when presented with a slot whose LinkML range is datetime, DH does not provide a datepicker

ddooley commented 1 year ago

Hmm! It is important to sort this out! I'm looking at one of our LinkML specifications for "sample collection date" which is marked as range "date", and DH is providing the popup datapicker, and saving only as "YYYY-MM-DD", no timezone.

So can I see a slot definition for one of your date fields?

Now I wouldn't be surprised if DH is using the "date" widget for managing datetime fields, but I'll have to double check that. I do recall the issue that DH wasn't showing a time picker for datetime fields, but rather was just defaulting to midnight local time. That needs to be improved.

ddooley commented 1 year ago

I see DH has a parser in /lib/utils/validation.js (and UI calendar control) for "date" (xsd:date), but doesn't have one for "time" (xsd:time) and "datetime" (xsd:datetime) fields. It doesn't supply calendar widget for datetime.

turbomam commented 1 year ago

@pkalita-lbl DH saves the date-ranged values as YYYY-MM-DD strings at lest in TSV and maybe all other formats. It looks like it's only the JSON output that casts it to a datetime

pkalita-lbl commented 1 year ago

Okay I think we're talking about two totally separate things here which is making this issue confusing.

1) How should DataHarmonizer map types to input controls? Handsontable provides a date cell type. We currently utilize that when the range is date. Handsontable also provides a time cell type which we could use when the range is time. I'm not seeing any way to tell Handsontable to show a combined date + time cell type without resorting to custom cell types. So maybe we're out of luck there. 2) When DataHarmonizer exports to JSON it's doing a totally naive serialization. This causes Date objects to be serialized as ISO formatted strings. However this isn't compatible with the date format that JSON Schema (which underpins most LinkML validation tools outside of DataHarmonizer) expects.

2 is just a fixable bug and I can use this issue to track fixing it. 1 is more of an open-ended discussion and we should open a different issue if we want to talk about that.

turbomam commented 1 year ago

I shouldn't have implied that implementing a date+time widget is important to me.

Thing 2, saving values from a date-range column in JSON so that they validate against LinkML JSON schema, is much more important.

ddooley commented 1 year ago

I have created a separate issue for having dh implement a time and a datetime datatype: https://github.com/cidgoh/DataHarmonizer/issues/391