search

BCODMO / webcomponent_data_viewer

Visualize a Frictionless Data Package
0 stars 1 forks source link

Make csv BCO-DMO provides a consistant csv #27

Open adyork opened 3 years ago

adyork commented 3 years ago

This isn't necessarily a datapackage viewer issue, but putting it here since this is the first time the datapackage csv is being provided directly.

And a reminder, I mentioned before I think we should consider how confusing it will be if we put out two separate csv products from BCO-DMO. I can see this making headaches for us down the road.

One from laminar and one from the csv download badge on the main page (ERDDAP). If they were the same csv I wouldn't think that was a problem but the csvs will have different contents.

I don't know if everyone is aware of this so I'm explaining the difference. Here are the two types: 1) The csv out of the datapackage. This is what the link in the data viewer is currently providing access to. One header line with column names. This is the exact version of the data the data managers dumped after Laminar. The format here matches the format we are entering in OSPREY (and what is shown in the datapackage viewer). image

2) The csv that is provided if you press "csv" download button on the current dataset landing pages for datasets that are in ERDDAP. There are two header lines, one for column names, the second row is units. Values in this dataset are changed in various ways when coming out of ERDDAP. For example all datetime columns will have values changed to add 00 padding until it gets to second resolution regardless of whether the format we are saying has them. This format no longer matches the formats we are entering into OSPREY. image

Also note that the missing data ids will vary between these two .csv files. The csv filenames differ too which is good i guess so they can't be mistaken for the same.

Example: Dataset landing page with csv download badge https://www.bco-dmo.org/dataset/813166 Data view with csv download link to csv in datapackage https://www.bco-dmo.org/dataset/813166/data/view

CSVs differ. datapackage csv porewater.csv image

ERDDAP csv (out of landing page) bcodmo_dataset_813166_712b_5843_9069.csv image

adyork commented 3 years ago

Also, injecting some personal feelings that may be irrelevant. But I really hate that ERDDAP changes values. I also really hate fake time resolution because it messes up plotting in synthesis projects.

ashepherd commented 3 years ago

discussed this with Danie, and we are OK with ERDDAP doing what it does, and we will communicate this from the new website that using ERDDAP file formats aren't producing isomorphic versions of DOI'd data file.