search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

Update netcdf files with new acknowledgement #457

Closed ocehugo closed 4 years ago

ocehugo commented 4 years ago

The IMOS acknowledgement changed from :

Any users of IMOS data are required to clearly acknowledge the source of the material derived from IMOS in the format: "Data was sourced from the Integrated Marine Observing System (IMOS) - IMOS is a national collaborative research infrastructure, supported by the Australian Government." If relevant, also credit other organisations involved in collection of this particular datastream (as listed in 'credit' in the metadata record).

to:

Any users of IMOS data are required to clearly acknowledge the source of the material derived from IMOS in the format: "Data was sourced from Australia's Integrated Marine Observing System (IMOS) - IMOS is enabled by the National Collaborative Research Infrastructure Strategy (NCRIS). It is operated by a consortium of institutions as an unincorporated joint venture, with the University of Tasmania as Lead Agent." If relevant, also credit other organisations involved in collection of this particular datastream (as listed in 'credit' in the metadata record).

Requirements:

atkinsn commented 4 years ago

@ocehugo we might need to chat further about this. I am pretty sure in the past when the IMOS office has changed this acknowledgment we haven't gone back and reprocessed all files, we have just updated the toolbox so that all new files (and any reprocessed files) would have it. I understood this task for you to be just about updating the toolbox. Will involve @kereid

ocehugo commented 4 years ago

@atkinsn - ATM, my understanding is that updating historical files are required but low prioirty.The task to update the toolbox is already on the toolbox repo.

I created this so we may discuss if this is really required as you said and/or how should be done.

I also don't know if any other data/metadata need to be changed, so this was another reason to put this here.

PS: I think we should do it because otherwise, we risk someone using the old thing (say inspecting an old file). If updating all files is a bummer and/or risky, then we need to think on how to make this feasible and safe.

atkinsn commented 4 years ago

@ocehugo have just chatted to @kereid and we need to have a chat with @mhidas (re NetCDF checker) and @ggalibert about this.

mhidas commented 4 years ago

I was just about to reply, to point out that the IMOS checker plugin would also need to be updated to allow the new wording.

Ideally the IMOS NetCDF Conventions document should also be updated. (Not that anyone can see that - I just spent 15 min trying to find a link to it on the IMOS website, without any luck!)

As for updating all the historical files, I think it would be a whole lot of effort for very little gain.

kereid commented 4 years ago

@mhidas yes we also discussed updating the document can find it in our userguide https://help.aodn.org.au/public-documents/imos/ , but on the list for discussion I'll send a meeting request out later today for us to all have the chat - possibly Thursday

ocehugo commented 4 years ago

As for updating all the historical files, I think it would be a whole lot of effort for very little gain.

I completely agree! However, I'm wondering how many other things related to "historical" reprocessing (from our part) were also put on the same bucket (e.g. other attributes updates, auxiliary variables missing, etc). My point is: maybe it's time to make an effort to take the "effort" out of updating/reprocessing historical files.

ggalibert commented 4 years ago

I think Natalia is right, in regards to datasets our usual approach is not to go back and reprocess in order to update acknowledgements, but ensuring all new files reflect the most recent one.

maybe it's time to make an effort to take the "effort" out of updating/reprocessing historical files.

This would be relevant to consider in a new implementation plan for example! This reprocessing historical files issue is big and is touching a lot of components in our infrastructure, it would be good to see how this could fit in a new design that would take into account other big issues like archiving of raw data, central metadata repository, central calibration metadata, pre-processing and QC of moorings data by regional expert yet in a centralised infrastructure, etc... It's a lot of effort and we know we're already commit to some other projects for next year! So maybe not quite the time? I would close this particular issue about re-procesing files with old acknowledgement for now.