Closed gothub closed 5 years ago
Sounds like a tough check to get right from the user's perspective. However, I think a really good approach would be to not find or create a list of non-proprietary formats since there are so many but to instead try to find common proprietary formats such as MS Excel and MS Access files and fail the check of those are found.
When the check passes, the message could indicate that the passing grade isn't the same thing as "you have not used any proprietary formats". What do you think?
Sounds reasonable. Finding the list of proprietary formats may just mean assembling it ourselves.
Yeah, maybe the NCEAS Data Team would be able to scrap together a short list of what they usually deal with and us developers could fill in any others they don't have much contact with.
We could start by going through the DataONE formatId list and marking each format ass proprietary or open or unknown/mixed. That should cover all of the files we have in DataONE.
Here is a list of formats from http://cn.dataone.org/cn/v2/query/solr/?q=formatType:DATA&facet=true&facet.field=formatId&facet.mincount=1&rows=0
Please mark formats that are proprietary:
Note that some of those are general types (e.g., application/octet-stream, text/xml) that may or may not be used for defining proprietary application types. We really should be more specifically typing those when possible.
Also, Matlab's most recent format is an HDF5 file format, and so is technically not proprietary, but their earlier .mat formats were proprietary, so it varies by version for Matlab. HDF5 seems to be missing from the list.
Also, the full list of registered formats is here:
(@mbjones I added a ticket for adding HDF 4 and 5 to DataONE's Object formats list.)
Thanks. And I note that the object format list has 5 formats for Matlab, only one of which would be non-proprietary (application/MATLAB-v7.3
).
<objectFormat>
<formatId>application/MATLAB</formatId>
<formatName>MATLAB programming language script</formatName>
<formatType>DATA</formatType>
<mediaType name="text/x-matlab"/>
<extension>m</extension>
</objectFormat>
<objectFormat>
<formatId>application/MATLAB-v7.3</formatId>
<formatName>Mathworks MATLAB version 7.3 (R2006b or later) binary file - HDF5 compatible</formatName>
<formatType>DATA</formatType>
<mediaType name="application/octet-stream"/>
<extension>mat</extension>
</objectFormat>
<objectFormat>
<formatId>application/MATLAB-v7</formatId>
<formatName>Mathworks MATLAB version 7 (R14 or later) binary file</formatName>
<formatType>DATA</formatType>
<mediaType name="application/octet-stream"/>
<extension>mat</extension>
</objectFormat>
<objectFormat>
<formatId>application/MATLAB-v6</formatId>
<formatName>Mathworks MATLAB version 6 (R8 or later) binary file</formatName>
<formatType>DATA</formatType>
<mediaType name="application/octet-stream"/>
<extension>mat</extension>
</objectFormat>
<objectFormat>
<formatId>application/MATLAB-v4</formatId>
<formatName>Mathworks MATLAB version 4 binary file</formatName>
<formatType>DATA</formatType>
<mediaType name="application/octet-stream"/>
<extension>mat</extension>
</objectFormat>
Here is the full list from DataONE formats service:
I marked Raw image files as proprietary. See https://en.wikipedia.org/wiki/Raw_image_format. However, they often contain critical data and metadata that are lost when the images are converted to open formats. Until a universal open Raw format is available, its probably still beneficial for us to capture raw images in addition to open versions, and so I am not sure we should penalize this in the check. Or maybe we do so and just recognize that there's no good alternative right now, as is similarly true with other binary sensor data formats.
Yeah, not sure, but if we flag (FAIL) formats that are not directly usable (without a conversion by the end user) then there is an incentive for the data creator to provide a more usable format.
This check is discussed in issue https://github.com/NCEAS/metadig-checks/issues/64
For the check 'NonProprietaryEntittyFormat', what lists should be used for
proprietary
andnon-proprietary
formats?