Closed jeffbaumes closed 3 years ago
@wdduncan I assigned you and @jbeezley to this one. When ingesting annotations, we need to make sure the schema validates only valid ids (e.g. ENVO, KEGG, etc.) and I believe some regexes on the schema would help here. Could you help us figure out where to add these?
Thanks @jeffbaumes
Can you post some examples of the bad data?
I believe there was an issue where function IDs sometimes came through with prefix KO:
and sometimes came through with prefix KEGG_ORTHOLOGY:
. A regex in schema validation could catch things like this. Basically if any ID should match a regex we should put a regex in the schema. @jbeezley may have further thoughts here.
Didn't see the KEGG case, but this is the same as the ENVO case in #294 - @wdduncan The request here is to identify fields for which the raw value must conform to a regex and then do that validation. Looks like JSON Schema can do that ( https://json-schema.org/understanding-json-schema/reference/regular_expressions.html ) but that doesn't mean that the JSON Schema that gets generated in NMDC can include that. Ideas Bill?
Moving to May sprint per Bill's request and adding Large size label.
Comment from Bill:
I need to investigate how to use regular expressions in jsonschema to validate data. So, I will give an estimate of one week, but the issue is quite open ended. The only use case of KEGG_ORTHOLOGY. I imagine there are other use cases.
Sorry just saw this now.
There are different levels of checks
The first is very easy to do with using the existing id_prefixes
annotated in the schema
The second we can do by adding additional regexes (slot_usage)
The 3rd is more difficult, a few approaches:
But we can tackle this incrementally, start with 1 then 2
@cmungall and @wdduncan what would you like to do with this issue? Can we close it as the considering is done? And open a new ticket for June to start implementing per Chris's comment?
Let's make one ticket for 1+2, another for 3
Created two GH issues for June, assigned to @cmungall @wdduncan and @turbomam. Please remove wrong assignments if any or let me know.
https://github.com/microbiomedata/nmdc-metadata/issues/360 https://github.com/microbiomedata/nmdc-metadata/issues/362
Closing this issue.
Hello, all. You may have seen that this was moved to https://github.com/microbiomedata/nmdc-schema/issues/69
@jeffbaumes , I don't think you and I have had much on-on-one conversation. I joined @cmungall's group mid March.
Can someone share some JSON objects that contain some of the error patterns mentioned in this original issue post?
I have reviewed the declared orthology prefixes, and I have added acceptable patterns for the local part of their IDs in
which corresponds to
I can continue to add local portion patterns for other classes in annotation.yaml
(or even other files) based on those that have id_prefixes
slots. See
There was an instance where two forms of KEGG id prefixes got into the pilot metadata so we want to guard against things like that in the future.