Closed paynejd closed 3 years ago
snyaggarwal
commented
3 years ago @jamlung-ri I looked at the OpenMRS concept validator and also tried to create a concept with same name in "Fully Specified" and "Short" and it succeeded. Can you please provide an import object which is failing?
jamlung-ri
commented
3 years ago I don't really know much about this one, so @paynejd or @bmamlin are you able to provide the an import object that fails?
snyaggarwal
commented
3 years ago @paynejd @jamlung-ri Any update on this?
bmamlin
commented
3 years ago When I try to turn on the OpenMRS custom validation schema for the PIH source on staging and click the “Update” button to save the change, it takes a long time (minutes) and eventually comes back with a message “Please fill mandatory fields” with these two errors in the browser console:
If I remove the custom validation schema (change it back to "None"), the update button works and the source is saved successfully.
snyaggarwal
commented
3 years ago There are following errors on changing the schema to OpenMRS:
{
"failed_concept_validations":[
{
"mnemonic":"4291",
"url":"/orgs/PIH/sources/PIH/concepts/4291/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"4301",
"url":"/orgs/PIH/sources/PIH/concepts/4301/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Agression sexuelle (locale: fr, preferred: yes)"
]
}
},
{
"mnemonic":"261",
"url":"/orgs/PIH/sources/PIH/concepts/261/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Celibate (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"306",
"url":"/orgs/PIH/sources/PIH/concepts/306/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"342",
"url":"/orgs/PIH/sources/PIH/concepts/342/",
"errors":{
"names":[
"Invalid name type: ltfu (locale: en, preferred: no)"
]
}
},
{
"mnemonic":"361",
"url":"/orgs/PIH/sources/PIH/concepts/361/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Confirmed (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"4356",
"url":"/orgs/PIH/sources/PIH/concepts/4356/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Dermatitis (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"434",
"url":"/orgs/PIH/sources/PIH/concepts/434/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"4391",
"url":"/orgs/PIH/sources/PIH/concepts/4391/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"615",
"url":"/orgs/PIH/sources/PIH/concepts/615/",
"errors":{
"names":[
"Invalid name type: inability, incapacity (locale: en, preferred: no)"
]
}
},
{
"mnemonic":"2791",
"url":"/orgs/PIH/sources/PIH/concepts/2791/",
"errors":{
"names":[
"Invalid name type: degree (locale: en, preferred: no)"
]
}
},
{
"mnemonic":"2915",
"url":"/orgs/PIH/sources/PIH/concepts/2915/",
"errors":{
"names":[
"Invalid name type: rhez (locale: en, preferred: no)"
]
}
},
{
"mnemonic":"2920",
"url":"/orgs/PIH/sources/PIH/concepts/2920/",
"errors":{
"names":[
"Invalid name type: art reason started (locale: en, preferred: no)"
]
}
},
{
"mnemonic":"2972",
"url":"/orgs/PIH/sources/PIH/concepts/2972/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"934",
"url":"/orgs/PIH/sources/PIH/concepts/934/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"4876",
"url":"/orgs/PIH/sources/PIH/concepts/4876/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Arthrite \\xc3\\xa0 bact\\xc3\\xa9ries pyog\\xc3\\xa8nes (locale: fr, preferred: no)"
]
}
},
{
"mnemonic":"4889",
"url":"/orgs/PIH/sources/PIH/concepts/4889/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"965",
"url":"/orgs/PIH/sources/PIH/concepts/965/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Mouri (locale: ht, preferred: yes)"
]
}
},
{
"mnemonic":"3067",
"url":"/orgs/PIH/sources/PIH/concepts/3067/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: HIV infected (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"3073",
"url":"/orgs/PIH/sources/PIH/concepts/3073/",
"errors":{
"names":[
"Invalid name type: ltfu (locale: en, preferred: no)"
]
}
},
{
"mnemonic":"3080",
"url":"/orgs/PIH/sources/PIH/concepts/3080/",
"errors":{
"names":[
"Invalid name type: hrz (locale: en, preferred: no)"
]
}
},
{
"mnemonic":"4962",
"url":"/orgs/PIH/sources/PIH/concepts/4962/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Infection respiratoire aigu\\xc3\\xab (locale: fr, preferred: yes)"
]
}
},
{
"mnemonic":"4971",
"url":"/orgs/PIH/sources/PIH/concepts/4971/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"3121",
"url":"/orgs/PIH/sources/PIH/concepts/3121/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Latrine (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"3206",
"url":"/orgs/PIH/sources/PIH/concepts/3206/",
"errors":{
"names":[
"Invalid name type: teenager (locale: en, preferred: no)"
]
}
},
{
"mnemonic":"5107",
"url":"/orgs/PIH/sources/PIH/concepts/5107/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: L\\xc3\\xa8pre borderline tuberculoide (locale: fr, preferred: yes)"
]
}
},
{
"mnemonic":"1189",
"url":"/orgs/PIH/sources/PIH/concepts/1189/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Burn (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"1195",
"url":"/orgs/PIH/sources/PIH/concepts/1195/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Difficulty breathing (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"1200",
"url":"/orgs/PIH/sources/PIH/concepts/1200/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Persistent diarrhea (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"1211",
"url":"/orgs/PIH/sources/PIH/concepts/1211/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Saignement au cours d\\'un effort de toux (locale: fr, preferred: yes)"
]
}
},
{
"mnemonic":"1221",
"url":"/orgs/PIH/sources/PIH/concepts/1221/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Yellow (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"1223",
"url":"/orgs/PIH/sources/PIH/concepts/1223/",
"errors":{
"names":[
"Concept preferred name must be unique for same source and locale: Green (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"1225",
"url":"/orgs/PIH/sources/PIH/concepts/1225/",
"errors":{
"names":[
"Concept preferred name must be unique for same source and locale: Red (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"1233",
"url":"/orgs/PIH/sources/PIH/concepts/1233/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"3344",
"url":"/orgs/PIH/sources/PIH/concepts/3344/",
"errors":{
"names":[
"Invalid name type: sanitary pad (locale: en, preferred: no)"
]
}
},
{
"mnemonic":"1320",
"url":"/orgs/PIH/sources/PIH/concepts/1320/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"5331",
"url":"/orgs/PIH/sources/PIH/concepts/5331/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Arthrite \\xc3\\xa0 bact\\xc3\\xa9ries pyog\\xc3\\xa8nes (locale: fr, preferred: yes)"
]
}
},
{
"mnemonic":"3559",
"url":"/orgs/PIH/sources/PIH/concepts/3559/",
"errors":{
"names":[
"Invalid name type: Labetolol (locale: en, preferred: no)"
]
}
},
{
"mnemonic":"5553",
"url":"/orgs/PIH/sources/PIH/concepts/5553/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Stillbirth (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"5603",
"url":"/orgs/PIH/sources/PIH/concepts/5603/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"5604",
"url":"/orgs/PIH/sources/PIH/concepts/5604/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"5606",
"url":"/orgs/PIH/sources/PIH/concepts/5606/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"5608",
"url":"/orgs/PIH/sources/PIH/concepts/5608/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"1685",
"url":"/orgs/PIH/sources/PIH/concepts/1685/",
"errors":{
"names":[
"Invalid name type: covid test (locale: en, preferred: no)"
]
}
},
{
"mnemonic":"5668",
"url":"/orgs/PIH/sources/PIH/concepts/5668/",
"errors":{
"names":[
"Concept fully specified name must be unique for same source and locale: Absent (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"5675",
"url":"/orgs/PIH/sources/PIH/concepts/5675/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"5688",
"url":"/orgs/PIH/sources/PIH/concepts/5688/",
"errors":{
"names":[
"Concept preferred name must be unique for same source and locale: Miscarriage (locale: en, preferred: yes)"
]
}
},
{
"mnemonic":"4103",
"url":"/orgs/PIH/sources/PIH/concepts/4103/",
"errors":{
"names":[
"Concept preferred name must be unique for same source and locale: R\\xc3\\xa9tention placentaire (locale: fr, preferred: yes)"
]
}
},
{
"mnemonic":"4112",
"url":"/orgs/PIH/sources/PIH/concepts/4112/",
"errors":{
"names":[
"Invalid name locale"
]
}
},
{
"mnemonic":"4197",
"url":"/orgs/PIH/sources/PIH/concepts/4197/",
"errors":{
"names":[
"Invalid name locale"
]
}
}
]
}Based on my analysis of the validation errors, I found 17 require PIH changes and 32 of them are resulting from three OCL issues...
in or en_GB locale, which OCL doesn't recognize. Remedy: OCL should accept in (Indonesia) and en_GB (British English) as valid locales
am, bn, en, en_GB, es, fr, ht, in, it, nl, pt, ru, rw, sw, ti, ur, and vi INDEX_TERM is considered an invalid name type: 342, 615, 2791, 2915, 2920, 3073, 3080, 3206, 3344, 3559, 1685. In each case, the validation error reported the first name with name_type of INDEX_TERM. Remedy: the name type of INDEX_TERM should be considered valid.snyaggarwal
commented
3 years ago @bmamlin @paynejd
en_GB locale in the system. It should be there in /orgs/OCL/sources/Locales/. Should we add this? And are there more to be added?INDEX_TERM in this exact way is invalid in OpenMRSSchemaValidator. Valid ones are:
Index-Termbmamlin
commented
3 years ago 
paynejd
commented
3 years ago FYI locales in OCL were loaded directly from ISO 639-1 (2-letter code) and ISO 639-2 (3-letter code). Each name is tagged with the appropriate name_type. We can definitely add the Java locales too, but we'll also need to think about the right way to tag them. What is the source for Java locales?
On Fri, May 14, 2021 at 8:52 AM Burke Mamlin @.***> wrote:
- @bmamlin https://github.com/bmamlin will fetch full list of official Java locales and create a separate ticket for adding these to OCL
- @snyaggarwal https://github.com/snyaggarwal will change OCL concept name types to exactly match "FULLY_SPECIFIED", "SHORT", and "INDEX_TERM"
- @snyaggarwal https://github.com/snyaggarwal will adapt OCL custom OpenMRS validator to exclude both SHORT and INDEX_TERM
- @bmamlin https://github.com/bmamlin will review the OpenMRS Custom Validator (specified in #750 https://github.com/OpenConceptLab/ocl_issues/issues/750) with the OCL for OpenMRS Squad to ensure any additional needed changes are properly enumerated
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OpenConceptLab/ocl_issues/issues/732#issuecomment-841224064, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJCOONH52GRQHEPQXY5M5DTNUMIHANCNFSM43TMAUEA .
bmamlin
commented
3 years ago FYI locales in OCL were loaded directly from ISO 639-1 (2-letter code) and ISO 639-2 (3-letter code). Each name is tagged with the appropriate name_type. We can definitely add the Java locales too, but we'll also need to think about the right way to tag them. What is the source for Java locales?
See #757.
lnball
commented
3 years ago @bmamlin @paynejd @snyaggarwal Do you advise that I change/remove any locales? PIH doesn't need 'in'. Assume this was used in CIEL or original OpenMRS concepts since we would not have added. Same for en_GB.
lnball
commented
3 years ago @bmamlin Do you have a script that I can run locally to make certain that the PIH concepts pass your vigorous standards?
lnball
commented
3 years ago Posted on Talk with link to my notes addressing all the issues. Will recreate a fresh database. Hopefully valid... https://docs.google.com/document/d/13O0nwwbRS0P7hw2PxKQ1F7vS6pN2uaNib9lsWJNPSJ4/edit#
bmamlin
commented
3 years ago @snyaggarwal I uploaded @lnball's latest updates as a fresh version of the PIH/PIH source on staging and tried turning on the OpenMRS Custom Validation. Can you share the errors resulting from changing the schema? I'm hoping it will be a shorter list this time. 🙂
snyaggarwal
commented
3 years ago Only one error --
{
"failed_concept_validations": [
{
"mnemonic": "193",
"url": "/orgs/PIH/sources/PIH/concepts/193/",
"errors": {
"names": [
"Concept fully specified name must be unique for same source and locale: Confirmed (locale: en, preferred: yes)"
]
}
}
]
}@bmamlin https://api.staging.openconceptlab.org/orgs/PIH/sources/PIH/concepts/843/ has Confirmed en synonym with no name_type. I think /orgs/PIH/sources/PIH/concepts/193/ clashed with this.
Right now we are excluding SHORT, INDEX_TERM from comparison, should we exclude empty name_type as well?
bmamlin
commented
3 years ago @bmamlin https://api.staging.openconceptlab.org/orgs/PIH/sources/PIH/concepts/843/ has
Confirmedensynonym with noname_type. I think/orgs/PIH/sources/PIH/concepts/193/clashed with this. Right now we are excluding SHORT, INDEX_TERM from comparison, should we exclude emptyname_typeas well?
Excellent! No. An empty name_type implies a synonym, which should not match a fully specified name of another concept. In this case, PIH 193 has an underspecified FSN – i.e., "Confirmed" for "Confirmed diagnosis". @lnball should be able to easily fix this.
bmamlin
commented
3 years ago Yesterday, I uploaded the updated PIH dictionary (2021-05-28) that @lnball sent me via email. This time, I deleted the PIH source and, when I recreated it, I set the PIH source to use "OpenMRS" custom validation up front (before uploading the import). After uploading the import file, nothing showed up in the import queue. I tried uploading again and got a message like "You've already uploaded this import file."
@snyaggarwal is investigating to see why the import is failing.
bmamlin
commented
3 years ago @snyaggarwal,
Here are the results of importing the PIH dictionary with custom validation set to OpenMRS:
| Import | OCL | Difference | |
|---|---|---|---|
| Concepts | 5711 | 5705 | -6 |
| Mappings | 28121 | 28112 | -9 |
Is it easy for you to find logs that might explain what happened for 6 concepts & 9 mappings that failed to import?
snyaggarwal
commented
3 years ago @bmamlin the results of job got lost due to deployment. The only way is to compare the script with ocl and get the difference and import just that diff again.
snyaggarwal
commented
3 years ago @bmamlin For import please use custom queue id. It will go into separate queue and wont have to wait
bmamlin
commented
3 years ago @snyaggarwal I performed a diff of PIH content on OCL and what I had imported. It turns out the differences are more confusing than I expected. While I see what looks like 5 concepts are missing from OCL (5711 from PIH, 5706 in OCL), a diff between what PIH gave us and what is on OCL shows only one missing concept:
{
"retired": true,
"datatype": "N/A",
"type": "Concept",
"concept_class": "Misc",
"source": "PIH",
"extras": {},
"names": [
{
"locale": "fr",
"external_id": "3e754086-55c0-4862-8a50-26e0b6f8f512",
"locale_preferred": true,
"name": "Dextrose 5% fluide",
"name_type": "FULLY_SPECIFIED"
},
{
"locale": "en",
"external_id": "5c702054-6a5c-11e2-b6f9-aa00f871a3e1",
"locale_preferred": true,
"name": "Dextrose 5%",
"name_type": "FULLY_SPECIFIED"
}
],
"owner": "PIH",
"owner_type": "Organization",
"external_id": "5c6b1834-6a5c-11e2-b6f9-aa00f871a3e1",
"id": "3779",
"descriptions": []
}I tried importing this via bulk import of raw json for the missing concept and got the exception:
JSONDecodeError('Expecting property name enclosed in double quotes: line 1 column 2 (char 1)')What looked like it might be 9 missing mappings (28121 from PIH, 28112 on OCL), the diff showed 264 missing mappings (not on OCL) and 256 "extra mappings" (mappings from OCL not in the PIH import file). That's going to take a little more digging to understand.
lnball
commented
3 years ago Thanks (and sorry, Burke).
Wondering if the missing maps or concepts have % or other characters. This is an example of mapping for 'Dextrose 10%'.
I'm on vacation next week. (Mark too) But I can dig deeper when I return.
lnball
commented
3 years ago @bmamlin @snyaggarwal Where are we? How can I help?
FYI @mogoodrich
bmamlin
commented
3 years ago @snyaggarwal I created a PIH-temp source and tried importing the PIH dictionary into it. 30 concepts were imported and then it seems to have stopped. Any ideas?
snyaggarwal
commented
3 years ago @bmamlin Can you please send me the PIH import file
snyaggarwal
commented
3 years ago @bmamlin pih-temp was imported many hours back. The result had all success except 1 failure:
[
{
"retired": true,
"datatype": "N/A",
"type": "Concept",
"concept_class": "Misc",
"source": "PIH-temp",
"extras": {},
"names": [
{
"locale": "fr",
"external_id": "3e754086-55c0-4862-8a50-26e0b6f8f512",
"locale_preferred": true,
"name": "Dextrose 5% fluide"
},
{
"locale": "en",
"external_id": "5c702054-6a5c-11e2-b6f9-aa00f871a3e1",
"locale_preferred": true,
"name": "Dextrose 5%"
}
],
"owner": "PIH",
"owner_type": "Organization",
"external_id": "5c6b1834-6a5c-11e2-b6f9-aa00f871a3e1",
"id": "3779",
"descriptions": [],
"errors": {
"names": [
"Concept fully specified name must be unique for same source and locale: Dextrose 5% (locale: en, preferred: yes)"
]
}
}
]Closing out this ticket. @bmamlin will create another ticket to track the remainder of the PIH import process.
Currently, the OpenMRS validation schema doesn't allow a "Fully Specified" concept name to be the same as a "Short" concept name. However, OpenMRS allows this and we are not able to import the PIH OpenMRS dictionary into OCL as a result. We want to modify the validation schema to allow this.
@snyaggarwal Assuming this is a quick fix -- let us know if not!
cc @bmamlin