search

chop-dbhi / data-models-validator

Set of tools for validating data that should conform to a data model.
1 stars 2 forks source link

Enforcing incorrect constraint #25

Closed murphyke closed 8 years ago

murphyke commented 8 years ago

Using the following pedsnet 2.1.0 drug_exposure file (fake data), the data-models-validator v1.0.2 incorrectly believes that the drug_exposure_end_time column is required, when http://dmsa.a0b.io/pedsnet/2.1.0/ddl/postgresql/tables/ shows that this column can be NULL.

Input:

"DAYS_SUPPLY","DOSE_UNIT_CONCEPT_ID","DOSE_UNIT_SOURCE_VALUE","DRUG_CONCEPT_ID","DRUG_EXPOSURE_END_DATE","DRUG_EXPOSURE_END_TIME","DRUG_EXPOSURE_ID","DRUG_EXPOSURE_ORDER_DATE","DRUG_EXPOSURE_ORDER_TIME","DRUG_EXPOSURE_START_DATE","DRUG_EXPOSURE_START_TIME","DRUG_SOURCE_CONCEPT_ID","DRUG_SOURCE_VALUE","DRUG_TYPE_CONCEPT_ID","EFFECTIVE_DRUG_DOSE","EFFECTIVE_DRUG_DOSE_SOURCE_VALUE","FREQUENCY","LOT_NUMBER","PERSON_ID","PROVIDER_ID","QUANTITY","REFILLS","ROUTE_CONCEPT_ID","ROUTE_SOURCE_VALUE","SIG","STOP_REASON","VISIT_OCCURRENCE_ID"
,,"","0",,,"666","2006-02-05","2006-02-05 16:06:03","2006-02-05","2006-02-05 15:58:57","0","amoxicillin-clavulanate","38000177",,"","","","666","666",,,"0","","","Physician Stop","666"

Output:

$ data-models-validator -model pedsnet -version 2.1.0 junk.csv:drug_exposure
Validating against model 'pedsnet/2.1.0'
* Evaluating 'drug_exposure' table in 'junk.csv'...
* Field-level issues were found.
+------------------------+------+-------------------+-------------+-------+------------+
|         FIELD          | CODE |       ERROR       | OCCURRENCES | LINES |  SAMPLES   |
+------------------------+------+-------------------+-------------+-------+------------+
| drug_exposure_end_time |  300 | Value is required | 1           | 2     | line 2: `` |
+------------------------+------+-------------------+-------------+-------+------------+
murphyke commented 8 years ago

The problem originates in the data-models service. http://data-models.origins.link/models/pedsnet/2.1.0?format=json returns incorrect required values for some fields. Will create an issue there.