Create a new Component called the Validator Service

[brightoibe]

BACKGROUND Dirty data is a nightmare for every analyst. Dirty data can create inconsistencies in the output of the Cohort Builder. In order to avoid this, A Java program called the "Validator" was created. The goal of the validator is to identify and flag Patients, Encounters, Obs that are not qualified to be included in the count results of the Cohort Builder. A flag is recorded against the record e.g ignore_patient=1 attribute so that the Cohort Builder and the ETL (Extract, Transform, Load) does not include that patient in the analysis.

ROLE OF VALIDATOR

• Flag Patients, Encounters and Obs that should not be used for analysis by adding associating a property to their UUID e.g ignore_patient=1 or ignore_encounter=1 or ignore_obs=1

• Validator also generates user friendly messages to the end users on the particular data quality issues that caused the record to be ignored.

• Not all validation errors will cause a record to be ignored. Some will produce a warning to the end user to do a further verification.

• Validator service runs immediately after the the Consumer service and populates the error table.

• ETL checks the error table before bringing your records to the analytic database.

[lordmaul]

Can we have a list of the requirements for flagging a record (patient record, encounter, obs etc) as dirty?