Clean-up data values for the release

[zperova]

Description

As a developer, I want to clean-up all the errors identified by the validator tool.

Acceptance criteria

• [x] all validator reports have been reviewed and data cleaned

[zperova]

This ticket needs to be broken up once the validator has been run on all datasets.

[zperova]

Results of the validator are here: https://app.zenhub.com/workspaces/pdxfinder-board-5f0f036188f27e0014a7bdfc/issues/pdxfinder/pdxfinder/497

[zperova]

Here are the suspects. Error count/provider. The count is subject to change: ? provider [TRACE] @Afollette Analysis done. Errors reported in validator but data seems to be ok. 1 provider [UMCG] @mauroz77 Analysis done. Errors reported in validator but data seems to be ok. 2 provider [NKI] @Afollet Analysis done. False positives in validator are reported 3 provider [Curie-BC] @mauroz77 Analysis done. Errors reported in validator but data seems to be ok. Fixed publications value separated with ; 4 provider [Curie-LC] @mauroz77 Analysis done. Fixed data. There are some false positives in validator. 5 provider [Curie-OC] @mauroz77 Analysis done. Fixed data. There are some false positives in validator. 5 provider [IRCC-GC] @mauroz77 Analysis done. Fixed data. There are some false positives in validator. 6 provider [IRCC-CRC] @Afollet Anaylsis done. Fixed data. There are some false positives in validator. 6 provider [UOM-BC] @mauroz77 Analysis done. Fixed data. There are some false positives in validator. 6 provider [VHIO-BC] @mauroz77 Analysis done. Fixed data. There are some false positives in validator. 7 provider [LIH] @mauroz77 Analysis done. Fixed data. There are some false positives in validator. 7 provider [PMLB] @Afollet Analysis done. Fixed data. There are some false positives in validator 7 provider [UOC-BC] @Afollet Analysis done. Fixed data. There are some false positives in validator. 7 provider [VHIO-CRC] @mauroz77 Analysis done. Fixed data. There are some false positives in validator. 9 provider [HCI-BCM] @mauroz77 Analysis done. Fixed data. There are some false positives in validator. 10 provider [VHIO-PC] @mauroz77 done 12 provider [PDMR] @Afollet ( I am going to pass on this, modifications will be erased when we pull new data from the Oracle database. It would be better to fix it at the PDX-Transformer than the template level ) 12 provider [SJCRH] @mauroz77 done 13 provider [CRL] @mauroz77 done 14 provider [PPTC] @Afollet ( waiting on correspondence ) 14 provider [WUSTL]@mauroz77 done 17 provider [MDAnderson]@mauroz77 done 18 provider [DFCI-CPDM]@mauroz77 waiting provider's response

[mauroz77]

Things to check:

• It seems like numeric sample_id (sample sheet) values are not recognised. Reported as "Missing value(s) in row numbers..."
• Values for passages_tested (model_validation sheet) are not recognised when the value can be casted to a number. If the value is detected as string, it is recognised. E.g:
  • 3 --> Not recognised
  • 1,20,44 --> Recognised
• Values in ethnicity_assessment_method (patient sheet) are not allowing the value "Not Specified". The error message shows as possible values: [self-assessed,genetic]. Is it correct to not allow "Not Specified" as a valid entry?
• sample_id, model_id, and patient_id can have values with "[" or "]". E.g: "WUSTL 356 [2]-07"
• collection_site can have values with "&". E.g: peritoneal & ascites, liver & lung, pancreatic net & liver, primary colon & liver (MDanderson)

[Afollet]

Thanks @mauroz77 I have made the same observations. I appears that the jtablesaw library is reading in the table and replacing the numeric types with blanks. If you use the "TYPE" function in excel you can see the cell value types are different. I'mma going open an issue on the jtablesaw repo about this.

[zperova]

Moving to the next sprint. Large job.

[Afollet]

I have resolved all the tumour_type changes besides for Jax, DFCI-CPDM Everything that has "not specified" as a tumour_type

I changed all the mappings to correspond with these tumour_type changes.

[zperova]

@Afollet I take it as I can start mapping now?

[Afollet]

@zperova I checked the mappings yesterday. There was not anything new. Is there suppose to be new mappings?

[zperova]

I thought there will be about 30 mappings to do @CsabaHalmagyi was talking about - possibly from JAX

[Afollet]

I have a few changes on my local machine that I will push when gitlab is running.

There are correspondence between PPTC and DFCI-CPDM ( as @mauroz77 said ), but the rest of the work has been done.

Report here: https://docs.google.com/document/d/1bPl1xfT5jylv0ntONCcfoa1MLOdsZHKzXcO1cjCauVA/edit#heading=h.t0vdk2nut12d

[Afollet]

@zperova I resolved all the mappings I changed yesterday. There is nothing for you to wait on here.

[zperova]

Great, thanks @Afollet and @mauroz77 Report is very helpful! Closing.