DPL-753 Consumer to receive new QC fields from message queue (S=L,C=M)

[emrojo]

User story As TOL I want to receive new QC fields from the queue:

Qc columns

• Sheared Femto Fragment Size (bp)
• Post SPRI Concentration (ng/ul)
• Post SPRI Volume (ul)
• Final NanoDrop 260/280
• Final NanoDrop 260/230
• Final NanoDrop ng/ul
• Shearing & QC comments (if applicable)
• Date submitted
• Priority level (if High or Medium)
• Date data required by
• Reason for priority

Who are the primary contacts for this story Eduardo Martin

Who is the nominated tester for UAT Charlie (TOL)

Acceptance criteria To be considered successful the solution must allow:

• [x] Update the schema in consumer
• [x] Process the new qc columns in consumer and send it to traction
• [x] Process the new qc message in Traction and send it to warehouse

Dependencies This story is blocked by the following dependencies:

• sanger/#
• DPL-753-1
• DPL-753-2
• DPL-754

References This story has a non-blocking relationship with:

• sanger/#

Additional context

Definition of fields in: https://docs.google.com/spreadsheets/d/1OW78PEbC8BzqV5TtRV9zV_4OU3t7HhCeBWSJv0rGno0/edit#gid=0

[emrojo]

As a suggestion, these fields could be provided from a new message, using the sampleUuid as reference.

[TWJW-SANGER]

James has confirmed that that the long read QC data in the message can be stored and viewed from the MLWH. Once this story is done we can review if we can help Long Read deprecate the Google Sheet they are using to store and track this information. By Long Read QC data I mean these fields:

• Sheared Femto Fragment Size (bp)
• Post SPRI Concentration (ng/ul)
• Post SPRI Volume (ul)
• Final NanoDrop 260/280
• Final NanoDrop 260/230
• Final NanoDrop ng/ul
• Shearing & QC comments (if applicable)

The other fields in this story:

• Date submitted
• Priority level (if High or Medium)
• Date data required by
• Reason for priority

These can also be stored in the MLWH but it's worth noting that these fields will be common for all samples being submitted to SeqOps, so likely be used in different views.

[emrojo]

Tasks

A) UPDATE THE SCHEMA - DPL-753-1

• [x] Setup local dev environment (Follow steps in https://github.com/sanger/tol-lab-share/blob/develop/SETUP_DOCKER.md)
• [x] Add new qc fields into the schema
• [x] Publish new schema in Redpanda in local (https://github.com/sanger/tol-lab-share/blob/develop/schemas/README.md)
• [x] Modify current demo in tol-lab-share to include the testing data fields (tol-lab-share/tools/demo/testing_data.py)
• [x] Start traction service in local and configure consumer to talk with traction service in local
• [x] Run demo to publish new message and check that the consumer processes the message correctly

B) PROCESS THE FIELDS IN CONSUMER - DPL-753-2

• [x] In tol_lab_share project, add new message properties for each of the new qc fields and add them as properties for a sample with no validation apart from checking is a string (check how it is with CommonName) (in message_properties/sample.py)
• [x] Create new message for qc_results traction to include the new fields in the payload we send
• [x] Add the new properties into the qc results traction message that we will send (in message_properties/labware.py). We need also to add the attributes to qc results traction message interface to add the new fields
• [x] Modify the CreateLabwareProcessor to send the message to qc_results traction endpoint if the creation of samples was successful in previous step
• [x] In case of errror while sending qc_result request, append the errors to the feedback message

C) PROCESS THE FIELDS IN TRACTION DPL-754

• [x] Add the new samples specific columns to the qc_results table in a migration: Date submitted, Priority level (if High or Medium), Date data required by, Reason for priority
• [x] Add the new fields to the qc_results API endpoint as well
• [x] Add the qc fields as new qc_assay_types in a migration: Sheared Femto Fragment Size (bp), Post SPRI Concentration (ng/ul), Post SPRI Volume (ul), Final NanoDrop 260/280, Final NanoDrop 260/230, Final NanoDrop ng/ul, Shearing & QC comments (if applicable)
• [x] Add functionality to send a new qc_results message to the warehouse with every new qc_results after_create callback

[harrietc52]

Traction Service tasks

• [x] update schema to add new columns for qc_results table (Date submitted, Priority level (if High or Medium), Date data required by, Reason for priority). These are optional/ only required for TOL?
• [x] new route in resource, for endpoint to create multiple qc results
• [x] new controller for route
• [x] new resource for route
• [x] new factory, possibly being passed all the qc results to create. this loops through, creating the qc result, and if successful, building the message
• [x] in factory, get the relevant assay types, using the tol used_by to filter them
• [x] in factory, validation required assay types are present
• [x] in the resource, if creation of qc results successful, access messages from the factory
• [x] add config to build MLWH message to pipelines.yml. this wants to be similar to the existing qc_result message (but without qc_decision etc). But it also wants to add new qc result fields such as (Date submitted, Priority level (if High or Medium), Date data required by, Reason for priority)
• [x] publish message using factory messages and config
• [x] edit rake task to add new assay types (confirm keys with tol, and check required units)
• [x] add new used_by to assay type model for tol

MLWH / Unified WH tasks

• [x] update MLWH table, via migration in unified warehouse, to add new qc_result fields (date submitted etc) possibly? See design of traction MLWH table here
• [x] check MLWH table inserts new qc assay type fields and values correctly

[Sangeetha-Bheeman]

Steps to deploy:

1. Publish the new schema to red panda:

push.sh <REDPANDA_URL> <API_KEY>

For UAT(Use the right API KEY from vault):

./push.sh https://redpanda.uat.tol.psd.sanger.ac.uk API_KEY

2. Deploy tol-lab-share:

UAT:

ansible-playbook -vD deploy_tol_lab_share.yml -i environments/uat -e "github_branch=0.2.0-5574405014-develop"

The new schema is deployed to UAT and it can be fetched as below (Use the right API-KEY):

curl -H "Content-Type: application/vnd.schemaregistry.v1+json" -H "X-API-KEY: API-KEY" "https://redpanda.uat.tol.psd.sanger.ac.uk/subjects/create-labware/versions/latest"

{ "subject": "create-labware", "version": 30, "id": 18, "schema": "{\"namespace\":\"uk.ac.sanger.psd\",\"type\":\"record\",\"name\":\"CreateLabwareMessage\",\"doc\":\"A create message to process new labware.\",\"fields\":[{\"name\":\"messageUuid\",\"doc\":\"Unique message ID.\",\"type\":{\"name\":\"version4MessageUuid\",\"type\":\"fixed\",\"size\":36}},{\"name\":\"messageCreateDateUtc\",\"doc\":\"Date (UTC) that the message was created.\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}},{\"name\":\"labware\",\"doc\":\"Labware to process.\",\"type\":{\"name\":\"Labware\",\"type\":\"record\",\"doc\":\"Labware data.\",\"fields\":[{\"name\":\"labwareType\",\"type\":\"string\"},{\"name\":\"barcode\",\"type\":\"string\",\"doc\":\"The barcode for this labware.\"},{\"name\":\"samples\",\"doc\":\"An array of data for samples on the labware.\",\"type\":{\"type\":\"array\",\"items\":{\"name\":\"Sample\",\"type\":\"record\",\"doc\":\"A labware sample\",\"fields\":[{\"name\":\"sampleUuid\",\"doc\":\"Unique sample ID in UUID format.\",\"type\":\"version4MessageUuid\"},{\"name\":\"studyUuid\",\"doc\":\"Unique study UUID where the sample belongs to\",\"type\":\"version4MessageUuid\"},{\"name\":\"sangerSampleId\",\"type\":\"string\",\"doc\":\"Unique id to identify the sample inside Sanger lims\"},{\"name\":\"location\",\"type\":\"string\",\"doc\":\"The location for the sample inside the labware, or null if location is not relevant in labware (Eg: tube).\"},{\"name\":\"supplierSampleName\",\"type\":\"string\",\"doc\":\"Name given by the supplier of the sample\"},{\"name\":\"volume\",\"type\":\"string\",\"doc\":\"Volume of the sample.\"},{\"name\":\"concentration\",\"type\":\"string\",\"doc\":\"Concentration of the sample.\"},{\"name\":\"publicName\",\"type\":\"string\",\"doc\":\"Public name of the sample.\"},{\"name\":\"taxonId\",\"type\":\"string\",\"doc\":\"Taxonomy Identifier.\"},{\"name\":\"commonName\",\"type\":\"string\",\"doc\":\"Common name for the sample\"},{\"name\":\"donorId\",\"type\":\"string\",\"doc\":\"Identifier for the donor of the sample\"},{\"name\":\"libraryType\",\"type\":\"string\",\"doc\":\"Type of library created for sequencing\"},{\"name\":\"countryOfOrigin\",\"type\":\"string\",\"doc\":\"Country of origin for the sample\"},{\"name\":\"sampleCollectionDateUtc\",\"doc\":\"Sample Collection of the sample in UTC format.\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}},{\"name\":\"costCode\",\"type\":\"string\",\"doc\":\"Code for the project for billing purposes\"},{\"name\":\"genomeSize\",\"type\":\"string\",\"doc\":\"Size of genome\"},{\"name\":\"accessionNumber\",\"type\":\"string\",\"doc\":\"Accession number for publication in ENA/EBI\"},{\"name\":\"shearedFemtoFragmentSize\",\"type\":\"string\",\"doc\":\"Sheared Femto Fragment Size\"},{\"name\":\"postSPRIConcentration\",\"type\":\"string\",\"doc\":\"Post SPRI Concentration (ng/ul)\"},{\"name\":\"postSPRIVolume\",\"type\":\"string\",\"doc\":\"Post SPRI Volume (ul)\"},{\"name\":\"finalNanoDrop280\",\"type\":\"string\",\"doc\":\"Final NanoDrop 260/280\"},{\"name\":\"finalNanoDrop230\",\"type\":\"string\",\"doc\":\"Final NanoDrop 260/230\"},{\"name\":\"finalNanoDrop\",\"type\":\"string\",\"doc\":\"Final NanoDrop ng/ul\"},{\"name\":\"shearingAndQCComments\",\"type\":\"string\",\"doc\":\"Shearing & QC comments\"},{\"name\":\"dateSubmittedUTC\",\"doc\":\"Date submitted in UTC format.\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}},{\"name\":\"priorityLevel\",\"type\":\"string\",\"doc\":\"Priority level (if High or Medium)\"},{\"name\":\"dateRequiredBy\",\"type\":\"string\",\"doc\":\"Date data required by\"},{\"name\":\"reasonForPriority\",\"type\":\"string\",\"doc\":\"Reason for priority\"}]}}}]}}]}" }

[Sangeetha-Bheeman]

Closing this, as all the tasks in this are being tracked as part of DPL- 753-1, DPL-753-2 and DPL-754.