Broken EPPs caused by an ongoing issue with cg-prod-services, 2023-03-29, 2023-04-03

Karl-Svard commented 1 year ago

Lims Issue

The LIMS connection our services on cg-prod-services is currently broken and is blocking most lab work. The following have been done to temporarily fix this.

Arnold

The Arnold upload EPPs have been turned of for the following steps.

EPP: Arnold Upload Sample Master step	Step	originally	changed to
Reception Control no placement v2	Reception Control TWIST	step, automatic upon entry	not used
Reception Control (RNA) v1	Reception Control (RNA)	step, automatic upon exit	not used
Reception Control (Dev) v3	Reception Control (Dev)	step, automatic upon exit	not used
Reception Control (Cov) v1	Reception Control (Cov)	step, automatic upon exit	not used
Reception Control no placement v1	Reception Control	step, automatic upon entry	not used
CG002 - Reception Control	CG002 - Reception Control	step, automatic upon exit	not used
CG002 - Reception Control	Reception Control NIPT	step, automatic upon exit	not used
CG002 - Reception Control	Reception Control v3	step, automatic upon exit	not used

EPP: Arnold Upload cov Master step	Step	originally	changed to
Aggregate QC (Library Validation) (Cov) v1	Aggregate QC (Library Validation) (Cov)	step, automatic upon exit	not used

EPP: Arnold Upload rna Master step	Step	originally	changed to
Aggregate QC (Library Validation) (RNA) v2	Aggregate QC (Library Validation) (RNA) (the one in post norm protocol)	record details, automatic upon exit	not used

EPP: Arnold Upload twist Master step	Step	originally	changed to
Aggregate QC (Library Validation) TWIST v2	Aggregate QC (Library Validation) TWIST (the one in post norm protocol)	step, automatic upon exit	not used

EPP: Arnold Upload wgs Master step	Step	originally	changed to
Aggregate QC (Library Validation)	Aggregate QC (Library Validation) (for the WGS wf)	step, automatic upon exit	not used

EPP: Arnold Upload mic Master step	Step	originally	changed to
CG002 - Normalization of microbial samples for sequencing	CG002 - Normalization of microbial samples for sequencing	step, automatic upon exit	not used

EPP: Arnold Upload Flow Cell AUTOMATED - NovaSeq Run, didn't need to be touched as Arnold got back online.

EPP: Arnold Upload sequencing and Set Next Steps - Remove Bcl Conversion & Demupltiplexing (Nova Seq), didn't need to be touched as Arnold got back online..

Atlas connection

Changed the host url for the atlas api in /home/glsai/.genologics.yaml to instead connect to the working stage environment (which has been set to master). ATLAS_HOST: https://atlas-api.scilifelab.se -> ATLAS_HOST: https://atlas-api-stage.scilifelab.se

Statusdb connection

Same solution as for Atlas, as the only thing fetched in statusdb is the apptags which seem to be up to date with prod. CG_URL: https://cg-statusdb.scilifelab.se/api/v1 -> CG_URL: https://cg-statusdb-stage.scilifelab.se/api/v1

Karl-Svard commented 1 year ago

Atlas and statusdb connections were reversed to their old states after cg-prod-services got back online.

ATLAS_HOST: https://atlas-api-stage.scilifelab.se -> ATLAS_HOST: https://atlas-api.scilifelab.se CG_URL: https://cg-statusdb-stage.scilifelab.se/api/v1-> CG_URL: https://cg-statusdb.scilifelab.se/api/v1

Karl-Svard commented 1 year ago

Inserting new data into Arnold seem to also be back to normal.

(cg_lims) karl.svard@karlsmbp cg_lims % lims -c ../config/genologics-prod.yaml epps -l ../rc_arnold_log -p 24-414890 arnold-upload sample
Sample documents inserted to arnold database
(cg_lims) karl.svard@karlsmbp cg_lims %

Will however wait with re-deploying the Arnold EPPs until after the lab day is complete.

Karl-Svard commented 1 year ago

Arnold EPPs re-deployed

[x] Reception Control TWIST
[x] Reception Control (RNA)
[x] Reception Control (Dev)
[x] Reception Control (Cov)
[x] Reception Control
[x] CG002 - Reception Control
[x] Reception Control NIPT
[x] Reception Control v3
[x] Aggregate QC (Library Validation) (Cov)
[x] Aggregate QC (Library Validation) (RNA) (the one in post norm protocol)
[x] Aggregate QC (Library Validation) TWIST (the one in post norm protocol)
[x] Aggregate QC (Library Validation) (for the WGS wf)
[x] CG002 - Normalization of microbial samples for sequencing

Karl-Svard commented 1 year ago

2023-04-03

The same issues are back again. The config file has been edited to look at statusdb and atlas stage, and the following Arnold steps have been undeployed:

[x] Reception Control TWIST
[x] Reception Control (RNA)
[x] Reception Control (Dev)
[x] Reception Control (Cov)
[x] Reception Control
[x] CG002 - Reception Control
[x] Reception Control NIPT
[x] Reception Control v3
[x] Aggregate QC (Library Validation) (Cov)
[x] Aggregate QC (Library Validation) (RNA) (the one in post norm protocol)
[x] Aggregate QC (Library Validation) TWIST (the one in post norm protocol)
[x] Aggregate QC (Library Validation) (for the WGS wf)
[x] CG002 - Normalization of microbial samples for sequencing

Karl-Svard commented 1 year ago

Config file was restored before lunch as cg-prod-services came online. Will wait with the Arnold parts until the VM seems more stable again.

Karl-Svard commented 1 year ago

Arnold steps were re-deployed again on 07:34, 2023-04-04, as the cause of the cg-prod-services issues has hopefully been solved with the following PR: https://github.com/Clinical-Genomics/servers/pull/1022

Clinical-Genomics / cg_lims