add storage to housekeeper function in Illumina service

Description

Closes https://github.com/Clinical-Genomics/add-new-tech/issues/24. Implement storage of Illumina demultiplexing files (fastq files, demux logs, and run parameters files) in the Illumina post-processing service.

The function from the old post-processing was copied and pasted to the new flow, considering that the old function will be removed. The intermediate functions used to handle the HK API defined in cg/meta/demultiplex/housekeeper_storage_functions.py were not moved to a new utils module, this is left for another PR where a better refactoring is done to these methods.

Added

Method store_illumina_data_in_housekeeper in IlluminaPostProcessingService.

Changed

Renamed method store_illumina_flow_cell_data to store_illumina_data_in_status_db for clarity

Fixed

How to prepare for test

No endpoint, changes will be tested when new post-processing is validated

Review

[x] "Merge and deploy" approved by CO Thanks for filling in who performed the code review and the test!

This version is a

[x] PATCH - when you make backwards compatible bug fixes or documentation/instructions

Implementation Plan

[x] Deployed to stage:

INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
repository is clean
Logging deploy ...
Getting deployer... done.
Getting last commit message and SHA... done.
Getting version of deploy scripts... /home/js.diazboada
done.
Log deploy... done.
cg, version 61.1.6
[js.diazboada@hasta:~] [S_base] $ up

[x] Deployed to production:

Logging deploy ...
Getting deployer... done.
Getting last commit message and SHA... done.
Getting version of deploy scripts... /home/js.diazboada
done.
Log deploy... done.
cg, version 61.1.6
remote: Enumerating objects: 577, done.
remote: Counting objects: 100% (396/396), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 201 (delta 161), reused 123 (delta 98), pack-reused 0
Receiving objects: 100% (201/201), 33.31 KiB | 0 bytes/s, done.
Resolving deltas: 100% (161/161), completed with 96 local objects.
From https://github.com/Clinical-Genomics/cg
9752cec..213bd4f  master     -> origin/master
4e4b7f9..a5bad31  add-raredisease-delivery-report -> origin/add-raredisease-delivery-report
f834b45..39f6673  nipt-qc-service -> origin/nipt-qc-service
a89672a..befa384  raredisease_store -> origin/raredisease_store
* [new tag]         v61.1.6    -> v61.1.6
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
Updating 9752cec..213bd4f
Fast-forward
.bumpversion.cfg                                                                                           |   2 +-
cg/__init__.py                                                                                             |   2 +-
cg/services/illumina_services/illumina_post_processing_service/housekeeper_storage.py                      | 145 +++++++++++++++++++++++++++++++++
cg/services/illumina_services/illumina_post_processing_service/illumina_post_processing_service.py         | 116 ++++++++++++++++++---------
cg/services/illumina_services/illumina_post_processing_service/utils.py                                    | 200 ++++++++++++++++++++++++++++++++++++++--------
cg/services/illumina_services/illumina_post_processing_service/validation.py                               |   7 +-
pyproject.toml                                                                                             |   2 +-
tests/conftest.py                                                                                          |  35 ++++----
tests/fixture_plugins/demultiplex_fixtures/metrics_fixtures.py                                             |  53 +++++++++++++
tests/fixture_plugins/demultiplex_fixtures/name_fixtures.py                                                |  45 +++++++++++
tests/fixture_plugins/demultiplex_fixtures/path_fixtures.py                                                |  12 +++
tests/fixture_plugins/demultiplex_fixtures/sample_fixtures.py                                              |   2 +-
tests/fixture_plugins/demultiplex_fixtures/sample_sheet_fixtures.py                                        |  14 ++++
tests/services/illumina_services/illumina_post_processing_service/conftest.py                              |  37 ---------
tests/services/illumina_services/illumina_post_processing_service/test_housekeeper_storage.py              | 129 ++++++++++++++++++++++++++++++
tests/services/illumina_services/illumina_post_processing_service/test_illumina_post_processing_service.py |  12 +--
tests/services/illumina_services/illumina_post_processing_service/test_utils.py                            | 340 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------
tests/store_helpers.py                                                                                     |   2 +-
18 files changed, 938 insertions(+), 217 deletions(-)
create mode 100644 cg/services/illumina_services/illumina_post_processing_service/housekeeper_storage.py
create mode 100644 tests/fixture_plugins/demultiplex_fixtures/metrics_fixtures.py
create mode 100644 tests/services/illumina_services/illumina_post_processing_service/test_housekeeper_storage.py
/home/js.diazboada
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
[js.diazboada@hasta:~] [P_base] $

Clinical-Genomics / cg