search

Clinical-Genomics / trailblazer

Keep track of and manage analyses
MIT License
6 stars 2 forks source link

Hidden analyses not found when uploading to scout #502

Open diitaz93 opened 2 weeks ago

diitaz93 commented 2 weeks ago

Description

Originates from https://github.com/Clinical-Genomics/bug-reports/issues/15 Development case petmako tried to be uploaded using cg upload -r -c petmako but got the error:

TrailblazerAnalysisNotFound: No completed analysis found for case petmako

Trailblazer (Cigrid) showed that the analysis was there.

As the analysis was for development (cust000), it was by default hidden in Trailblazer. When the analysis was manually set to "unhidden" the command worked as expected.

Steps to reproduce

Using taxprofiler development case petmako, run:

cg upload -r -c petmako

The same bug is expected for any development case

Expected behaviour

That development analyses can be uploaded without manually setting them as "unhidden" in trailblazer

Additional context

Vince-janv commented 2 weeks ago

@diitaz93 assigned to reproduce the bug

RasmusBurge-CG commented 1 week ago

In TB: on case petmako perform action Mark as hidden.

Then running command cg upload -r -c petmako:

Output is:

Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
----------------- UPLOAD -----------------
Case petmako exists in Status DB
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
--------------- DELIVERY REPORT ---------------
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Using analysis started at: 2024-10-29 12:01:13
Delivery report already in housekeeper: /home/proj/production/housekeeper-bundles/petmako/2024-10-29/delivery-report.html
Setting log dir to: /home/proj/production/rsync/petmako_241113_09_38_50_010410
Creating folder: /home/proj/production/rsync/petmako_241113_09_38_50_010410
Submit sbatch /home/proj/production/rsync/petmako_241113_09_38_50_010410/petmako_rsync.sh
Running command sbatch /home/proj/production/rsync/petmako_241113_09_38_50_010410/petmako_rsync.sh
Submitted batch job 7600292
Writing slurm jobs to /home/proj/production/rsync/petmako_241113_09_38_50_010410/slurm_job_ids.yaml
Traceback (most recent call last):
  File "/home/proj/production/bin/miniconda3/envs/P_cg/bin/cg", line 8, in <module>
    sys.exit(base())
             ^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 1666, in invoke
    rv = super().invoke(ctx)
         ^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/cli/upload/base.py", line 99, in upload
    upload_api.upload(ctx=context, case=case, restart=restart)
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/meta/upload/nf_analysis.py", line 44, in upload
    self.upload_files_to_customer_inbox(case)
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/meta/upload/error_handling.py", line 14, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/meta/upload/upload_api.py", line 103, in upload_files_to_customer_inbox
    delivery_service.deliver_files_for_case(
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/services/deliver_files/deliver_files_service/error_handling.py", line 18, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/services/deliver_files/deliver_files_service/deliver_files_service.py", line 82, in deliver_files_for_case
    self._add_trailblazer_tracking(case=case, job_id=job_id, dry_run=dry_run)
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/services/deliver_files/deliver_files_service/deliver_files_service.py", line 148, in _add_trailblazer_tracking
    self.analysis_service.add_upload_job(slurm_id=job_id, case_id=case.internal_id)
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/services/analysis_service/analysis_service.py", line 17, in add_upload_job
    analysis: TrailblazerAnalysis = self.analysis_client.get_latest_completed_analysis(case_id)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/apps/tb/api.py", line 82, in get_latest_completed_analysis
    raise TrailblazerAnalysisNotFound(f"No completed analysis found for case {case_id}")
cg.exc.TrailblazerAnalysisNotFound: No completed analysis found for case petmako

In TB: on case petmako perform action Mark as unhidden

Then: command cg upload -r -c petmako.

Output:

Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
----------------- UPLOAD -----------------
Case petmako exists in Status DB
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
--------------- DELIVERY REPORT ---------------
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Using analysis started at: 2024-10-29 12:01:13
Delivery report already in housekeeper: /home/proj/production/housekeeper-bundles/petmako/2024-10-29/delivery-report.html
/home/proj/production/customers/cust000/inbox/1234/kraken2_k2_pluspf_combined_reports.txt already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/kraken2_k2_pluspf_combined_reports.txt
/home/proj/production/customers/cust000/inbox/1234/kraken2_k2_pluspf.html already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/kraken2_k2_pluspf.html
/home/proj/production/customers/cust000/inbox/1234/centrifuge_p_compressed+h+v.html already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/centrifuge_p_compressed+h+v.html
/home/proj/production/customers/cust000/inbox/1234/bracken_bracken_combined_reports.txt already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/bracken_bracken_combined_reports.txt
/home/proj/production/customers/cust000/inbox/1234/centrifuge_p_compressed+h+v_combined_reports.txt already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/centrifuge_p_compressed+h+v_combined_reports.txt
/home/proj/production/customers/cust000/inbox/1234/multiqc_report.html already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/multiqc_report.html
/home/proj/production/customers/cust000/inbox/1234/nf_core_pipeline_software_mqc_versions.yml already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/nf_core_pipeline_software_mqc_versions.yml
/home/proj/production/customers/cust000/inbox/1234/multiqc_data.json already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/multiqc_data.json
/home/proj/production/customers/cust000/inbox/1234/delivery-report.html already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/delivery-report.html
/home/proj/production/customers/cust000/inbox/1234/IF-04-25_pe_k2_pluspf.kraken2.kraken2.report.txt already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/IF-04-25_pe_k2_pluspf.kraken2.kraken2.report.txt
/home/proj/production/customers/cust000/inbox/1234/IF-04-25_pe_bracken.bracken.tsv already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/IF-04-25_pe_bracken.bracken.tsv
/home/proj/production/customers/cust000/inbox/1234/IF-04-25_pe_p_compressed+h+v.centrifuge.txt already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/IF-04-25_pe_p_compressed+h+v.centrifuge.txt
Setting log dir to: /home/proj/production/rsync/petmako_241113_09_44_46_411326
Creating folder: /home/proj/production/rsync/petmako_241113_09_44_46_411326
Submit sbatch /home/proj/production/rsync/petmako_241113_09_44_46_411326/petmako_rsync.sh
Running command sbatch /home/proj/production/rsync/petmako_241113_09_44_46_411326/petmako_rsync.sh
Submitted batch job 7600294
Writing slurm jobs to /home/proj/production/rsync/petmako_241113_09_44_46_411326/slurm_job_ids.yaml
Transfer of case petmako started with SLURM job id 7600294
Upload of case petmako was successful. Setting uploaded at to 2024-11-13 09:44:48.334048
petmako - uploaded at set to 2024-11-13 09:44:48
petmako analysis has been successfully uploaded
diitaz93 commented 1 week ago

UPDATE:

The request sent from cg to Trailblazer is based on this:

f"analyses?case_id={case_id}&status[]={AnalysisStatus.COMPLETED}&limit=1"

In Trailblazer, the response is parsed into this model:

class AnalysesRequest(BaseModel):
    workflow: str = ""
    search: str | None = None
    page_size: int = Field(alias="pageSize", default=250)
    page: int = 1
    sort_field: AnalysisSortField = Field(alias="sortField", default=AnalysisSortField.STARTED_AT)
    sort_order: SortOrder = Field(alias="sortOrder", default=SortOrder.DESC)
    status: list[TrailblazerStatus] = []
    priority: list[TrailblazerPriority] = []
    type: list[TrailblazerTypes] = []
    has_comment: bool | None = Field(alias="hasComment", default=None)
    order_id: int | None = Field(alias="orderId", default=None)
    case_id: str | None = None
    delivered: bool | None = None
    include_hidden: bool | None = Field(alias="includeHidden", default=None)

This request is used to parse the TB database using:

    def _filter_analyses(self, request: AnalysesRequest) -> Query:
        filters: list[AnalysisFilter] = [
            AnalysisFilter.BY_WORKFLOW,
            AnalysisFilter.BY_HAS_COMMENT,
            AnalysisFilter.BY_ORDER_ID,
            AnalysisFilter.BY_PRIORITIES,
            AnalysisFilter.BY_STATUSES,
            AnalysisFilter.BY_TYPES,
            AnalysisFilter.BY_CASE_ID,
            AnalysisFilter.BY_SEARCH_TERM,
            AnalysisFilter.BY_IS_VISIBLE,
            AnalysisFilter.BY_DELIVERED,
            AnalysisFilter.SORTING,
        ]
        show_hidden = bool(request.search) or request.include_hidden
        return apply_analysis_filter(
            filter_functions=filters,
            analyses=self.get_query(Analysis),
            has_comment=request.has_comment,
            order_id=request.order_id,
            priorities=request.priority,
            statuses=request.status,
            types=request.type,
            case_id=request.case_id,
            workflow=request.workflow,
            search_term=request.search,
            show_hidden=show_hidden,
            sort_field=request.sort_field,
            sort_order=request.sort_order,
            delivered=request.delivered,
        )

Which assigns False to the attribute show_hidden and excludes hidden analyses from the query.