search

umccr-illumina / libica

:snake: Python SDK for Illumina Connected Analytics (ICA) :dragon:
MIT License
5 stars 4 forks source link

New v4 content type causing issues with return values on 'create_nextflow_analysis' #135

Closed alexiswl closed 3 months ago

alexiswl commented 3 months ago

Reprex

# Standard imports
from os import environ

# Libica imports
from libica.openapi.v2 import ApiClient, ApiException, Configuration
from libica.openapi.v2.api.project_analysis_api import ProjectAnalysisApi
from libica.openapi.v2.model.analysis import Analysis
from libica.openapi.v2.model.analysis_data_input import AnalysisDataInput
from libica.openapi.v2.model.analysis_output_mapping import AnalysisOutputMapping
from libica.openapi.v2.model.analysis_parameter_input import AnalysisParameterInput
from libica.openapi.v2.model.analysis_tag import AnalysisTag
from libica.openapi.v2.model.create_nextflow_analysis import CreateNextflowAnalysis
from libica.openapi.v2.model.nextflow_analysis_input import NextflowAnalysisInput

configuration = Configuration(
    host=environ['ICAV2_BASE_URL'],
    access_token=environ['ICAV2_ACCESS_TOKEN']
)

with ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = ProjectAnalysisApi(api_client)

project_id = "7595e8f2-32d3-4c76-a324-c6a85dae87b5"
nextflow_analysis = CreateNextflowAnalysis(
    user_reference='PTC-ctTSO-v2-launch-test',
    pipeline_id='fdef5902-3f50-4ee7-ae17-15d38d4b489c',
    tags=AnalysisTag(
        reference_tags=[],
        technical_tags=[
            'portal_run_id=20240308abcd1234'
        ],
        user_tags=[
            'subject_id=SBJ04405',
            'library_id=L2301368',
            'instrument_run_id=231116_A01052_0172_BHVLM5DSX7',
            'project_owner=UMCCR',
            'project_name=testing'
        ]
    ),
    analysis_input=NextflowAnalysisInput(
        inputs=[
            AnalysisDataInput(
                data_ids=['fol.58422302edd141213e0f08dc3cace45e'],
                parameter_code='run_folder'
            ),
            AnalysisDataInput(
                data_ids=['fil.10f0c3dd7cf14a88a8f608dc3c3824cb'],
                parameter_code='sample_sheet'
            ),
        ],
        parameters=[
            AnalysisParameterInput(
                code="StartsFromFastq",
                value="true",
            ),
            AnalysisParameterInput(
                code="sample_pair_ids",
                multi_value=["L2301368"]
            )
        ],
        activation_code_detail_id='7f03a57e-2cfc-4b35-9cbb-d19e6ce9984b',
        analysis_storage_id='3fab13dd-46e7-4b54-bb34-b80a01a99379',
        analysis_output=[
            AnalysisOutputMapping(
                source_path='out/',
                target_path='/ilmn_cttso_fastq_cache/20240308abcd1234/',
                target_project_id='7595e8f2-32d3-4c76-a324-c6a85dae87b5',
                type='FOLDER'
            )
        ]
    )
)

# Create and start an analysis for a CWL pipeline.
api_response: Analysis = api_instance.create_nextflow_analysis(
    project_id,
    nextflow_analysis
)

Traceback

/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/bin/python -X pycache_prefix=/home/alexiswl/.cache/JetBrains/RemoteDev-PY/_media_UMCCR_GitHub_orcabus/cpython-cache /home/alexiswl/.cache/JetBrains/RemoteDev/dist/43709941d6ca1_pycharm-professional-2023.3.2/plugins/python/helpers/pydev/pydevd.py --multiprocess --qt-support=auto --client 127.0.0.1 --port 46003 --file /home/alexiswl/.config/JetBrains/RemoteDev-PY/_media_UMCCR_GitHub_orcabus/scratches/scratch_6.py 
Connected to pydev debugger (build 233.13135.95)
Traceback (most recent call last):
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/model_utils.py", line 1453, in attempt_convert_item
    return deserialize_model(input_value, valid_class,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/model_utils.py", line 1373, in deserialize_model
    return model_class._new_from_openapi_data(**kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/model_utils.py", line 45, in wrapped_init
    return fn(_self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/model_utils.py", line 369, in _new_from_openapi_data
    return cls._from_openapi_data(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/model_utils.py", line 45, in wrapped_init
    return fn(_self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: AnalysisV4._from_openapi_data() missing 2 required positional arguments: 'owner' and 'tenant'
python-BaseException

This actually kicks off the workflow, so must fail trying to convert the output into AnalysisV4

alexiswl commented 3 months ago

If I run with 

# Create and start an analysis for a CWL pipeline.
api_response: Analysis = api_instance.create_nextflow_analysis(
    project_id,
    nextflow_analysis,
    _content_type="application/vnd.illumina.v3+json"
)

I get

Traceback (most recent call last):
  File "/home/alexiswl/.config/JetBrains/RemoteDev-PY/_media_UMCCR_GitHub_orcabus/scratches/scratch_6.py", line 76, in <module>
    api_response: Analysis = api_instance.create_nextflow_analysis(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/api/project_analysis_api.py", line 1230, in create_nextflow_analysis
    return self.create_nextflow_analysis_endpoint.call_with_http_info(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/api_client.py", line 854, in call_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/api_client.py", line 409, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/api_client.py", line 203, in __call_api
    raise e
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/api_client.py", line 196, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/api_client.py", line 455, in request
    return self.rest_client.POST(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/rest.py", line 268, in POST
    return self.request("POST", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/UMCCR/GitHub/orcabus/.bssh_manager_venv/lib/python3.11/site-packages/libica/openapi/v2/rest.py", line 227, in request
    raise ApiException(http_resp=r)
libica.openapi.v2.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Date': 'Fri, 08 Mar 2024 03:27:14 GMT', 'Content-Type': 'application/problem+json', 'Content-Length': '500', 'Connection': 'keep-alive', 'Set-Cookie': 'INGRESSCOOKIE=e462810085a8e0e72fc27896a1df01f6|82b2b547c6ffaa8346869fdb96dccf96; Max-Age=43200; Path=/ica/rest/api; HttpOnly', 'Strict-Transport-Security': 'max-age=31536000', 'X-Frame-Options': 'SAMEORIGIN', 'Referrer-Policy': 'origin-when-cross-origin', 'Content-Security-Policy': "default-src 'self' 'unsafe-inline' 'unsafe-eval' data: ws: wss: https://*.illumina.com https://*.googletagmanager.com https://*.google-analytics.com https://*.walkmeusercontent.com https://*.walkme.com https://*.amazonaws.com https://*.windows.net;", 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Methods': 'GET, PUT, POST, DELETE, PATCH, OPTIONS', 'Access-Control-Allow-Headers': 'DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization', 'Access-Control-Max-Age': '7200'})
HTTP response body: {"id":"2971c387-8f2b-4b03-a25e-538a0cb7491d","type":"about:blank","title":"ICA_API_004","status":400,"detail":"The versions provided in request headers 'Content-Type' and 'Accept' do not match.","instance":"http://ica.illumina.com/ica/rest/api/projects/7595e8f2-32d3-4c76-a324-c6a85dae87b5/analysis:nextflow","parameters":{},"timestamp":"2024-03-08T03:27:14Z","method":"POST","userId":"0c51d29a-8ffa-38e9-bf18-736330d7c65a","tenantName":"umccr-prod","tenantId":"1555b441-c3be-40b0-a8f0-fb9dc7500545"}
alexiswl commented 3 months ago

I tried specifying returning a v4 accept through the post request? But this still returned a v3 json

analysis_req_obj = requests.post(
    headers={
        "Authorization": f"Bearer {configuration.access_token}",
        "Content-Type": "application/vnd.illumina.v4+json",
        "Accept": "application/vnd.illumina.v4+json"
    },
    url=configuration.host + f"/api/projects/{project_id}/analysis:nextflow",
    data=json.dumps(recursively_build_open_api_body_from_libica_item(nextflow_analysis))
)
analysis_req_obj.json()

Yields

{
    "id": "474cc1fe-f4a9-488d-ba18-f5a3fb81d361",
    "timeCreated": "2024-03-08T06:32:21Z",
    "timeModified": "2024-03-08T06:32:22Z",
    "ownerId": "0c51d29a-8ffa-38e9-bf18-736330d7c65a",
    "tenantId": "1555b441-c3be-40b0-a8f0-fb9dc7500545",
    "tenantName": "umccr-prod",
    "reference": "PTC-ctTSO-v2-launch-test-DRAGEN TSO500 ctDNA RUO v2-1-1 3_8_2023 Official Analysis Software Pipeline-4bd243a0-2ee2-4731-a07b-3c80009a8ed9",
    "userReference": "PTC-ctTSO-v2-launch-test",
    "pipeline": {
        "id": "fdef5902-3f50-4ee7-ae17-15d38d4b489c",
        "timeCreated": "2023-03-09T00:44:06Z",
        "timeModified": "2023-03-09T00:54:30Z",
        "ownerId": "ee171059-4283-3acb-b0b0-34d5b356be3f",
        "tenantId": "25a4d4b2-ea16-4075-b09b-65ca3fee6d31",
        "tenantName": "ilmn-tso500",
        "code": "DRAGEN TSO500 ctDNA RUO v2-1-1 3_8_2023 Official Analysis Software Pipeline",
        "urn": "urn:ilmn:ica:pipeline:fdef5902-3f50-4ee7-ae17-15d38d4b489c#DRAGEN_TSO500_ctDNA_RUO_v2-1-1_3_8_2023_Official_Analysis_Software_Pipeline",
        "description": "Official Release DRAGEN TSO500 ctDNA RUO v2-1-1 3_8_2023 Official Analysis Software Pipeline\nDocker::docker-oncology.dockerhub.illumina.com/acadia/acadia-liquid-nextflow-dragen:ruo-ica-2.1.1.2",
        "language": "NEXTFLOW",
        "languageVersion": {
            "id": "b1585d18-f88c-4ca0-8d47-34f6c01eb6f3",
            "name": "22.04.3",
            "language": "NEXTFLOW"
        },
        "pipelineTags": {
            "technicalTags": []
        },
        "analysisStorage": {
            "id": "3fab13dd-46e7-4b54-bb34-b80a01a99379",
            "timeCreated": "2021-11-05T10:28:20Z",
            "timeModified": "2023-05-31T16:38:14Z",
            "ownerId": "8ec463f6-1acb-341b-b321-043c39d8716a",
            "tenantId": "f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3",
            "tenantName": "ica-cp-admin",
            "name": "Large",
            "description": "7.2TB"
        },
        "proprietary": true
    },
    "status": "REQUESTED",
    "analysisStorage": {
        "id": "3fab13dd-46e7-4b54-bb34-b80a01a99379",
        "timeCreated": "2021-11-05T10:28:20Z",
        "timeModified": "2023-05-31T16:38:14Z",
        "ownerId": "8ec463f6-1acb-341b-b321-043c39d8716a",
        "tenantId": "f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3",
        "tenantName": "ica-cp-admin",
        "name": "Large",
        "description": "7.2TB"
    },
    "analysisPriority": "LOW",
    "tags": {
        "technicalTags": [
            "portal_run_id=20240308abcd1234"
        ],
        "userTags": [
            "subject_id=SBJ04405",
            "library_id=L2301368",
            "instrument_run_id=231116_A01052_0172_BHVLM5DSX7",
            "project_owner=UMCCR",
            "project_name=testing"
        ],
        "referenceTags": []
    }
}

Which is still v3 json

alexiswl commented 3 months ago

Thought I'd do one last confirmation via the swagger API Page

curl -X 'POST' \
  'https://ica.illumina.com/ica/rest/api/projects/7595e8f2-32d3-4c76-a324-c6a85dae87b5/analysis:nextflow' \
  -H 'accept: application/vnd.illumina.v4+json' \
  -H 'Authorization: Bearer __JWT__' \
  -H 'Content-Type: application/vnd.illumina.v4+json' \
  -d '{
    "userReference": "PTC-ctTSO-v2-launch-test",
    "pipelineId": "fdef5902-3f50-4ee7-ae17-15d38d4b489c",
    "tags": {
        "technicalTags": [
            "portal_run_id=20240308abcd1234"
        ],
        "userTags": [
            "subject_id=SBJ04405",
            "library_id=L2301368",
            "instrument_run_id=231116_A01052_0172_BHVLM5DSX7",
            "project_owner=UMCCR",
            "project_name=testing"
        ],
        "referenceTags": []
    },
    "analysisInput": {
        "inputs": [
            {
                "parameterCode": "run_folder",
                "dataIds": [
                    "fol.58422302edd141213e0f08dc3cace45e"
                ]
            },
            {
                "parameterCode": "sample_sheet",
                "dataIds": [
                    "fil.94813f45b9e94977b0a308dc388cf24f"
                ]
            }
        ],
        "parameters": [
            {
                "code": "StartsFromFastq",
                "value": "true"
            },
            {
                "code": "sample_pair_ids",
                "multiValue": [
                    "L2301368"
                ]
            }
        ],
        "null": [
            {
                "sourcePath": "out/",
                "targetProjectId": "7595e8f2-32d3-4c76-a324-c6a85dae87b5",
                "targetPath": "/ilmn_cttso_fastq_cache/20240308abcd1234/",
                "type": "FOLDER"
            }
        ]
    }
}'

Had a 201 returncode with the following response body

{
  "id": "9bc738f5-dddb-4dac-8108-d03a0e6eab08",
  "timeCreated": "2024-03-08T06:58:41Z",
  "timeModified": "2024-03-08T06:58:41Z",
  "ownerId": "0c51d29a-8ffa-38e9-bf18-736330d7c65a",
  "tenantId": "1555b441-c3be-40b0-a8f0-fb9dc7500545",
  "tenantName": "umccr-prod",
  "reference": "PTC-ctTSO-v2-launch-test-DRAGEN TSO500 ctDNA RUO v2-1-1 3_8_2023 Official Analysis Software Pipeline-c96004db-3c01-43d7-bccb-f5740aef8f6f",
  "userReference": "PTC-ctTSO-v2-launch-test",
  "pipeline": {
    "id": "fdef5902-3f50-4ee7-ae17-15d38d4b489c",
    "timeCreated": "2023-03-09T00:44:06Z",
    "timeModified": "2023-03-09T00:54:30Z",
    "ownerId": "ee171059-4283-3acb-b0b0-34d5b356be3f",
    "tenantId": "25a4d4b2-ea16-4075-b09b-65ca3fee6d31",
    "tenantName": "ilmn-tso500",
    "code": "DRAGEN TSO500 ctDNA RUO v2-1-1 3_8_2023 Official Analysis Software Pipeline",
    "urn": "urn:ilmn:ica:pipeline:fdef5902-3f50-4ee7-ae17-15d38d4b489c#DRAGEN_TSO500_ctDNA_RUO_v2-1-1_3_8_2023_Official_Analysis_Software_Pipeline",
    "description": "Official Release DRAGEN TSO500 ctDNA RUO v2-1-1 3_8_2023 Official Analysis Software Pipeline\nDocker::docker-oncology.dockerhub.illumina.com/acadia/acadia-liquid-nextflow-dragen:ruo-ica-2.1.1.2",
    "language": "NEXTFLOW",
    "languageVersion": {
      "id": "b1585d18-f88c-4ca0-8d47-34f6c01eb6f3",
      "name": "22.04.3",
      "language": "NEXTFLOW"
    },
    "pipelineTags": {
      "technicalTags": []
    },
    "analysisStorage": {
      "id": "3fab13dd-46e7-4b54-bb34-b80a01a99379",
      "timeCreated": "2021-11-05T10:28:20Z",
      "timeModified": "2023-05-31T16:38:14Z",
      "ownerId": "8ec463f6-1acb-341b-b321-043c39d8716a",
      "tenantId": "f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3",
      "tenantName": "ica-cp-admin",
      "name": "Large",
      "description": "7.2TB"
    },
    "proprietary": true
  },
  "status": "REQUESTED",
  "analysisStorage": {
    "id": "3fab13dd-46e7-4b54-bb34-b80a01a99379",
    "timeCreated": "2021-11-05T10:28:20Z",
    "timeModified": "2023-05-31T16:38:14Z",
    "ownerId": "8ec463f6-1acb-341b-b321-043c39d8716a",
    "tenantId": "f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3",
    "tenantName": "ica-cp-admin",
    "name": "Large",
    "description": "7.2TB"
  },
  "analysisPriority": "LOW",
  "tags": {
    "technicalTags": [
      "portal_run_id=20240308abcd1234"
    ],
    "userTags": [
      "subject_id=SBJ04405",
      "library_id=L2301368",
      "instrument_run_id=231116_A01052_0172_BHVLM5DSX7",
      "project_owner=UMCCR",
      "project_name=testing"
    ],
    "referenceTags": []
  }
}

And the following response headers

 access-control-allow-credentials: true 
 access-control-allow-headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization 
 access-control-allow-methods: GET,PUT,POST,DELETE,PATCH,OPTIONS 
 access-control-allow-origin: * 
 access-control-max-age: 7200 
 connection: keep-alive 
 content-encoding: gzip 
 content-length: 964 
 content-security-policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' data: ws: wss: https://*.illumina.com https://*.googletagmanager.com https://*.google-analytics.com https://*.walkmeusercontent.com https://*.walkme.com https://*.amazonaws.com https://*.windows.net; 
 content-type: application/vnd.illumina.v4+json 
 date: Fri,08 Mar 2024 06:58:43 GMT 
 location: http://ica.illumina.com/ica/rest/api/execution/70908055 
 referrer-policy: origin-when-cross-origin 
 strict-transport-security: max-age=31536000 
 x-frame-options: SAMEORIGIN

Which has content-type: application/vnd.illumina.v4+json even though the response is a v3 json?

alexiswl commented 3 months ago

Also -- pressed wrong button, accidentally closed this issue

alexiswl commented 3 months ago

Ah this is the wrong issue to assign ilmn team to

alexiswl commented 3 months ago

Also copying over comments intended for https://github.com/umccr-illumina/ica_v2/issues/173

pr80ik commented 3 months ago

@jdevries-ilmn can you take a look.

@ckunard we should file a set ticket

ckunard commented 3 months ago

@alexiswl since this item and #173 seem to be connected here, I'm wondering if your comments above indicate that I should file a ticket to Illumina only on #173, or this one as well. Can you confirm?

alexiswl commented 3 months ago

@ckunard I don't think this is a 'libica' issue. This is an issue with the v4 endpoints returning a v3 json and declaring it as a v4 json.

An issue on https://github.com/umccr-illumina/ica_v2/issues/173 should definitely be filed.

alexiswl commented 3 months ago

Also should move commentary over to https://github.com/umccr-illumina/ica_v2/issues/173, I pasted a few of these code snippets here that should have been posted on https://github.com/umccr-illumina/ica_v2/issues/173 instead and copied them over there accordingly

ckunard commented 3 months ago

I will file a ticket for #173 and leave this one without an action for Illumina unless you clarify what should be documented for this one.

alexiswl commented 3 months ago

unless you clarify what should be documented for this one.

No this one is up to Victor on if he wants to add in a workaround / exception to the error here (or point me in the direction on how we can avoid the openapi generated code validating against the expected schema)

victorskl commented 3 months ago

Yup. Pls see PR #136 or code snippet in examples/issue_135.py.