umccr / libica

:snake: Python SDK for Illumina Connected Analytics (ICA) :dragon:
MIT License
6 stars 4 forks source link

Cannot rename tool_cwl_file attribute on ProjectPipeline/createCwlPipeline endpoint #74

Closed alexiswl closed 1 year ago

alexiswl commented 2 years ago

Hello,

I would like to add additional files to the workflow through the tool_cwl_files form, see here for more infor.

tool_cwl_files takes in an array of files, but uses the basename of the file to place the file into the top directory of the workflow.

Some files ought to be nested instead. Is there a way to use the filename attribute, like one would on a curl --form parameter?

alexiswl commented 2 years ago

So this is what I tried

workflow_files = [
    workflow_file 
    for workflow_file in (Path(extraction_tempdir.name) / "bclconvert-with-qc-pipeline__4.0.3").rglob("*") 
    if workflow_file.is_file()
]

workflow_file = list(filter(lambda x: x.name == "workflow.cwl", workflow_files))[0]
tool_files = list(filter(lambda x: x.name != "workflow.cwl", workflow_files))

relative_dir_obj = os.open(str(Path(extraction_tempdir.name) / "bclconvert-with-qc-pipeline__4.0.3"), os.O_RDONLY)
def cwl_file_opener(cwl_file_path, flags):
    return os.open(cwl_file_path, flags, dir_fd=relative_dir_obj)

with libica.openapi.v2.ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = project_pipeline_api.ProjectPipelineApi(api_client)

    project_id = project_id

    code = code

    description = description

    workflow_cwl_file = open(workflow_file, "rb", opener=cwl_file_opener)

    parameters_xml_file = open("params.xml", "rb")

    analysis_storage_id = "3fab13dd-46e7-4b54-bb34-b80a01a99379"

    tool_cwl_files = [
        open(tool_file.relative_to(Path(extraction_tempdir.name) / "bclconvert-with-qc-pipeline__4.0.3"), "rb", opener=cwl_file_opener)
        for tool_file in tool_files
    ]

    try:
        # Create a CWL pipeline within a project.
        api_response = api_instance.create_cwl_pipeline(
            project_id=project_id, 
            code=code, 
            description=description, 
            workflow_cwl_file=workflow_cwl_file, 
            parameters_xml_file=parameters_xml_file, 
            analysis_storage_id=analysis_storage_id,
            tool_cwl_files=tool_cwl_files
        )
    except libica.openapi.v2.ApiException as e:
        raise ValueError("Exception when calling ProjectPipelineApi->create_cwl_pipeline: %s\n" % e)

However, was not able to set the filename attribute for any of the tool_cwl_files to relative paths, compare this to the curl command ran

curl_command_list = [
    "curl", 
    "--fail", "--silent", "--location", 
    "--request", "POST",
    "--header", "Accept: application/vnd.illumina.v3+json",
    "--header", f"Authorization: Bearer {icav2_access_token}", 
    "--header", "Content-Type: multipart/form-data",
    "--url", f"https://ica.illumina.com/ica/rest/api/projects/{project_id}/pipelines:createCwlPipeline",
    "--form", f"code={code}",
    "--form", f"description={description}", 
    "--form", f"workflowCwlFile=@{workflow_file};filename=workflow.cwl", 
    "--form", f"parametersXmlFile=@{parameters_xml_file};filename=params.xml;type=text/xml", 
    "--form", f"analysisStorageId={analysis_storage_id}"
]

for tool_file in tool_files:
    curl_command_list.extend([
        "--form", f"toolCwlFiles=@{tool_file};filename={tool_file.relative_to(Path(extraction_tempdir.name) / 'bclconvert-with-qc-pipeline__4.0.3')}"
    ])

create_pipeline_proc = run(curl_command_list, capture_output=True)
victorskl commented 1 year ago

This will be a bit tough. @alexiswl Form parameter, the filename field get hard bind to filename = os.path.basename(file_instance.name) at Line 551. Let me see whether we can intercept and override this; before sending the API call.

https://github.com/umccr-illumina/libica/blob/38b83ea44d253a3acea0ae38121033612926f402/libica/openapi/v2/api_client.py#L527-L558

victorskl commented 1 year ago

Actually. Easy! We can just remove basename wrapping there! i.e.

filename = file_instance.name

This allows filename to retain user pass-in as-is.

Let me see how I can make this avail! May be config option to ApiClient to on/off this feature.

victorskl commented 1 year ago

Also. Sharing tip on how to enter into "Sage Mode"

icav2_access_token = os.environ['ICAV2_ACCESS_TOKEN']
ica_url = "https://ica.illumina.com/ica/rest"

configuration = Configuration(
    host=ica_url,
    access_token=icav2_access_token,
)
configuration.debug = True
export ICAV2_ACCESS_TOKEN=blah

Then, run the script that use libica. You get what goes there for HTTP POST payload.

🙂

victorskl commented 1 year ago

This is now supported. Configure ApiClient as follows.

configuration = Configuration(
    host=ica_url,
    access_token=icav2_access_token,
    form_filename_basename=False,
)

See create_cwl_pipeline.py in examples for working CWL pipeline demo. The same approach should work for Nextflow pipeline, too. Happy to add NF example later, yes.