Cannot rename tool_cwl_file attribute on ProjectPipeline/createCwlPipeline endpoint

[alexiswl]

Hello,

I would like to add additional files to the workflow through the tool_cwl_files form, see here for more infor.

tool_cwl_files takes in an array of files, but uses the basename of the file to place the file into the top directory of the workflow.

Some files ought to be nested instead. Is there a way to use the filename attribute, like one would on a curl --form parameter?

[alexiswl]

So this is what I tried

workflow_files = [
    workflow_file 
    for workflow_file in (Path(extraction_tempdir.name) / "bclconvert-with-qc-pipeline__4.0.3").rglob("*") 
    if workflow_file.is_file()
]

workflow_file = list(filter(lambda x: x.name == "workflow.cwl", workflow_files))[0]
tool_files = list(filter(lambda x: x.name != "workflow.cwl", workflow_files))

relative_dir_obj = os.open(str(Path(extraction_tempdir.name) / "bclconvert-with-qc-pipeline__4.0.3"), os.O_RDONLY)
def cwl_file_opener(cwl_file_path, flags):
    return os.open(cwl_file_path, flags, dir_fd=relative_dir_obj)

with libica.openapi.v2.ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = project_pipeline_api.ProjectPipelineApi(api_client)

    project_id = project_id

    code = code

    description = description

    workflow_cwl_file = open(workflow_file, "rb", opener=cwl_file_opener)

    parameters_xml_file = open("params.xml", "rb")

    analysis_storage_id = "3fab13dd-46e7-4b54-bb34-b80a01a99379"

    tool_cwl_files = [
        open(tool_file.relative_to(Path(extraction_tempdir.name) / "bclconvert-with-qc-pipeline__4.0.3"), "rb", opener=cwl_file_opener)
        for tool_file in tool_files
    ]

    try:
        # Create a CWL pipeline within a project.
        api_response = api_instance.create_cwl_pipeline(
            project_id=project_id, 
            code=code, 
            description=description, 
            workflow_cwl_file=workflow_cwl_file, 
            parameters_xml_file=parameters_xml_file, 
            analysis_storage_id=analysis_storage_id,
            tool_cwl_files=tool_cwl_files
        )
    except libica.openapi.v2.ApiException as e:
        raise ValueError("Exception when calling ProjectPipelineApi->create_cwl_pipeline: %s\n" % e)

However, was not able to set the filename attribute for any of the tool_cwl_files to relative paths, compare this to the curl command ran

curl_command_list = [
    "curl", 
    "--fail", "--silent", "--location", 
    "--request", "POST",
    "--header", "Accept: application/vnd.illumina.v3+json",
    "--header", f"Authorization: Bearer {icav2_access_token}", 
    "--header", "Content-Type: multipart/form-data",
    "--url", f"https://ica.illumina.com/ica/rest/api/projects/{project_id}/pipelines:createCwlPipeline",
    "--form", f"code={code}",
    "--form", f"description={description}", 
    "--form", f"workflowCwlFile=@{workflow_file};filename=workflow.cwl", 
    "--form", f"parametersXmlFile=@{parameters_xml_file};filename=params.xml;type=text/xml", 
    "--form", f"analysisStorageId={analysis_storage_id}"
]

for tool_file in tool_files:
    curl_command_list.extend([
        "--form", f"toolCwlFiles=@{tool_file};filename={tool_file.relative_to(Path(extraction_tempdir.name) / 'bclconvert-with-qc-pipeline__4.0.3')}"
    ])

create_pipeline_proc = run(curl_command_list, capture_output=True)

[victorskl]

This will be a bit tough. @alexiswl Form parameter, the filename field get hard bind to filename = os.path.basename(file_instance.name) at Line 551. Let me see whether we can intercept and override this; before sending the API call.

https://github.com/umccr-illumina/libica/blob/38b83ea44d253a3acea0ae38121033612926f402/libica/openapi/v2/api_client.py#L527-L558

[victorskl]

Actually. Easy! We can just remove basename wrapping there! i.e.

filename = file_instance.name

This allows filename to retain user pass-in as-is.

Let me see how I can make this avail! May be config option to ApiClient to on/off this feature.

[victorskl]

Also. Sharing tip on how to enter into "Sage Mode"

icav2_access_token = os.environ['ICAV2_ACCESS_TOKEN']
ica_url = "https://ica.illumina.com/ica/rest"

configuration = Configuration(
    host=ica_url,
    access_token=icav2_access_token,
)
configuration.debug = True

export ICAV2_ACCESS_TOKEN=blah

Then, run the script that use libica. You get what goes there for HTTP POST payload.

🙂

[victorskl]

This is now supported. Configure ApiClient as follows.

configuration = Configuration(
    host=ica_url,
    access_token=icav2_access_token,
    form_filename_basename=False,
)

See create_cwl_pipeline.py in examples for working CWL pipeline demo. The same approach should work for Nextflow pipeline, too. Happy to add NF example later, yes.