search

ddbj / workflow-registry

Workflow registry published by DDBJ
https://ddbj.github.io/workflow-registry-browser/
Apache License 2.0
5 stars 3 forks source link

Fix endpoint for TRS #15

Closed kinow closed 1 year ago

kinow commented 2 years ago

I am experimenting running WfExS (cc @jmfernandez) with Yevis/DDBJ. Here's what I did so far:

# file: tests/yevis-test.yaml
trs_endpoint: https://ddbj.github.io/workflow-registry/
workflow_id: 0d2ae4c2-fe4c-48f7-811a-ac277776533e
version: 1.0.0
workflow_config:
  secure: false
# All the inputs must be URLs or CURIEs from identifiers.org
params:
outputs:

It had failed with an error about toolclass vs. tool_class, already fixed by @suecharo (thanks!).

But now it is failing with another error:

(venv) kinow@ranma:~/Development/python/workspace/WfExS-backend$ python WfExS-backend.py execute -W tests/yevis-test.yaml
2022-07-01 14:08:43,132 - [ERROR] FUSE filesystem command encfs not found. Please install it if you are going to use a secured staged workdir
* Command "execute".
    - Working directory will be /tmp/WfExS-workdirlu5yc1pnbackend/fade65fc-bee8-4ea0-8aa3-9b595a6c0c71
    - Instance fade65fc-bee8-4ea0-8aa3-9b595a6c0c71 (nickname 'handled cider') (to be used with -J)
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/WfExS-backend/WfExS-backend.py", line 568, in <module>
    main()
  File "/home/kinow/Development/python/workspace/WfExS-backend/WfExS-backend.py", line 550, in main
    stagedSetup = wfInstance.stageWorkDir()
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/workflow.py", line 1254, in stageWorkDir
    self.fetchWorkflow()
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/workflow.py", line 715, in fetchWorkflow
    i_workflow = self.getWorkflowRepoFromTRS(offline=offline)
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/workflow.py", line 2271, in getWorkflowRepoFromTRS
    _, trsFilesDir, trsFilesMeta, _ = self.wfexs.cacheFetch(INTERNAL_TRS_SCHEME_PREFIX + ':' + toolFilesURL, CacheType.TRS, offline)
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/wfexs_backend.py", line 852, in cacheFetch
    return self.cacheHandler.fetch(
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/cache_handler.py", line 638, in fetch
    raise nested_exception
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/cache_handler.py", line 619, in fetch
    raise che from nested_exception
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/cache_handler.py", line 582, in fetch
    inputKind, fetched_metadata_array, fetched_licences = schemeHandler(the_remote_file, tempCachedFilename, secContext=usableSecContext if usableSecContext else None)
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/fetchers/trs_files.py", line 102, in fetchTRSFiles
    _ , metaprimary, _ = fetchClassicURL(file_url, descriptorMeta)
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/fetchers/__init__.py", line 213, in fetchClassicURL
    raise FetcherException("Error fetching {} : {} {}".format(orig_remote_file, he.code, he.reason))
wfexs_backend.fetchers.FetcherException: Error fetching https://ddbj.github.io/workflow-registry/tools/0d2ae4c2-fe4c-48f7-811a-ac277776533e/versions/1.0.0/CWL/descriptor/https://zenodo.org/api/files/2422dda0-1bd9-4109-aa44-53d55fd934de/download-sra.cwl : 404 Not Found

Thanks! :+1:

suecharo commented 1 year ago

Hello @kinow -san, I apologize for the delayed response. This error appears to occur because WfExS-backend.py is expecting to be able to fetch the file itself at /tools/{id}/versions/{version_id}/{type}/descriptor/{relative_path} in TRS.

As stated at https://editor.swagger.io/?url=https://raw.githubusercontent.com/ga4gh/tool-registry-schemas/develop/openapi/openapi.yaml, "Descriptors can often include imports that refer to additional descriptors. This returns additional descriptors for the specified tool in the same or other directories that can be reached as a relative path. This endpoint can be useful for workflow engine implementations like cwltool to programmatically download all the descriptors for a tool and run it. This can optionally include other files described with FileWrappers such as test parameters and containerfiles.", the original TRS definition indeed expects that behavior.

However, in yevis, due to the specification that returns TRS responses via GitHub pages, we are unable to implement this behavior. Instead, we're including the download link in the path of the /files response.

Given yevis's constraints of not wanting to have a REST API server (or any server resource), this is quite a predicament. Any suggestions on how we could navigate this issue would be much appreciated. (@inutano)

kinow commented 1 year ago

Thanks for the reply @suecharo , no worries about delay :ok_man:

Given yevis's constraints of not wanting to have a REST API server (or any server resource), this is quite a predicament.

Hmm, can't think of any suggestion on how to workaround this issue. Maybe we can just document this for users of the endpoint?

Thanks!

jmfernandez commented 1 year ago

Hi @kinow (and @suecharo) ! If you provide me the WfExS-backend staging declaration file I can try figuring out a way to either detect or circumvent this kind of non-standard case in WfExS code.

kinow commented 1 year ago

Hi @jmfernandez ! :wave:

The WfExS file I used was this one from the top comment:

# file: tests/yevis-test.yaml
trs_endpoint: https://ddbj.github.io/workflow-registry/
workflow_id: 0d2ae4c2-fe4c-48f7-811a-ac277776533e
version: 1.0.0
workflow_config:
  secure: false
# All the inputs must be URLs or CURIEs from identifiers.org
params:
outputs:

Not sure if the syntax changed. The last time I tried it it was months ago. I think it'd be useful if you can find a way to circumvent or give some feedback to users about the response not matching the expected standard.

jmfernandez commented 1 year ago

I have just pushed a fix to the main branch of WfExS-backend. Hope this helps!

inutano commented 1 year ago

Thank you @kinow @suecharo and @jmfernandez !! I understand this is the limitation of our strategy to implement a registry with gh pages, a fully static website. I'm closing this, but as @kinow suggested, at least we need to mention this in our documentation. Thanks for your testing and additional contribution @kinow @jmfernandez 🙏 🙏