inab / WfExS-backend

Workflow Execution Service Backend
Apache License 2.0
16 stars 6 forks source link

Add several checks in the code to detect containers unavailable for the current hardware architecture #34

Open jmfernandez opened 1 year ago

jmfernandez commented 1 year ago

Thanks to the tests from @dcl10 some issues have been uncovered related to workflows which depend on container images which are not available for the current processor architecture.

A way to reproduce the chain of issues is trying to execute cosifer workflow, which depends on a single container prepared for x86_64 / amd64 architecture, in a different architecture like linux arm64.

cosifer "toy" workflow uses a single custom container which is only available for x86_64. WfExS-backend tries materializing the container by itself, and most probably it is doing wrongly despite the architecture mismatch, but it should have complained before even trying to run cwltool. So, when cwltool tries running it, it is surely failing because either the previously materialized container is for the wrong architecture, or because cwltool is not able to fetch any container suitable for the task. So, cwltool is returning an empty description of its outputs which is deserialized to None instead to a dictionary, and the code is failing trying to access key "class" because None is not a dictionary.

Also, the caching directory should have container images directory per supported architecture, so it can hold cached versions for x86_64 and arm64, in case the caching directory is used in an heterogeneous HPC environment.