search

snakemake / snakemake

This is the development home of the workflow management system Snakemake. For general information, see
https://snakemake.github.io
MIT License
2.25k stars 547 forks source link

Error running in Azure Kubernetes Service #1167

Open jayantad opened 3 years ago

jayantad commented 3 years ago

Snakemake version

Latest version

Describe the bug

The workflow deployed on Azure Kubernetes Services do not execute. The pod gets created but the status is shown as Error.

Logs

snakemake pod logs:

WorkflowError:
The Python 3 package 'azure-storage-blob' need to be installed to use Azure Storage remote() file functionality. No module named 'azure'
File "/opt/conda/envs/snakemake/lib/python3.9/importlib/__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed

Minimal example

snakemake --kubernetes \
    --default-remote-prefix snakemake-tutorial --default-remote-provider AzBlob \
    --envvars AZ_BLOB_ACCOUNT_URL AZ_BLOB_CREDENTIAL --use-conda --jobs 3

Additional context

I'm trying to evaluate whether snakemake workflows can be executed in Azure Kubernetes Services. I followed the steps as mentioned in the tutorial Auto-scaling Azure Kubernetes cluster without shared filesystem. I understand that this uses the default Snakemake image from Dockerhub.

arnikz commented 2 years ago

Hi,

I encountered the same issue (following the pip installation) and also tried the latest Azure SDK for Python (via Microsoft conda channel) without success...

kuytje commented 2 years ago

Hi I had the same problem, and fixed it temporary using the --container-image (https://stackoverflow.com/questions/68681883/snakemake-auto-scaling-azure-kubernetes-cluster-without-shared-filesystem-fail)

Now the docker.io/snakemake/snakemake:latest seems to be deleted and most other images I can find on dockerhub bring me back to the error above.

--edit-- I found out that a snakemake version in docker that works is --container-image docker.io/snakemake/snakemake:v6.1.1. Somehow the dependency is not picked up anymore in many later versions. I cannot find any difference in v6.1.1 (works) and for example v6.6.1 (doesn't work)

milesroberts-123 commented 1 year ago

I get the same error trying to go through the Azure Kubernetes Service snakemake tutorial :(

It appears that not even using --container-image docker.io/snakemake/snakemake:v6.1.1 in the snakemake call doesn't works anymore as of this post.

I just get the error: azure.core.exceptions.ResourceNotFoundError: The specified container does not exist.

Guess the solution is to just keep trying snakemake versions on docker that do work?

EDIT

My solution was to create my own snakemake image including the azure-storage-blob dependency to pass to kubernetes.

Here is a tutorial outlining my thought process and including a modified tutorial dataset:

https://github.com/milesroberts-123/snakemake-aks-tutorial

And a working command using my custom snakemake image:

snakemake --kubernetes --container-image docker.io/milesroberts/snakemake-aks:latest --default-remote-prefix snakemake-tutorial --default-remote-provider AzBlob --envvars AZ_BLOB_ACCOUNT_URL AZ_BLOB_CREDENTIAL --use-conda --jobs 3

I guess anyone could take advantage of this new image if they wanted