search

snakemake / snakemake-storage-plugin-gcs

A Snakemake storage plugin for Google Cloud Storage
MIT License
2 stars 4 forks source link

Error trying to use gcs storage plugin #18

Open jairav opened 4 months ago

jairav commented 4 months ago

Attempting to use the new gcs storage plugin using the default provider options. Running on Snakemake 8.4.7, python 3.12.1.

This seems to be an issue with my argument. The bucket exists, authenticates off gcloud cli, and works fine on Snakemake 7.32.4

I have tried to remove the dash from the bucket name but it gives the same error.

(snakemake) % snakemake --executor kubernetes --software-deployment-method conda --default-storage-provider gcs --default-storage-prefix test-bucket --storage-gcs-project test-sandbox  
WorkflowError:
Error applying default storage provider gcs. Make sure to provide a valid --default-storage-prefix (see https://snakemake.github.io/snakemake-plugin-catalog/plugins/storage/{provider}.html). {validation_res}
jairav commented 4 months ago

Also worth noting that Error message URL does not resolve.

vsoch commented 4 months ago

Looks like it's missing a format string! I was never able to understand the logic of these plugins (and get this one to work myself) so we might need to tag team with @johanneskoester. I've been using aws s3 instead.

jairav commented 4 months ago

I think I have a solution. @vsoch It appears to just be that the query string used is "gcs://" where it should be "gs://"

I have created a fork of the plugin here which works on a simple workflow: 


rule all:
    input:
        storage.gcs("gs://bkt-test-tmp-100/blah.txt")
    output:
        storage.gcs("gs://bkt-test-tmp-100/scratch/SUCCESS.txt")
    shell:
        """
        cp {input} {output} 
        """

I can create a PR with this fork. I also updated the pyproject.toml file as mentioned in #19 .