search

DataBiosphere / dsub

Open-source command-line tool to run batch computing tasks and workflows on backend services such as Google Cloud.
Apache License 2.0
265 stars 44 forks source link

Problem with --image on Artifact Repository #257

Closed Rnsras closed 6 months ago

Rnsras commented 1 year ago

Hello,

when I use --image on Artifact Repository,

it would be error: denied: Permission "artifactregistry.repositories.downloadArtifacts"

the reason is that when dsub starts the machine, it will deploy the image in the form of docker pull [image],

but Artifact Registry needs to set gcloud auth configure-docker asia-east1-docker.pkg.dev before pulling image to make docker pull normally.

what is the solution to this error?

Thanks.

mbookman commented 1 year ago

Hi @Rnsras !

Thanks for reporting this. At the moment, we are stuck between two worlds.

To be clear, the change for the google-v2 and google-cls-v2 providers would need to be made by Google in the Life Sciences API. dsub just passes the image through the API, and it is on-VM harness code from Google that sets things up and does the docker pull.

I think the best workaround at the moment would be to use Container Registry and push your dependent images to asia.gcr.io.

We are still working on the google-batch provider, but can't provide a timeline at the moment for feature parity with the existing providers.

Rnsras commented 1 year ago

so, real problem is Life Sciences service (google-cls-v2) not support Artifact Repository yet ?

thanks for reply

carbocation commented 1 year ago

You can't conveniently use Artifact Repository, but you can use it if you enable mapping it to gcr.io and configure it in a very specific way. See our discussion here: https://github.com/DataBiosphere/dsub/issues/255 and the guide here.

Rnsras commented 1 year ago

got it, thanks.

it's really helpful.

rivershah commented 1 year ago

@mbookman @wnojopra Any updates on ETA for batch and full artifact registry support please As @carbocation discovered, egress charges are now getting pretty significant and would like to move to batch / AR earliest possible. Thanks

mbookman commented 1 year ago

Hey @rivershah and @carbocation !

I just went looking into this issue again and realize that my original assertion about the Life Sciences API not supporting Artifact Registry was wrong. It does indeed support Artifact Registry (since late 2020 / early 2021). I'm really sorry about getting that wrong!

I thought that I tested this back in March when you originally reported the problem, but I realize that I actually misunderstood your report.

The error that you reported:

when I use --image on Artifact Repository,

it would be error: denied: Permission "artifactregistry.repositories.downloadArtifacts"

I assumed you meant with either the google-v2 or google-cls-v2 provider. I believe that I found an open request for the Life Sciences API to support Artifact Registry and took that and your (misunderstood) report for truth.

I now believe you meant with the local provider. I have tested this and reproduced that error.

I have now tested using the google-v2 and google-cls-v2 with an image in artifact registry and it worked fine.

So coming back to the original request - are you looking for support for Artifact Registry in the local provider or one of the google-v2 or google-cls-v2 providers? The latter should work.

Please let us know. Thanks!

carbocation commented 1 year ago

@mbookman No, Artifact Registry does not work for me with google-cls-v2 unless I use my fork. (as discussed here) This two-line commit allows me to use the new URLs from Artifact Registry, without having to go through the hoops to create the gcr.io-compatible URLs. dsub uses string matching and the normal Artifact Registry URLs don't match the string that dsub expects. You can see the change that is needed in my ~two-line commit at that link. At least, according to github, my fork is 1 commit ahead and 0 commits behind your repo, so I don't believe this is fixed on the public-facing side of your code.

mbookman commented 1 year ago

Got it. @wnojopra and I were looking at that and didn't see use-private-address mentioned in this issue, so weren't sure. That's an easy change for us to roll into the next release.

Thanks!

mbookman commented 11 months ago

With 0.4.10, we have updated the support for Artifact Registry for the Google providers as per #255. PTAL and let us know if you are still having issues.

Thanks!