Closed Rnsras closed 6 months ago
Hi @Rnsras !
Thanks for reporting this. At the moment, we are stuck between two worlds.
dsub
providers google-v2
and google-cls-v2
depend on, don't support Artifact Registrydsub
doesn't yet fully support, does support Artifact RegistryTo be clear, the change for the google-v2
and google-cls-v2
providers would need to be made by Google in the Life Sciences API. dsub
just passes the image through the API, and it is on-VM harness code from Google that sets things up and does the docker pull
.
I think the best workaround at the moment would be to use Container Registry and push your dependent images to asia.gcr.io
.
We are still working on the google-batch
provider, but can't provide a timeline at the moment for feature parity with the existing providers.
so, real problem is Life Sciences service (google-cls-v2) not support Artifact Repository yet ?
thanks for reply
You can't conveniently use Artifact Repository, but you can use it if you enable mapping it to gcr.io and configure it in a very specific way. See our discussion here: https://github.com/DataBiosphere/dsub/issues/255 and the guide here.
got it, thanks.
it's really helpful.
@mbookman @wnojopra Any updates on ETA for batch and full artifact registry support please As @carbocation discovered, egress charges are now getting pretty significant and would like to move to batch / AR earliest possible. Thanks
Hey @rivershah and @carbocation !
I just went looking into this issue again and realize that my original assertion about the Life Sciences API not supporting Artifact Registry was wrong. It does indeed support Artifact Registry (since late 2020 / early 2021). I'm really sorry about getting that wrong!
I thought that I tested this back in March when you originally reported the problem, but I realize that I actually misunderstood your report.
The error that you reported:
when I use --image on Artifact Repository,
it would be error: denied: Permission "artifactregistry.repositories.downloadArtifacts"
I assumed you meant with either the google-v2
or google-cls-v2
provider. I believe that I found an open request for the Life Sciences API to support Artifact Registry and took that and your (misunderstood) report for truth.
I now believe you meant with the local
provider. I have tested this and reproduced that error.
I have now tested using the google-v2
and google-cls-v2
with an image in artifact registry and it worked fine.
So coming back to the original request - are you looking for support for Artifact Registry in the local
provider or one of the google-v2
or google-cls-v2
providers? The latter should work.
Please let us know. Thanks!
@mbookman No, Artifact Registry does not work for me with google-cls-v2 unless I use my fork. (as discussed here) This two-line commit allows me to use the new URLs from Artifact Registry, without having to go through the hoops to create the gcr.io-compatible URLs. dsub uses string matching and the normal Artifact Registry URLs don't match the string that dsub expects. You can see the change that is needed in my ~two-line commit at that link. At least, according to github, my fork is 1 commit ahead and 0 commits behind your repo, so I don't believe this is fixed on the public-facing side of your code.
Got it. @wnojopra and I were looking at that and didn't see use-private-address
mentioned in this issue, so weren't sure. That's an easy change for us to roll into the next release.
Thanks!
Hello,
when I use --image on Artifact Repository,
it would be error:
denied: Permission "artifactregistry.repositories.downloadArtifacts"
the reason is that when dsub starts the machine, it will deploy the image in the form of
docker pull [image]
,but Artifact Registry needs to set
gcloud auth configure-docker asia-east1-docker.pkg.dev
before pulling image to make docker pull normally.what is the solution to this error?
Thanks.