mirror packages on google cloud storage

hdl / conda-eda

Conda recipes for FPGA EDA tools for simulation, synthesis, place and route and bitstream generation.

https://anaconda.org/LiteX-Hub

Apache License 2.0

94 stars 27 forks source link

mirror packages on google cloud storage #204

Open proppy opened 2 years ago

proppy commented 2 years ago

Since conda has the capability of installing package from arbitrary static http host (as long as it can fetch the index), we could consider mirroring the packages under a google cloud storage bucket.

That would have the following advantages:

no more arbitrary quota limitation
faster fetch from GCP infra (incl dedicated runners #184)
more relevant package name (@mithro suggested https://conda.hdl.tools as the canonical channel name)

proppy commented 2 years ago

@kgugala looks like there is already a bucket named hdl-conda that we could reuse.

proppy commented 2 years ago

depending if we want to land this before or after #184, we could use either https://github.com/google-github-actions/upload-cloud-storage#authenticating-via-workload-identity-federation or https://github.com/google-github-actions/upload-cloud-storage#via-application-default-credentials for authorizing the action to access the bucket.

proppy commented 1 year ago

@umarcor as discussed with @mithro on https://github.com/hdl/conda-eda/issues/184#issuecomment-1052923605, do you think we can get a bucket set-up on github-hdl with either https://cloud.google.com/iam/docs/workload-identity-federation or https://cloud.google.com/iam/docs/creating-managing-service-accounts authorization scheme set so that we can upload artefact from the Upload packages workflow?

proppy commented 1 year ago

@PiotrZierhoffer @ajelinski @AdamOlech can we get a bucket created in the same project that's currently running the runners?

umarcor commented 1 year ago

I can see the settings of github-hdl (I think). There, in Compute Engine, I can see the gha-runner-coordinator-conda-eda VM, which I believe is the runner (coordinator) used in this repository; and the Disks attached to it. However, Section Cloud Storage > Buckets is empty (I cannot see anything at least). I can neither see the container images from hdl/containers. If I browse gcr.io/hdl-containers I can see the images (left), but the "selected project" is empty. If I set the project to github-hdl, it seems that the feature is disabled (right):

I don't know who can clarify whether the images from hdl/containers and the runner in this repo belong to the same project and whether that project is github-hdl. I believe it should be the same (a single) project and we should be able to create a bucket there. Yet, I'm unsure.