Upload artifacts on a custom AWS S3 (with CloudFront)

[cedoor]

Description:

The currently available public CDNs turn out to be rather slow. Providing a default private CDN with high performance would allow libraries used in production to rely on a more stable and efficient infrastructure.

A Github workflow could automatically upload the artifacts and other relevant files to an AWS S3 instance, and speed up distribution with CloudFront.

Tasks:

• [x] Create a Github workflow to upload the SNARK artifacts on AWS
• [x] #94
• [ ] #95

[sripwoud]

I think we knew that time would come ^^.

have you started working on this already?

• AWS S3 instance: what do you have in mind? Just a static files server with that structure: /project/version/file ? Plus maybe an extra endpoint: GET /project/versions ?

• Cloudflare CDN: do you really mean Cloudflare? I am no expert about the differences. But if we are to use the AWS PSE/EF resources, why not S3 with Cloudfront (only AWS services for easier integration and maintenance) ?

[cedoor]

Hey @sripwoud!

1. I think that's great! the extra endpoint returns the latest right?
2. Yess, my bad, it's Cloudfront. I fixed it in the description.

[sripwoud]

my idea was that GET /project/versions would return the list of versions for a given project. But this would require having a dedicated API server for that http endpoint, (and not just the bucket where we can simply copy assets to).

But that is maybe not so useful in the end. We can get that info from the npm registry responses https://registry.npmjs.org/@zk-kit/semaphore-artifacts anyway.

[cedoor]

my idea was that GET /project/versions would return the list of versions for a given project. But this would require having a dedicated API server for that http endpoint, (and not just the bucket where we can simply copy assets to).

But that is maybe not so useful in the end. We can get that info from the npm registry responses https://registry.npmjs.org/@zk-kit/semaphore-artifacts anyway.

I see, yes I agree that's something we can get from the NPM API. We do need an endpoint that points to the latest tho. Wdyt?

[numtel]

So, as I was figuring out where to put artifacts for circuitscan.org, I originally thought I'd use NPM then came to the same conclusion that I can't be sticking giant archives onto NPM.

I haven't look through the code for this but I've been making a couple Cloudformation templates lately that are similar to this job with the help of ChatGPT. Go ahead an take this for a spin: https://gist.github.com/numtel/7492ee983bcf27501411c9e4769786a5

Edit: looks like lambda max post size is 6mb but you're probably better off just creating a signed url for the upload from the client: https://chatgpt.com/share/adbd600c-1269-4038-98e1-26a3515ec436

[numtel]

my idea was that GET /project/versions would return the list of versions for a given project. But this would require having a dedicated API server for that http endpoint, (and not just the bucket where we can simply copy assets to).

How often does a project update its artifacts? If it never happens concurrently, there's no problem just putting this file in the bucket and updating it as necessary. I originally had a lambda to do this in Circuitscan but then I realized I could cut costs if I just stuck the data on s3.

[sripwoud]

So, as I was figuring out where to put artifacts for circuitscan.org, I originally thought I'd use NPM then came to the same conclusion that I can't be sticking giant archives onto NPM.

I haven't look through the code for this but I've been making a couple Cloudformation templates lately that are similar to this job with the help of ChatGPT. Go ahead an take this for a spin: https://gist.github.com/numtel/7492ee983bcf27501411c9e4769786a5

Edit: looks like lambda max post size is 6mb but you're probably better off just creating a signed url for the upload from the client: https://chatgpt.com/share/adbd600c-1269-4038-98e1-26a3515ec436

Hi @numtel Thanks for sharing. I can't give much feedback on the AWS stuff and tell how much it differs from what we are doing. @ntampakas is your guy here

[sripwoud]

How often does a project update its artifacts?

We came up with a rough estimation of releasing 5 to 10 new versions per year. Each package has around between 1 to 20 artifacts x 2. Only 3 packages now.

Updates are triggered by new pkg releases which are triggered by PR merge events. So unless concurrent PR merges happen we are safe no? cc @cedoor what do you think?

How much artifacts are you dealing with on https://circuitscan.org/

[cedoor]

cc @cedoor what do you think?

I agree, we'll have even less than 5 to 10 new versions per year IMO, at least now.

[numtel]

How much artifacts are you dealing with on https://circuitscan.org/

I'm aiming to catalog every circuit verifier on every chain. It's all still under construction though. Working on detailed roadmap today as part of my EF grant application. (edit: If you have any feedback: check the sheet)

Afaict, Circuitscan could be considered a superset of this repo since it does provide the build artifacts of every verified verifier. Already, the frontend will download the WASM+zkey out of the build zip for proof generation or developers can download the full build zip to incorporate proof generation/verification into their apps.

The structure of the catalog is yet to be determined though. In addition to queries by contract address, having a directory like https://l2beat.com/zk-catalog seems helpful.