S3 backup issue tracker

[marcocastignoli]

We discussed in the meeting today about the tasks left about the S3 and Filebase IPFS sync. This is an issue to track the tasks regarding this.

• [ ] Setup IPNS update task on GCP that has to run after aws s3 completes
  • There is no straightforward option to trigger another Cloud Run Job / Cloud Function after a Cloud Run jobs finishes. I found a solution that uses LogRouter to forward sourcify-staging-s3-sync logs (filtered by "has completed successfully", probably there is a better filter) to a PubSub topic. Every time this PubSub topic receives a message a Cloud Function is executed.
  • [ ] Remember to delete GPC test entities: LogRouter,Cloud Function,PubSubTopic
• [ ] The "Generate CID" button was not working on Filebase. Check with Filebase team about this. We assume this is the root CID of the repo. Eventually we need to have the root CID to publish this as the IPNS value.
  • I contacted Filebase support, waiting for a response
  • They told me that there is a limit for buckets with max 25k files in order to generate the CID, I asked if they can do something specifically for us; waiting for a response
• [ ] Setup regular s3 syncs with schedule
  • [x] Setup staging schedule
• [x] Do some cleaning on the GCP jobs (rename, delete)
• [x] Fix the s3-backup-check test on CircleCI running daily. Since we don't use the Amazon S3 bucket as backup anymore, we should point this check to the Filebase S3. https://github.com/ethereum/sourcify/pull/1580

[marcocastignoli]

Updates on this:

• Staging s3 sync runs inside a Cloud run job daily
• Production s3 sync is running inside a VM with a service I made
• I pinged multiple times Filebase but I'm not getting a response regarding "Generate CID" and "IPNS update"

[manuelwedler]

Why is staging and production differently set up (job vs service)?

[manuelwedler]

I really wonder if we can use Filebase then. If we cannot have the root CID and IPNS name, it's not possible to address the root of the repo.

I am thinking what the problem was with our old IPFS setup. The main issue seems to me that we generated the root CID every day again. That means that the hash of each file of the repo was calculated every day. That's quite expensive and as we saw prone to erros. I wonder if we could implement a StorageService around IPFS's MFS feature. That would be much more efficient as not each hash would need to be calculated for generating the root. But this would also add a lot more complexity of course and infrastructure to maintain.

[marcocastignoli]

Why is staging and production differently set up (job vs service)?

Because production takes more than 24 hours to complete, and the maximum time for a job is 24 hours

I wonder if we could implement a StorageService around IPFS's MFS feature.

We already had this, and it wasn't working properly. Here the commit in which we removed it: https://github.com/ethereum/sourcify/commit/e3ae40c3a57989554d4cd3f75132bbc507185446

[kuzdogan]

I guess we can't get the root CID of the repository so IPNS doesn't seem to be an option.

The important thing for now is that the files of the verified contracts are pinned on IPFS individually, and it seems this is happening now regularly, although the whole thing takes longer than a day.

Let's just call this done for now and not go down the whole MFS road. It will take too long and we've already spend a lot of time on this. The above requirement is for now enough. We provide better ways to get the whole repo anyways with #1586

So once #1602 and #1586 are ready let's move forward with communicating all the changes in docs (#1588) and that's enough for now

I'm closing the issue but feel free to continue the discussion

[kuzdogan]

Why is staging and production differently set up (job vs service)?

Just alluding to this, I think they should be created in the same way, i.e. through a VM. The staging VM can be a much cheaper one @marcocastignoli

[marcocastignoli]

I can move the staging environment to a VM, I'll add a task in the backlog