Stop temporarily running IPFS nodes and pin externally

[kuzdogan]

As we are facing resource overload, we need to turn off the IPFS node for some time. For that we'll temporarily outsource the pinning externally. The downside will be, we won't be able to update the IPNS for a while with newly verified contracts, but we will pin the newly verified contracts separately.

Preliminary

Do these before starting the rest:

1. Persist the data

• [x] Persist the .ipfs folder and do not necessarily do a complete ipfs add when the ipfs container starts running. This is actually long overdue, and will make sure we can pick up where we left, in case something goes wrong in the next steps, and have to turn on the ipfs-server again.

2. Check performance increase

• [x] See if the server gets relief turning off IPFS nodes

Server

• [x] Create external pinning service account
• We probably need to first copy the .ipfs folder before shutting down the container, or is there a way to persist it on the go. I.e. will those files vanish if we change the ipfs.yaml and restart the container?

Migrating the pinning

• [x] Pin everything on an external pinning service. We have several options:
  • If it is possible to do an ipfs add via scp etc. such that the process will run on another host but the files will be pulled from the server machine. This will not consume extra resources on the server and we don't need to copy everything.
  • Directly provide the CID to the pinning service and it will pull from the local node. This will use resources from the local node and we won't be able to turn it off at the time. But we can't turn off the node until everything gets pinned anyway.
  • Maybe test pinning from the node and see how many resources it takes. Make sure we can cancel pinning otherwise we'll just end up pinning everything without being able to test and stop.
  • Copy the files directly from the server or from S3 to another computer and push to pin from there.
  • See if we can do this with a script, or directly running a node with ipfs add with external pinning ipfs pin remote add --service=estuary $rootHash --background --name=$pinName

Migrating the IPNS

• [x] See if we can still provide the IPNS temporarily via an external provider. For the IPNS to resolve, we have to run our node with the keys. Or maybe there's a way to publish the public key --> CID mapping once and then the node can go offline?

Changing how we pin in the server

• [x] Change the pinning in the current server to not place files under MFS but to pin them directly in the external pinning.

Finally

• [x] Check if everything works and turn off the ipfs-server.

Gateway

• [x] Also replace the local gateway with an external gateway. The ipfs-gateway is also consuming quite a lot of resources and we should outsource this temporarily too.
  • Optionally, first check from the IPFS Web UI, how much bandwidth it consumed to get an estimation of costs. If the Web UI is available.

_{View in Huly HI-447}

[marcocastignoli]

After stopping both IPFS nodes, now the server has at 60-80% of the full CPU capacity. Before it was always at > than 100%.

[kuzdogan]

Pinned an up-to-date version of the repository on a local node and published under the IPNS https://ipfs.io/ipns/repo.sourcify.dev/

It wasn't possible to directly import a CID to pin in Infura, and their ipfs-copy didn't work well. For web3.storage, waiting for approval to use the bulk pinning API.

Closing it for now as local pinning seems to be sufficient for now. Will check back next week.