Outline blocking dependencies on centralised tools (e.g. etherscan)

[SidestreamColdMelon]

Goal

Checklists do not enforce processes that are blocked in case a single service is down

Context

Recently, spell team experienced downtime of etherscan, which caused a multi-hour delay in the spell handover and later confusion among delegates on why contract is not verified on etherscan. The delay was resolved by unanimous agreement within the spell team to proceed with an alternative verifier service and later still use etherscan to verify contract and resolve the confusion. Another reason to not depend on a single service is of course security: it's much easier to compromise a single crucial service documented in the process, than try to attack multiple independent services at the same time.

In order to prepare to such events, we should 1) evaluate existing dependencies 2) evaluate potential circumventions 3) proceed with removing dependencies one-by-one.

Todo

• [x] List existing dependencies found in the checklists / known processes
• [x] List potential circumventions (general or applicable to each specific dependency)
• [x] Create first specific issue to remove dependency on etherscan

[SidestreamColdMelon]

Existing dependencies found in the checklists / known processes

• https://docs.google.com/spreadsheets/

  • Get list of planned instructions from the Exec Sheet https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L6-L7

• http://github.com

  • Open pull request at https://github.com/makerdao/spells-mainnet/
  • Get latest version of checklists at https://github.com/makerdao/pe-checklists/
  • Get Exec Doc from https://github.com/makerdao/community/ https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L320
  • Check CI tests at https://github.com/makerdao/spells-mainnet/ https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L302-L303
  • References to other github repos in the checklists
    • DSS Exec Library https://github.com/makerdao/dss-exec-lib/, 40 occurrences, e.g.: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L38
    • Multi Collateral Dai https://github.com/makerdao/dss/, 2 occurrences, e.g.: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L57
    • Mainnet spells https://github.com/makerdao/spells-mainnet/, 4 occurrences, e.g.: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L66
    • Debt Ceiling Instant Access Module https://github.com/makerdao/dss-auto-line/, 3 occurrences, e.g.: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L180
    • Developer Guides https://github.com/makerdao/developerguides/, 1 occurrence: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L199
    • Tinlake Manager https://github.com/centrifuge/tinlake-maker-lib/, 1 occurrence: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L204
    • DSS vest https://github.com/makerdao/dss-vest/, 1 occurrence: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L248
    • DSS interfaces https://github.com/makerdao/dss-interfaces/, 4 occurrences, e.g.: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-crafter-mainnet-workflow.md?plain=1#L92
    • JoinFab https://github.com/makerdao/JoinFab/, 1 occurrence: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-crafter-mainnet-workflow.md?plain=1#L104
    • DSS deploy https://github.com/makerdao/dss-deploy/, 2 occurrences, e.g.: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-crafter-mainnet-workflow.md?plain=1#L109
    • DSS cron https://github.com/makerdao/dss-cron/, 1 occurrence: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-crafter-mainnet-workflow.md?plain=1#L252
  • Update Foundry (depends on github releases) https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L32
  • Fetch git submodules in spells-mainnet https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L33
  • Publish spell reviews at https://github.com/makerdao/spells-mainnet/

• https://ipfs.io

  • Get pre-computed rates, 2 occurrences: https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L71C30-L71C45 https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L103

• https://www.epochconverter.com

  • Convert timestamp https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L77C11-L78

• https://etherscan.io

  • Check Rely events https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L92
  • Check source code of a new contract https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L93
  • Check deployed spell (manually) https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L347 https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L368
  • Checks deployed spell (using API) https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L355-L360
  • Get priority fee https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-crafter-mainnet-workflow.md?plain=1#L182

• https://tenderly.co

  • Create testnet (using API) https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-crafter-mainnet-workflow.md?plain=1#L200-L203
  • Review testnet (manually) https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L370-L376

• https://discord.com

  • Handover deployed spell https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L393-L394

• https://signal.org

  • Spell-related communication

[SidestreamColdMelon]

Potential circumventions

Most used services are not lock-in (e.g.: git, ipfs, xlsx, messaging) to circumvent their offline/compromised state we can just pre-define their alternatives. Less portable are spreadsheet comments, github PR reviews and discord channels as they have additional permissions attached to them. The most centralized services that does not have easily interchangeable alternatives are etherscan.io and tenderly.co.

1. Pre-define alternative services / communication channels. Applicable to:

   • Exec Sheet hosted on https://docs.google.com/spreadsheets/
     • [ ] Define a process in which instructions (e.g. raw xlsx file) can be sent over a group message or hosted on a different server if google docs are down
   • All repositories hosted on http://github.com
     • [ ] Mirror existing repositories into another git providers, use CI to keep them in sync
   • Source code verified on https://etherscan.io
     • [ ] Use multiple different services to verify the source code
   • Rates hosted on https://ipfs.io
     • [ ] Define different trusted ipfs gateways

2. Where possible, use local tools instead of services. Explicitly mark service-based checks as additional/non-blocking. Applicable to:

   • CI tests on http://github.com
     • [ ] Define this step as additional (as we're already running tests locally) or use 2 other CI providers
   • References to other repos at http://github.com
     • [ ] Write a script to pre-fetch all referenced repositories locally
   • Timestamp converter on https://www.epochconverter.com
     • [ ] Replace with local converter command / script which is part of the repo

3. Replace non-portable linked resources (PR comments / issues / releases / wikis and other information stored outside git) with git or ipfs. Applicable to:

   • Mentioned wiki page https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L57
     • [ ] Find another source for the same information inside a different repo (as wikis are actually more easily editable)
   • Mentioned github issue https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-crafter-mainnet-workflow.md?plain=1#L92
     • [ ] Describe the problem inside the checklist, otherwise make it clear that the link is optional
   • Mentioned release https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L38
     • [ ] Ensure alternative check is described (local tag) or alternative remote is listed

4. Use on-chain registry of the team + attestations for the most security-crucial operations. Applicable to:

   • Spell review and deployment confirmation at https://github.com/makerdao/spells-mainnet/
     • [ ] Use attestation service to store hash + address of a new spell
   • Spell handover on https://discord.com
     • [ ] Use 3 attestations by pre-specified users to fire a handover event

[SidestreamColdMelon]

First specific issue for etherscan is created: https://github.com/makerdao/pe-checklists/issues/31

[0xp3th1um]

I am making a few comments below in order to move forward with this. I think each topic should be taken into consideration separately in a different issue/thread.

• https://docs.google.com/spreadsheets/ => we can use excel compatible formats that can be stored on cloud or ipfs and then opened locally. It is of low severity since different people can create a sheet and it is hard to have censorship.

• http://github.com => Alternative options: Gitlab, Bitbucket etc. We should keep local copies in case the Maker account gets suspended e.g. techops making a local copy once a day.

• https://ipfs.io/ => Alternative options Arweave,Swarm etc. Pretty decentralized with good availability.

• https://etherscan.io/ => no good alternative. Blockscout or sourceify not widely used and without the same features.

• https://discord.com/ = > https://signal.org/ or telegram? Create communication channels.

• https://signal.org/ => use also telegram? We can share emergency emails for all governance facilitators and spell casters/revierwers.

• https://tenderly.co/ => There is no other alternative replacement since the testenets are deprecated.

• Foundry = > Hard to replace. Low severity.