Create detailed documentation for executing critical flows

[bingyanglin]

Create detailed documentation of how to execute the following flows, to ease of future testing and investigation.

1. spin up a node and sync it from genesis
2. spin up a node and sync with rocksdb database snapshot
3. run a node without the change, stop it, apply the change, start it again (in the middle of an epoch)
4. run a node without the change, stop it, apply the change, start it again (after an epoch passed)

[bingyanglin]

• The docs for running local network
  • Confluence page
  • Slack discussion
• The full node docs
  • Note that the user/password for https://docs.iota.org is shared in the slack channel here.
• The docker local private network docs

[bingyanglin]

For the docker local private network, because we don't have images in the Docker Hub yet (or use the private image registry which can be accessed by DevOps team only now), the docker image should be built by our own.

[bingyanglin]

Encountered the following errors when running the full node via docker

WARN[0000] /root/proj/iota/docker/fullnode/docker-compose.yaml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 1/1
 ✘ fullnode Error pull access denied for iotaledger/iota-node, repository does not exist or may require 'dock...                 1.2s 
Error response from daemon: pull access denied for iotaledger/iota-node, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Checked with the DevOps team @junwei0117 => We have private Docker Registry but it is only open for DevOps team for now.

[bingyanglin]

In the full node docs, the --config-path in compiling code command is necessary, like

cargo run --release --bin iota-node -- --config-path fullnode.yaml 

However, no migration.blob is provided; the migration-tx-data-path: is essential to provide, otherwise encounters

Failed to start node: no file location set

Thanks @junwei0117 for providing the migration.bob (link here), after downloading it and set its path, cargo run --release --bin iota-node -- --config-path fullnode.yaml is now running.

[junwei0117]

By the way, if you’re going to test RocksDB database snapshot, I suggest also testing the formal snapshot. It is much faster, but it requires an archive node, which we already have in alphanet.

Additionally, here are some commands I’ve tested, but you may need to use your own Docker image or binary instead. You can use these as a reference for database restoration

• RocksDB:

  docker run --rm \
      -v ./:/iota/s3_backup \
      -w /iota \
      -e AWS_SNAPSHOT_ENDPOINT=https://iota-rebased-alphanet-snapshots.s3.eu-west-1.amazonaws.com \
      docker-registry.iota.org/iota-tools:v0.5.0-alpha \
      /usr/local/bin/iota-tool download-db-snapshot \
      --latest \
      --path /iota/s3_backup \
      --num-parallel-downloads 50 \
      --no-sign-request \
      --skip-indexes
      --verbose

• Formal:

  docker run --rm \
      -v ./:/iota/s3_backup \
      -v /opt/iota/genesis:/iota/genesis \
      -w /iota \
      -e RUST_LOG=debug=true \
      -e MAINNET_FORMAL_UNSIGNED_BUCKET=iota-rebased-alphanet-formal \
      -e AWS_SNAPSHOT_ENDPOINT=https://iota-rebased-alphanet-formal.s3.eu-west-1.amazonaws.com \
      -e FORMAL_SNAPSHOT_ARCHIVE_BUCKET=iota-rebased-alphanet-archive \
      -e AWS_ARCHIVE_ENDPOINT=https://iota-rebased-alphanet-archive.s3.eu-west-1.amazonaws.com \
      -e AWS_ARCHIVE_VIRTUAL_HOSTED_REQUESTS=true \
      docker-registry.iota.org/iota-tools:v0.5.0-alpha \
      /usr/local/bin/iota-tool download-formal-snapshot \
      --latest \
      --genesis /iota/genesis/genesis.blob \
      --path /iota/s3_backup \
      --num-parallel-downloads 50 \
      --no-sign-request \
      --verify none \
      --verbose

[bingyanglin]

Important:

• The live folder is the epoch db files that you will sync from it. If there is no db files then you will sync from genesis.
• Try to reduce the -num-parallel-downloads if your connections dropped

For the download the snapshots w/o using docker image.

• RockdDB

  export AWS_SNAPSHOT_ENDPOINT=https://iota-rebased-alphanet-snapshots.s3.eu-west-1.amazonaws.com

 ./target/release/iota-tool download-db-snapshot \
    --latest \
    --path ./s3_backup \
    --num-parallel-downloads 50 \
    --no-sign-request \
    --skip-indexes \
    --verbose

Then, in the s3_backup folder will have a epoch_<N> folder. Then rename the epoch_<N> to be live folder. Remove the whole live folder in the db-path (set in the fullnode.yaml file). Replace the live folder with the renamed epoch_<N> folder.

In the end, run cargo run --release --bin iota-node -- --config-path fullnode.yaml

• FormalDB

  export MAINNET_FORMAL_UNSIGNED_BUCKET=iota-rebased-alphanet-formal
  export AWS_SNAPSHOT_ENDPOINT=https://iota-rebased-alphanet-formal.s3.eu-west-1.amazonaws.com
  export FORMAL_SNAPSHOT_ARCHIVE_BUCKET=iota-rebased-alphanet-archive
  export AWS_ARCHIVE_ENDPOINT=https://iota-rebased-alphanet-archive.s3.eu-west-1.amazonaws.com
  export AWS_ARCHIVE_VIRTUAL_HOSTED_REQUESTS=true

Note that you might need to change the genesis.blob folder

 ./target/release/iota-tool download-formal-snapshot \
    --latest \
    --genesis ./genesis.blob \
    --path ./s3_backup \
    --num-parallel-downloads 50 \
    --no-sign-request \
    --verify none \
    --verbose

Then, in the s3_backup folder will have a live folder in the db-path (set in the fullnode.yaml file). Replace the live folder directly.

In the end, run cargo run --release --bin iota-node -- --config-path fullnode.yaml

[bingyanglin]

The node operation guides in the Confluence