search

hyperledger-iroha / iroha

Iroha - A simple, enterprise-grade decentralized ledger
https://wiki.hyperledger.org/display/iroha
Apache License 2.0
438 stars 279 forks source link

[bug] Fix iroha2 deployment #2594

Closed BAStos525 closed 2 years ago

BAStos525 commented 2 years ago

Feature request

New iroha2:dev image is not able to be deployed on remote instances. When we trying to use the follow docker-compose.yml file through Ansible deployment that was fine on old CI:

version: "2.4"

services:
  iroha2:
    container_name: iroha2
    image: hyperledger/iroha2:dev
    restart: always
    mem_limit: 14g
    environment:
      IROHA_PUBLIC_KEY: 'ed01201c61faf8fe94e253b93114240394f79a607b7fa55f9e5a41ebec74b88055768b'
      IROHA_PRIVATE_KEY: '{"digest_function": "ed25519", "payload": "282ed9f3cf92811c3818dbc4ae594ed59dc1a2f78e4241e31924e101d6b1fb831c61faf8fe94e253b93114240394f79a607b7fa55f9e5a41ebec74b88055768b"}'
      TERMINAL_COLORS: 'false'
      DISABLE_PANIC_TERMINAL_COLORS: 'true'
    command: iroha --submit genesis
    volumes:
      - /opt/docker_data/chain:/chain
      - /opt/iroha2-deploy/configs/peer:/config

    ports:
      - "1337:1337"
      - "8080:8080"
      - "8180:8180"

    labels:
      - 'filebeat.fields.app=iroha2'
      - 'filebeat.fields.env=test'
      - 'filebeat.fields.project=iroha2'

    networks:
      - iroha2-net

    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "3"

networks:
  iroha2-net:
    external:
      name: iroha2-net

Every time we get the error:

Error: Failed to retrieve required environment variable: IROHA_TORII

Caused by:
    environment variable not found

Location:
    cli/src/main.rs:50:37

Attempt to move a variables from config.json to docker env vars was unsuccessful and we got the same error:

version: "3.8"
services:
  iroha2:
    container_name: iroha2
    image: hyperledger/iroha2:dev
    restart: always

    environment:
      KURA_BLOCK_STORE_PATH: /storage
      TORII_P2P_ADDR: 0.0.0.0:1337
      TORII_API_URL: 0.0.0.0:8080
      TORII_TELEMETRY_URL: 0.0.0.0:8180
      IROHA_PUBLIC_KEY: 'ed01201c61faf8fe94e253b93114240394f79a607b7fa55f9e5a41ebec74b88055768b'
      IROHA_PRIVATE_KEY: '{"digest_function": "ed25519", "payload": "282ed9f3cf92811c3818dbc4ae594ed59dc1a2f78e4241e31924e101d6b1fb831c61faf8fe94e253b93114240394f79a607b7fa55f9e5a41ebec74b88055768b"}'
      SUMERAGI_TRUSTED_PEERS: '[{"address":"s1.stg1.iroha2.iroha.tech:1337", "public_key": "ed01201c61faf8fe94e253b93114240394f79a607b7fa55f9e5a41ebec74b88055768b"}, {"address":"s2.stg1.iroha2.iroha.tech:1337", "public_key": "ed0120cc25624d62896d3a0bfd8940f928dc2abf27cc57cefeb442aa96d9081aae58a1"}, {"address": "s3.stg1.iroha2.iroha.tech:1337", "public_key": "ed0120faca9e8aa83225cb4d16d67f27dd4f93fc30ffa11adc1f5c88fd5495ecc91020"}, {"address": "s4.stg1.iroha2.iroha.tech:1337", "public_key": "ed01208e351a70b6a603ed285d666b8d689b680865913ba03ce29fb7d13a166c4e7f1f"}]'

...

Motivation

We should find the reason in deployment of new iroha2-dev image. By the way, iroha2:dev is able to be successfully deployed locally using this docker-compose file

Who can help?

@appetrosyan @s8sato

s8sato commented 2 years ago

According to the code, the path to the config.json seems to be wrong. One solution will be to overwrite the path with IROHA2_CONFIG_PATH environment variable

BAStos525 commented 2 years ago

What do you mean under the wrong path to config.json? /opt/iroha2-deploy/configs/peer contains this file. At least, it works fine with iroha2:stable image.

s8sato commented 2 years ago

Let me know the image IDs (preferably the original git commits) of the failing and succeeding containers @BAStos525

BAStos525 commented 2 years ago

The last one failed hyperledger/iroha2:dev: DIGEST:sha256:e28ffd9e89f90b3982d1e5c222be104c62002bff7b43c98e447017cec7e462e4

Present iroha2:stable images works.

BAStos525 commented 2 years ago

I tried again with this docker-compose.yaml where IROHA_TORII is declared as env var:

version: "2.4"

services:
  iroha2:
    container_name: iroha2
    image: hyperledger/iroha2:dev
    restart: always
    mem_limit: 14g
    environment:
      IROHA_PUBLIC_KEY: 'ed01201c61faf8fe94e253b93114240394f79a607b7fa55f9e5a41ebec74b88055768b'
      IROHA_PRIVATE_KEY: '{"digest_function": "ed25519", "payload": "282ed9f3cf92811c3818dbc4ae594ed59dc1a2f78e4241e31924e101d6b1fb831c61faf8fe94e253b93114240394f79a607b7fa55f9e5a41ebec74b88055768b"}'
      TORII_P2P_ADDR: 0.0.0.0:1337
      TORII_API_URL: 0.0.0.0:8080
      TORII_TELEMETRY_URL: 0.0.0.0:8180
      TERMINAL_COLORS: 'false'
      DISABLE_PANIC_TERMINAL_COLORS: 'true'
    command: iroha --submit-genesis
    init: true

    volumes:
      - /opt/docker_data/storage:/storage
      - /opt/iroha2-deploy/configs/peer:/config

    ports:
      - "1337:1337"
      - "8080:8080"
      - "8180:8180"

    labels:
      - 'filebeat.fields.app=iroha2'
      - 'filebeat.fields.env=stage1'
      - 'filebeat.fields.project=iroha2-longevity'

    networks:
      - iroha2-net

    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "3"

networks:
  iroha2-net:
    external:
      name: iroha2-net

Also config.json is mounted as volume:

            "Binds": [
                "/opt/docker_data/storage:/storage:rw",
                "/opt/iroha2-deploy/configs/peer:/config:rw"

The same error:

Location:
    cli/src/main.rs:50:37
Error: Failed to retrieve required environment variable: IROHA_TORII

Caused by:
    environment variable not found

Image:

hyperledger/iroha2:dev
DIGEST:sha256:ac90e8279b54478ac2ccd47f3a2385981ec2f1d2521a53b34ec9c13d9cee96b6

But there is no problem to deploy iroha2:dev locally using this docker-compose file Also with the presented docker-compose.yaml we can deploy iroha2:stable without any problems. What could be go wrong?

BAStos525 commented 2 years ago

Also I noticed that config.json is required in any case: if volume is not active, IROHA_TORII via env var doesn't work for local deployment as well.

s8sato commented 2 years ago

Have you checked that the remote instance and your local machine have the same hashes of hyperledger/iroha2:dev ?

BTW this can be a issue specific to the current iroha2:dev image, which is not up-to-date due to the failure to publish iroha2-base:dev image, which is exactly what I'm fixing. So could you put this on hold for a while @BAStos525 ?

s8sato commented 2 years ago

If the necessity for config.json is the case with the latest image after the fix, we have to reopen #1990

BAStos525 commented 2 years ago

If the necessity for config.json is the case with the latest image after the fix, we have to reopen #1990

As far as I know, config.json is required by all iroha peers from this May, isn't it?

BAStos525 commented 2 years ago

Have you checked that the remote instance and your local machine have the same hashes of hyperledger/iroha2:dev ?

BTW this can be a issue specific to the current iroha2:dev image, which is not up-to-date due to the failure to publish iroha2-base:dev image, which is exactly what I'm fixing. So could you put this on hold for a while @BAStos525 ?

Yes, I checked that these iroha2:dev images are the same both for local and remote deployment. By the way, iroha2-bas:dev image has been published.

s8sato commented 2 years ago

Ok, then we can proceed to the question of what is remote-specific. The error seems to be the same as when config.json is absent

s8sato commented 2 years ago

This worked to me under absence of config.json @BAStos525

image

BAStos525 commented 2 years ago

Ok, I rolled back to docker-compose template that was before this May. Note that I use iroha2:dev image from my docker hub account that was build based on ea47790e7a001ba7f55e1dbd07e2dd55de836dcb. docker-compose for the first genesis peer:

version: "2.4"

services:
  iroha2:
    container_name: iroha2
    image: bastos525/iroha2:dev
    restart: always
    mem_limit: 14g
    environment:
      TORII_P2P_ADDR: '0.0.0.0:1337'
      TORII_API_URL: '0.0.0.0:8080'
      TORII_TELEMETRY_URL: '0.0.0.0:8180'
      IROHA_PUBLIC_KEY: 'ed01201c61faf8fe94e253b93114240394f79a607b7fa55f9e5a41ebec74b88055768b'
      IROHA_PRIVATE_KEY: '{"digest_function": "ed25519", "payload": "282ed9f3cf92811c3818dbc4ae594ed59dc1a2f78e4241e31924e101d6b1fb831c61faf8fe94e253b93114240394f79a607b7fa55f9e5a41ebec74b88055768b"}'
      SUMERAGI_TRUSTED_PEERS: '[{"address": "s1.stg1.iroha2.iroha.tech:1337", "public_key": "ed01201c61faf8fe94e253b93114240394f79a607b7fa55f9e5a41ebec74b88055768b"}, {"address": "s2.stg1.iroha2.iroha.tech:1337", "public_key": "ed0120cc25624d62896d3a0bfd8940f928dc2abf27cc57cefeb442aa96d9081aae58a1"}, {"address": "s3.stg1.iroha2.iroha.tech:1337", "public_key": "ed0120faca9e8aa83225cb4d16d67f27dd4f93fc30ffa11adc1f5c88fd5495ecc91020"}, {"address": "s4.stg1.iroha2.iroha.tech:1337", "public_key": "ed01208e351a70b6a603ed285d666b8d689b680865913ba03ce29fb7d13a166c4e7f1f"}]'
      KURA_BLOCK_STORE_PATH: '/storage'
      SUMERAGI_COMMIT_TIME_MS: '10000'
      TERMINAL_COLORS: 'false'
      DISABLE_PANIC_TERMINAL_COLORS: 'true'

    command:
      - "iroha --submit-genesis" 
    volumes:
      - /opt/docker_data/storage:/storage
      - /opt/iroha2-deploy/configs/peer:/config

    ports:
      - "1337:1337"
      - "8080:8080"
      - "8180:8180"

    labels:
      - 'filebeat.fields.app=iroha2'
      - 'filebeat.fields.env=stage1'
      - 'filebeat.fields.project=iroha2-longevity'

    networks:
      - iroha2-net

    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "3"

networks:
  iroha2-net:
    external:
      name: iroha2-net

Error:

Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "iroha --submit-genesis": executable file not found in $PATH: unknown

Typical other peers:

version: "2.4"

services:
  iroha2:
    container_name: iroha2
    image: bastos525/iroha2:dev
    restart: always
    mem_limit: 14g
    environment:
      TORII_P2P_ADDR: '0.0.0.0:1337'
      TORII_API_URL: '0.0.0.0:8080'
      TORII_TELEMETRY_URL: '0.0.0.0:8180'
      IROHA_PUBLIC_KEY: 'ed0120faca9e8aa83225cb4d16d67f27dd4f93fc30ffa11adc1f5c88fd5495ecc91020'
      IROHA_PRIVATE_KEY: '{"digest_function": "ed25519", "payload": "1261a436d36779223d7d6cf20e8b644510e488e6a50bafd77a7485264d27197dfaca9e8aa83225cb4d16d67f27dd4f93fc30ffa11adc1f5c88fd5495ecc91020"}'
      SUMERAGI_TRUSTED_PEERS: '[{"address": "s1.stg1.iroha2.iroha.tech:1337", "public_key": "ed01201c61faf8fe94e253b93114240394f79a607b7fa55f9e5a41ebec74b88055768b"}, {"address": "s2.stg1.iroha2.iroha.tech:1337", "public_key": "ed0120cc25624d62896d3a0bfd8940f928dc2abf27cc57cefeb442aa96d9081aae58a1"}, {"address": "s3.stg1.iroha2.iroha.tech:1337", "public_key": "ed0120faca9e8aa83225cb4d16d67f27dd4f93fc30ffa11adc1f5c88fd5495ecc91020"}, {"address": "s4.stg1.iroha2.iroha.tech:1337", "public_key": "ed01208e351a70b6a603ed285d666b8d689b680865913ba03ce29fb7d13a166c4e7f1f"}]'
      KURA_BLOCK_STORE_PATH: '/storage'
      SUMERAGI_COMMIT_TIME_MS: '10000'
      TERMINAL_COLORS: 'false'
      DISABLE_PANIC_TERMINAL_COLORS: 'true'

    volumes:
      - /opt/docker_data/storage:/storage
      - /opt/iroha2-deploy/configs/peer:/config

    ports:
      - "1337:1337"
      - "8080:8080"
      - "8180:8180"

    labels:
      - 'filebeat.fields.app=iroha2'
      - 'filebeat.fields.env=stage1'
      - 'filebeat.fields.project=iroha2-longevity'

    networks:
      - iroha2-net

    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "3"

networks:
  iroha2-net:
    external:
      name: iroha2-net

Error:

Error: Failed to retrieve required environment variable: IROHA_TORII

Caused by:
    environment variable not found

Location:
    cli/src/main.rs:50:37

And as expected, no any issues with hyperledger/iroha2:stable image.

s8sato commented 2 years ago

Interestingly, the genesis and other peers are failing with different errors. Since this is a consistency problem between the internal repository and the external (e.g. the compose file), could you share a link to the external state, preferably a specific commit?

BAStos525 commented 2 years ago

By the way, is it really works to declare TORII, SUMERAGI... values via env vars as json within absence of config.json? If I am not mistaken, it was idea to move iroha configuration from env vars to config.json, but does the old way still work? I use this Jinja template for docker compose and config.json.

s8sato commented 2 years ago

Yes, you can take the latter way to configure, as long as config.json is actually accessible. The possibility is that the path is correct while the permission is wrong

If you cannot access the metadata of the file, e.g. because of a permission error or broken symbolic links, this will return false. https://doc.rust-lang.org/std/path/struct.PathBuf.html#method.exists

BAStos525 commented 2 years ago

I don't think that there are problems with config.json path or with volume permissions, otherwise iroha stable image should has the same issues.

BAStos525 commented 2 years ago

iroha2:stable container:

root@247766eb5feb:/# ls -la /config/
total 12
drwx------ 2 root root 4096 Aug 26 08:14 .
drwxr-xr-x 1 root root 4096 Aug 26 08:51 ..
-rw------- 1 root root 1629 Aug 26 08:14 config.json

iroha2:dev container:

/ $ ls -la /config/
total 20
drwxrwxr-x    2 iroha    iroha         4096 Aug 25 08:50 .
drwxr-xr-x    1 root     root          4096 Aug 31 08:38 ..
-rw-rw-r--    1 iroha    iroha         1518 Aug 11 12:59 config.json
-rw-rw-r--    1 iroha    iroha         6866 Aug 25 08:50 genesis.json

I am not sure, but it could be a reason as well.

BAStos525 commented 2 years ago

The last suggestion is correct and the reason of this issue. We can close the ticket.