search

hyperledger / caliper

A blockchain benchmark framework to measure performance of multiple blockchain solutions https://wiki.hyperledger.org/display/caliper
https://hyperledger.github.io/caliper/
Apache License 2.0
642 stars 403 forks source link

launch master hung #996

Closed litong01 closed 3 years ago

litong01 commented 3 years ago

after launch master using caliper container 0.3.2, caliper will hang

Context

Setup fabric network, network config and benchmark config file to launch caliper master to run the test, caliper launch master will hang

Expected Behavior

caliper launch master should return after txs all finish

Actual Behavior

the process hang after some txs finished successfully

Possible Fix

Steps to Reproduce

  1. Setup your fabric network
  2. Create connection config and benchmark config file accordingly
  3. Using hyperledger/caliper:0.3.2 to start a container then start caliper launch master command inside of the command.
  4. After awhile, the caliper command will block forever, going inside the container do ps -a command will produce the following 
    Mem: 3451452K used, 579208K free, 3732K shrd, 173092K buff, 1771436K cached
CPU:   5% usr   5% sys   0% nic  88% idle   0% io   0% irq   0% sirq
Load average: 0.34 0.87 0.93 9/508 999
PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
7     1 node     S     827m  21%   0   0% node /home/node/.npm-global/bin/caliper launch master --caliper-bind-sut fabric:2.1.0 --caliper-flow-only-t
993     0 node     S     1592   0%   0   0% /bin/sh
1     0 node     S     1584   0%   0   0% {caliperrun.sh} /bin/sh /hyperledger/caliper/workspace/run/caliperrun.sh
999   993 node     R     1528   0%   1   0% top

Existing issues

network config.json

{
  "caliper": {
    "blockchain": "fabric"
  },
  "clients": {
    "Admin@org0.example.com": {
      "client": {
        "credentialStore": {
          "path": "tmp/hfc-kvs/org1",
          "cryptoStore": {"path": "tmp/hfc-kvs/org1"}
        },
        "organization": "org0.example.com",
        "clientPrivateKey": {
          "path": "/hyperledger/caliper/workspace/keyfiles/peerOrganizations/org0.example.com/users/Admin@org0.example.com/msp/keystore/priv_sk"
        },
        "clientSignedCert": {
          "path": "/hyperledger/caliper/workspace/keyfiles/peerOrganizations/org0.example.com/users/Admin@org0.example.com/msp/signcerts/Admin@org0.example.com-cert.pem"
        },
        "connection": {
          "timeout": { "peer": { "endorser": "300" } }
        }
      }
    }
  },
  "channels": {
      "mychannel": {
          "created" : true,
          "chaincodes": [
              { "id": "samplecc", "version": "1.0" }
          ]
      }
  },
  "name": "test-d679af657c_net",
  "organizations":{
    "org0.example.com": {
      "mspid": "org0-example-com",
      "peers": [
        "peer1.org0.example.com",
        "peer2.org0.example.com"
      ],
      "certificateAuthorities": [
         "ca1.org0.example.com"
      ],
      "adminPrivateKey": {
        "path": "/hyperledger/caliper/workspace/keyfiles/peerOrganizations/org0.example.com/users/Admin@org0.example.com/msp/keystore/priv_sk"
      },
      "signedCert": {
        "path": "/hyperledger/caliper/workspace/keyfiles/peerOrganizations/org0.example.com/users/Admin@org0.example.com/msp/signcerts/Admin@org0.example.com-cert.pem"
      }
    }
  },
  "peers": {
    "peer1.org0.example.com": {
      "url": "grpcs://peer1.org0.example.com:7051",
      "grpcOptions": {
        "ssl-target-name-override": "peer1.org0.example.com",
        "hostnameOverride": "peer1.org0.example.com"
      },
      "tlsCACerts": {
        "pem": "-----BEGIN CERTIFICATE-----\nMIICgDCCAiegAwIBAgIUZxLmg7xzBLgwmNl8v+9bYtZJVQswCgYIKoZIzj0EAwIw\ndTELMAkGA1UEBhMCVVMxFzAVBgNVBAgMDk5vcnRoIENhcm9saW5hMRAwDgYDVQQH\nDAdSYWxlaWdoMRkwFwYDVQQKDBBvcmcwLmV4YW1wbGUuY29tMSAwHgYDVQQDDBd0\nbHNjYTEub3JnMC5leGFtcGxlLmNvbTAeFw0yMDA5MTQyMTU0MTlaFw0zMDA5MTIy\nMTU0MTlaMHUxCzAJBgNVBAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBDYXJvbGluYTEQ\nMA4GA1UEBwwHUmFsZWlnaDEZMBcGA1UECgwQb3JnMC5leGFtcGxlLmNvbTEgMB4G\nA1UEAwwXdGxzY2ExLm9yZzAuZXhhbXBsZS5jb20wWTATBgcqhkjOPQIBBggqhkjO\nPQMBBwNCAATVrEfNpXjYBP4HigR3rkMSfnRvont9mZwuJ4y7y92aCBQtimyKw7Se\nl2ddpfIB/BAYI1FUEkKALp6RckczO0a6o4GUMIGRMB0GA1UdDgQWBBRdHuEzDnwM\nVDKaJO/+fft0Wa979TAfBgNVHSMEGDAWgBRdHuEzDnwMVDKaJO/+fft0Wa979TAP\nBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBpjAdBgNVHSUEFjAUBggrBgEF\nBQcDAQYIKwYBBQUHAwIwDwYDVR0RBAgwBocEwKg4IDAKBggqhkjOPQQDAgNHADBE\nAiA6OPm6MoBk/b2mg5oF4o2dGtuEGNFyVyK+L+pJOplmswIgWn4Q6HT4ieQCjeTL\nrP7OpmPKtCkOszdtai+JrnSn9mw=\n-----END CERTIFICATE-----"
      }
    },
    "peer2.org0.example.com": {
      "url": "grpcs://peer2.org0.example.com:7051",
      "grpcOptions": {
        "ssl-target-name-override": "peer2.org0.example.com",
        "hostnameOverride": "peer2.org0.example.com"
      },
      "tlsCACerts": {
        "pem": "-----BEGIN CERTIFICATE-----\nMIICgDCCAiegAwIBAgIUZxLmg7xzBLgwmNl8v+9bYtZJVQswCgYIKoZIzj0EAwIw\ndTELMAkGA1UEBhMCVVMxFzAVBgNVBAgMDk5vcnRoIENhcm9saW5hMRAwDgYDVQQH\nDAdSYWxlaWdoMRkwFwYDVQQKDBBvcmcwLmV4YW1wbGUuY29tMSAwHgYDVQQDDBd0\nbHNjYTEub3JnMC5leGFtcGxlLmNvbTAeFw0yMDA5MTQyMTU0MTlaFw0zMDA5MTIy\nMTU0MTlaMHUxCzAJBgNVBAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBDYXJvbGluYTEQ\nMA4GA1UEBwwHUmFsZWlnaDEZMBcGA1UECgwQb3JnMC5leGFtcGxlLmNvbTEgMB4G\nA1UEAwwXdGxzY2ExLm9yZzAuZXhhbXBsZS5jb20wWTATBgcqhkjOPQIBBggqhkjO\nPQMBBwNCAATVrEfNpXjYBP4HigR3rkMSfnRvont9mZwuJ4y7y92aCBQtimyKw7Se\nl2ddpfIB/BAYI1FUEkKALp6RckczO0a6o4GUMIGRMB0GA1UdDgQWBBRdHuEzDnwM\nVDKaJO/+fft0Wa979TAfBgNVHSMEGDAWgBRdHuEzDnwMVDKaJO/+fft0Wa979TAP\nBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBpjAdBgNVHSUEFjAUBggrBgEF\nBQcDAQYIKwYBBQUHAwIwDwYDVR0RBAgwBocEwKg4IDAKBggqhkjOPQQDAgNHADBE\nAiA6OPm6MoBk/b2mg5oF4o2dGtuEGNFyVyK+L+pJOplmswIgWn4Q6HT4ieQCjeTL\nrP7OpmPKtCkOszdtai+JrnSn9mw=\n-----END CERTIFICATE-----"
      }
    }
  },
  "certificateAuthorities": {
    "ca1.org0.example.com": {
      "url": "https://ca1.org0.example.com:7054",
      "tlsCACerts": {
        "pem": "-----BEGIN CERTIFICATE-----\nMIICfDCCAiGgAwIBAgIUaXBz4L8/jjW3nbjIXZxrBQ8KIEswCgYIKoZIzj0EAwIw\ncjELMAkGA1UEBhMCVVMxFzAVBgNVBAgMDk5vcnRoIENhcm9saW5hMRAwDgYDVQQH\nDAdSYWxlaWdoMRkwFwYDVQQKDBBvcmcwLmV4YW1wbGUuY29tMR0wGwYDVQQDDBRj\nYTEub3JnMC5leGFtcGxlLmNvbTAeFw0yMDA5MTQyMTU0MTlaFw0zMDA5MTIyMTU0\nMTlaMHIxCzAJBgNVBAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBDYXJvbGluYTEQMA4G\nA1UEBwwHUmFsZWlnaDEZMBcGA1UECgwQb3JnMC5leGFtcGxlLmNvbTEdMBsGA1UE\nAwwUY2ExLm9yZzAuZXhhbXBsZS5jb20wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC\nAATViAnV+uXT6lHY3bbFifKm7avg4xCv4IWWtM+ch3/0632za0ZNlh4pZbAbzp0b\nSBuXtnSktSTMNG8Ph6Uz/Dg7o4GUMIGRMB0GA1UdDgQWBBSvnbvh7RMyLcLYhUtq\nXx0qAclxuTAfBgNVHSMEGDAWgBSvnbvh7RMyLcLYhUtqXx0qAclxuTAPBgNVHRMB\nAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBpjAdBgNVHSUEFjAUBggrBgEFBQcDAQYI\nKwYBBQUHAwIwDwYDVR0RBAgwBocEwKg4IDAKBggqhkjOPQQDAgNJADBGAiEAsxxc\nSHolDGnRjwtPelDjg1IRP5yzl+jOm2w8Sq4AsVUCIQDtVjXaY8Sb2mEnxK+vFi1G\niHvldv9/2WhdfTqnzx4T+g==\n-----END CERTIFICATE-----"
      },
      "httpOptions": { "verify": "false" },
      "caName": "ca1"
    }
  },
  "version": "1.0"
}

benchmark config yaml file

test:
    name: samplecc-benchmark
    description: test benchmark
    workers:
      type: local
      number: 2
    rounds:
      - label: samplecc test
        description: samplecc benchmark
        chaincodeId: samplecc
        txDuration: 60
        rateControl:
          type: fixed-backlog
          opts:
            unfinished_per_client: 2
        callback: /hyperledger/caliper/workspace/app/app.js
        arguments:
          assets: 10

monitor:
  type:
  - none

observer:
  type: local
  interval: 5

This is using fabric 2.1.0 sdk.

Context

This basically blocks any process invokes the test.

Your Environment

nklincoln commented 3 years ago

There has been a significant reworking of the caliper internals within the 0.4.0 release, so it is possible that this is now fixed in those changes

davidkel commented 3 years ago

This was due to trying to invoke the caliper cli as follows

docker run --network mynetwork --name calipertester --hostname calipertester --rm
    -e "ROOTPATH=/vars/keyfiles"
    -v /var/run/docker.sock:/var/run/docker.sock
    -v $(pwd)/vars:/vars
    hyperledger/caliper:0.3.2
    "launch master --caliper-benchconfig /vars/run/caliperbenchmarkconfig.yaml
    --caliper-networkconfig /vars/run/calipernetworkconfig.json
    --caliper-flow-only-test --caliper-fabric-gateway-usegateway --caliper-fabric-gateway-discovery"

The solution is to remove the quotes around the launch master command

litong01 commented 3 years ago

@davidkel that is not the cause of the issue. The command has been changed to this, but same issue.

docker run --network {{ NETNAME }} --name calipertester --hostname calipertester --rm
    --entrypoint /hyperledger/caliper/workspace/run/caliperrun.sh
    -e "ROOTPATH=/hyperledger/caliper/workspace/keyfiles"
    -e "CALIPER-BENCHCONFIG=run/caliperbenchmarkconfig.yaml"
    -e "CALIPER-NETWORKCONFIG=run/calipernetworkconfig.json"
    -v /var/run/docker.sock:/var/run/docker.sock
    -v {{ hostroot }}/vars:/hyperledger/caliper/workspace
    hyperledger/caliper:0.3.2

You can see that it no longer uses the that.

The caliperrun.sh looks like this.

#!/bin/sh
rm -rf /hyperledger/caliper/workspace/caliper.log

caliper launch master \
  --caliper-flow-only-test --caliper-fabric-gateway-usegateway --caliper-fabric-gateway-discovery \
  --caliper-fabric-gateway-gatewaylocalhost false
litong01 commented 3 years ago

@nklincoln @davidkel since 0.4.0 is out, let me give it a try anyway and see what happens.