hyperledger / fabric-private-chaincode

FPC enables Confidential Chaincode Execution for Hyperledger Fabric using Intel SGX.
Apache License 2.0
159 stars 91 forks source link

initEnclave failed in the helloword tutorial #659

Closed azizkayumov closed 2 years ago

azizkayumov commented 2 years ago

Describe the bug I am trying to reproduce the results in the hello world example on Ubuntu 20.04 VM but it seems to be failing at initEnclave.

To Reproduce The host PC is Ubuntu 20.04 with VirtualBox installed.

  1. Start Ubuntu 20.04 VM with minimal installation option enabled.
  2. Install git
  3. Install go and set GOPATH
    
    sudo apt install golang
    go get github.com/golang/example/hello
    ~/go/bin/hello

go env export GOPATH="/home/ec/go" // 'ec' is just a VM username

5. Install Docker Engine:

sudo apt-get remove docker docker-engine docker.io containerd runc sudo apt-get update sudo apt-get install \ ca-certificates \ curl \ gnupg \ lsb-release

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io

Enable running Docker as  non-root user:

sudo groupadd docker sudo usermod -aG docker $USER newgrp docker

6. Install GCC:

sudo apt install build-essential gcc --version

7. Install Nanopb:

sudo apt-get install protobuf-compiler python-protobuf libprotobuf-dev export NANOPB_PATH=$HOME/nanopb git clone https://github.com/nanopb/nanopb.git $NANOPB_PATH cd $NANOPB_PATH git checkout nanopb-0.4.3 cd generator/proto && make

8. Clone FPC:

export FPC_PATH=$GOPATH/src/github.com/hyperledger/fabric-private-chaincode sudo git clone --recursive https://github.com/hyperledger/fabric-private-chaincode.git $FPC_PATH

9. Pull FPC docker images:

cd $FPC_PATH/utils/docker make pull-dev make run-dev

10. Build FPC:

cd $FPC_PATH make

11. Start helloworld [tutorial](https://github.com/hyperledger/fabric-private-chaincode/blob/main/samples/chaincode/helloworld/README.md):

cd $FPC_PATH/samples/chaincode/helloworld touch helloworld_cc.cpp

12. Install vim:

apt-get update apt-get install vim

13. Edit `helloworld_cc.cpp`:

vim helloworld_cc.cpp

And copy-paste the [example cc code](https://github.com/hyperledger/fabric-private-chaincode/blob/a0fe00b27df083a55ecaf23c84abe48d53a8a8b4/samples/chaincode/helloworld/README.md?plain=1#L158).
14. Edit Cmake build config according to [this](https://github.com/hyperledger/fabric-private-chaincode/blob/a0fe00b27df083a55ecaf23c84abe48d53a8a8b4/samples/chaincode/helloworld/README.md?plain=1#L248):

touch CMakeLists.txt vim CMakeLists.txt

15. Follow the "Time to Test" section as described [here](https://github.com/hyperledger/fabric-private-chaincode/blob/a0fe00b27df083a55ecaf23c84abe48d53a8a8b4/samples/chaincode/helloworld/README.md?plain=1#L305).
16. Run `test.h`:

cd $FPC_PATH/samples/chaincode/helloworld bash ./test.sh


Here is the complete log:

root@ec:/project/src/github.com/hyperledger/fabric-private-chaincode/s amples/chaincode/helloworld# bash ./test.sh test.sh: Setup ledger ... /project/src/github.com/hyperledger/fabric-private-chaincode/integration/config /project/src/github.com/hyperledger/fabric-private-chaincode/samples/chaincode/helloworld sampleorg.example.com 2022-03-02 06:17:52.870 UTC [common.tools.configtxgen] main -> INFO 001 Loading configuration 2022-03-02 06:17:52.898 UTC [common.tools.configtxgen.localconfig] completeInitialization -> INFO 002 orderer type: etcdraft 2022-03-02 06:17:52.900 UTC [common.tools.configtxgen.localconfig] Load -> INFO 003 Loaded configuration: /project/src/github.com/hyperledger/fabric-private-chaincode/integration/config/configtx.yaml 2022-03-02 06:17:52.902 UTC [common.tools.configtxgen] doOutputBlock -> INFO 004 Generating genesis block 2022-03-02 06:17:52.902 UTC [common.tools.configtxgen] doOutputBlock -> INFO 005 Creating system channel genesis block 2022-03-02 06:17:52.902 UTC [common.tools.configtxgen] doOutputBlock -> INFO 006 Writing genesis block 2022-03-02 06:17:55.950 UTC [common.tools.configtxgen] main -> INFO 001 Loading configuration 2022-03-02 06:17:55.978 UTC [common.tools.configtxgen.localconfig] Load -> INFO 002 Loaded configuration: /project/src/github.com/hyperledger/fabric-private-chaincode/integration/config/configtx.yaml 2022-03-02 06:17:55.979 UTC [common.tools.configtxgen] doOutputChannelCreateTx -> INFO 003 Generating new channel configtx 2022-03-02 06:17:55.983 UTC [common.tools.configtxgen] doOutputChannelCreateTx -> INFO 004 Writing new channel tx 2022-03-02 06:17:56.041 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized 2022-03-02 06:17:56.060 UTC [cli.common] readBlock -> INFO 002 Expect block, but got status: &{NOT_FOUND} 2022-03-02 06:17:56.061 UTC [channelCmd] InitCmdFactory -> INFO 003 Endorser and orderer connections initialized 2022-03-02 06:17:56.263 UTC [cli.common] readBlock -> INFO 004 Expect block, but got status: &{SERVICE_UNAVAILABLE} 2022-03-02 06:17:56.264 UTC [channelCmd] InitCmdFactory -> INFO 005 Endorser and orderer connections initialized 2022-03-02 06:17:56.466 UTC [cli.common] readBlock -> INFO 006 Expect block, but got status: &{SERVICE_UNAVAILABLE} 2022-03-02 06:17:56.468 UTC [channelCmd] InitCmdFactory -> INFO 007 Endorser and orderer connections initialized 2022-03-02 06:17:56.670 UTC [cli.common] readBlock -> INFO 008 Expect block, but got status: &{SERVICE_UNAVAILABLE} 2022-03-02 06:17:56.671 UTC [channelCmd] InitCmdFactory -> INFO 009 Endorser and orderer connections initialized 2022-03-02 06:17:56.873 UTC [cli.common] readBlock -> INFO 00a Expect block, but got status: &{SERVICE_UNAVAILABLE} 2022-03-02 06:17:56.875 UTC [channelCmd] InitCmdFactory -> INFO 00b Endorser and orderer connections initialized 2022-03-02 06:17:57.077 UTC [cli.common] readBlock -> INFO 00c Expect block, but got status: &{SERVICE_UNAVAILABLE} 2022-03-02 06:17:57.078 UTC [channelCmd] InitCmdFactory -> INFO 00d Endorser and orderer connections initialized 2022-03-02 06:17:57.281 UTC [cli.common] readBlock -> INFO 00e Received block: 0 peer.sh: Deriving channel id 'mychannel' from channel block file '/tmp/hyperledger/test/mychannel.block', relying on naming convention '..../.block' for that file! 2022-03-02 06:17:57.358 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized 2022-03-02 06:17:57.402 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel Installing ercc on channel 'mychannel' ... Packaging ercc ... { "path":"/project/src/github.com/hyperledger/fabric-private-chaincode/fabric/bin/../..//ercc", "type":"ercc-type", "label":"ercc_0" }

Installing ercc ... 2022-03-02 06:18:01.984 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 001 Installed remotely: response:<status:200 payload:"\nGercc_0:0891be591fb94db99892b838b941b12b265c0b9ad5592352d9feb19b64a4a220\022\006ercc_0" > 2022-03-02 06:18:01.984 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 002 Chaincode code package identifier: ercc_0:0891be591fb94db99892b838b941b12b265c0b9ad5592352d9feb19b64a4a220

Querying installed chaincodes to find package id..

Approve for my org 2022-03-02 06:18:07.102 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [b1780f0c5ee6f5dffe9db43e8faa70451bc3534f5825a4277eebf3b1885beb3d] committed with status (VALID) at 127.0.0.1:7051

Checking for commit readiness { "approvals": { "SampleOrg": true } }

Committing chaincode definition.... 2022-03-02 06:18:15.227 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [0359af831fbc4bb3de806f622587327cc0ebd865ec33f0259c51174b1a1f9d5c] committed with status (VALID) at 127.0.0.1:7051

Query commited chaincodes on the channel Committed chaincode definitions on channel 'mychannel': Name: ercc, Version: 0, Sequence: 1, Endorsement Plugin: escc, Validation Plugin: vscc

/project/src/github.com/hyperledger/fabric-private-chaincode/samples/chaincode/helloworld

test.sh: Run helloworld test ... test.sh: - install helloworld chaincode 2022-03-02 06:18:24.857 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 001 Installed remotely: response:<status:200 payload:"\nPhelloworld_test:35a29277dcc0543ec2f6ac5db8a12ea1264e00daa003e4c86fec2befb1f04d0a\022\017helloworld_test" > 2022-03-02 06:18:24.858 UTC [cli.lifecycle.chaincode] submitInstallProposal -> INFO 002 Chaincode code package identifier: helloworld_test:35a29277dcc0543ec2f6ac5db8a12ea1264e00daa003e4c86fec2befb1f04d0a metadata.json metadata.json 2022-03-02 06:18:27.140 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [8056f5a073f3ac247b0a7566d67234d391114a638b9f3eae2dc5f5ad7018084d] committed with status (VALID) at 127.0.0.1:7051 Chaincode definition for chaincode 'helloworld_test', version '8DC6ABEBD59FFCCAB415CDB6648DD4125153D4E34C04370973A0D34B972AA7AA', sequence '1' on channel 'mychannel' approval status by org: SampleOrg: true 2022-03-02 06:18:29.300 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [0d0608baea7f4f2cf9a52ef1cb800d117c1c8b603e144eade7335906b423325e] committed with status (VALID) at 127.0.0.1:7051 peer.sh: /project/src/github.com/hyperledger/fabric-private-chaincode/fabric/bin/../..//fabric/_internal/bin/peer chaincode query -o localhost:7050 --peerAddresses localhost:7051 -C mychannel -n helloworld_test -c {"Args":["__initEnclave", "Cg5sb2NhbGhvc3Q6NzA1MRIsZXlKaGRIUmxjM1JoZEdsdmJsOTBlWEJsSWpvaWMybHRkV3hoZEdWa0luMEs="]} Error: endorsement failure during query. response: status:500 message:"error in simulation: failed to execute transaction 8eae5768f0668c6ee0ac1b57eab9c39b7af60304ed908982d03913c9aed10c8a: error sending: chaincode stream terminated" peer.sh: initEnclave failed test.sh: - invoke storeAsset transaction to store value 100 in asset1 peer.sh: /project/src/github.com/hyperledger/fabric-private-chaincode/fabric/bin/../..//fabric/_internal/bin/peer chaincode query -o localhost:7050 -C mychannel -n ercc -c {"Args":["QueryChaincodeEncryptionKey", "helloworld_test"]} Error: endorsement failure during query. response: status:500 message:"no such key" peer.sh: looking up chaincode encryption key failed test.sh: - invoke retrieveAsset transaction to retrieve current value of asset1 peer.sh: /project/src/github.com/hyperledger/fabric-private-chaincode/fabric/bin/../..//fabric/_internal/bin/peer chaincode query -o localhost:7050 -C mychannel -n ercc -c {"Args":["QueryChaincodeEncryptionKey", "helloworld_test"]} Error: endorsement failure during query. response: status:500 message:"no such key" peer.sh: looking up chaincode encryption key failed test.sh: - invoke query with retrieveAsset transaction to retrieve current value of asset1 peer.sh: /project/src/github.com/hyperledger/fabric-private-chaincode/fabric/bin/../..//fabric/_internal/bin/peer chaincode query -o localhost:7050 -C mychannel -n ercc -c {"Args":["QueryChaincodeEncryptionKey", "helloworld_test"]} Error: endorsement failure during query. response: status:500 message:"no such key" peer.sh: looking up chaincode encryption key failed


**Expected behavior**
"Helloworld test PASSED" as decribed [here](https://github.com/hyperledger/fabric-private-chaincode/tree/main/samples/chaincode/helloworld)

**Log-filesScreenshots**
![image](https://user-images.githubusercontent.com/14273342/156312847-16dd5a40-22ef-4592-9493-a3ab0630e2a9.png)

**Environment (please complete the following information):**
 - OS (distro+version): Ubuntu 20.04 VM running on Ubuntu 20.04.3 PC without SGX (AMD® Ryzen 5 3600 6-core processor × 12)
 - FPC Version: [FPC latest development environment ](https://github.com/hyperledger/fabric-private-chaincode/blob/main/README.md#option-1-using-the-docker-based-fpc-development-environment)

**Additional context**
Here are the enviroment variables:

NANOPB_PATH=/usr/local/nanopb/ PKG_CONFIG_PATH=:/opt/intel/sgxsdk/pkgconfig HOSTNAME=ec OPENSSL_VERSION= SGX_MODE=SIM PWD=/project/src/github.com/hyperledger/fabric-private-chaincode/samples/chaincode/helloworld SGXSSL_VERSION= PROTOC_CMD=/usr/local/proto3/bin/protoc SGX_SSL=/opt/intel/sgxssl FPC_VERSION= HOME=/root FPC_PATH=/project/src/github.com/hyperledger/fabric-private-chaincode DOCKERD_FPC_PATH=/home/ec/go/src/github.com/hyperledger/fabric-private-chaincode/ PROTO_DIR=/usr/local/proto3 FABRIC_PATH=/project/src/github.com/hyperledger/fabric NANOPB_VERSION=0.4.3 GOROOT=/usr/local/go SGX_BUILD=DEBUG GO_VERSION=1.16.7 LESSCLOSE=/usr/bin/lesspipe %s %s TERM=xterm LESSOPEN=| /usr/bin/lesspipe %s SGX_SDK=/opt/intel/sgxsdk SHLVL=1 LD_LIBRARY_PATH=:/opt/intel/sgxsdk/sdklibs PATH=/project/bin:/usr/local/go/bin:/opt/intel/sgxsdk.extras/external/toolset/ubuntu20.04:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/intel/sgxsdk/bin:/opt/intel/sgxsdk/bin/x64 CI=true OLDPWD=/project/src/github.com/hyperledger/fabric-private-chaincode/samples/chaincode/helloworld GOPATH=/project =/usr/bin/env



[Using the FPC client SDK](https://github.com/hyperledger/fabric-private-chaincode/tree/main/samples/chaincode/helloworld#using-the-fpc-client-sdk) step also fails next.

I am using Ubuntu VM since it is easy to clean up after experiments. 
Let me know if I am missing something. 

Thank you.
mbrandenburger commented 2 years ago

Hi @azizkayumov

Thank you for this detailed bug report! I've tried to reproduce your issue but for me the hello world test runs through.

Could you please look at the chaincode and peer log and post them here?! You can find the logs in /tmp/fpc-extbuilder.$$SOME_DATE$$.helloworld_test:$$RNDID$$/chaincode.log and /tmp/hyperledger/test/peer.err.

A few more comments:

azizkayumov commented 2 years ago

Hello @mbrandenburger

Thank you for the quick response.

Sorry for the confusion, I tried everything with sudo at first, then changed all. I see that it is recommended to not to use sudo in the docs.

I followed the steps until Step 10:

cd $FPC_PATH
make

I noticed that 2 / 3 tests failed, here is the log:

.....
[ 94%] Built target get_attestation_app
make[5]: Entering directory '/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build'
make[5]: Leaving directory '/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build'
[100%] Built target verify_evidence_app
make[4]: Leaving directory '/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build'
make[3]: Leaving directory '/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build'
env CTEST_OUTPUT_ON_FAILURE=1 \
make -C _build test || \
    (echo "PDO crypto tests failed. Define PDO_DEBUG_BUILD=1 environment for enabling logging in tests." && exit 1)
make[3]: Entering directory '/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build'
Running tests...
Test project /project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build
    Start 1: utest
1/3 Test #1: utest ............................   Passed    0.50 sec
    Start 2: ttest
2/3 Test #2: ttest ............................***Exception: Illegal  0.23 sec

    Start 3: attested_evidence_test.sh
3/3 Test #3: attested_evidence_test.sh ........***Failed    0.16 sec
attested_evidence_test.sh: Testing simulated attestation
./attested_evidence_test.sh: line 27:  4880 Illegal instruction     (core dumped) ./get_attestation_app

33% tests passed, 2 tests failed out of 3

Total Test time (real) =   0.89 sec

The following tests FAILED:
      2 - ttest (ILLEGAL)
      3 - attested_evidence_test.sh (Failed)
Errors while running CTest
make[3]: *** [Makefile:84: test] Error 8
make[3]: Leaving directory '/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build'
PDO crypto tests failed. Define PDO_DEBUG_BUILD=1 environment for enabling logging in tests.
make[2]: *** [Makefile:25: test] Error 1
make[2]: Leaving directory '/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto'
make[1]: *** [Makefile:11: test] Error 2
make[1]: Leaving directory '/project/src/github.com/hyperledger/fabric-private-chaincode/common'
make: *** [Makefile:22: test] Error 2

Here is the environment vars:

NANOPB_PATH=/usr/local/nanopb/
PKG_CONFIG_PATH=:/opt/intel/sgxsdk/pkgconfig
HOSTNAME=ec
OPENSSL_VERSION=
SGX_MODE=SIM
PWD=/project/src/github.com/hyperledger/fabric-private-chaincode
SGXSSL_VERSION=
PROTOC_CMD=/usr/local/proto3/bin/protoc
SGX_SSL=/opt/intel/sgxssl
FPC_VERSION=
HOME=/root
FPC_PATH=/project/src/github.com/hyperledger/fabric-private-chaincode
DOCKERD_FPC_PATH=/home/ec/go/src/github.com/hyperledger/fabric-private-chaincode/
PROTO_DIR=/usr/local/proto3
FABRIC_PATH=/project/src/github.com/hyperledger/fabric
NANOPB_VERSION=0.4.3
GOROOT=/usr/local/go
GO_VERSION=1.16.7
LESSCLOSE=/usr/bin/lesspipe %s %s
TERM=xterm
LESSOPEN=| /usr/bin/lesspipe %s
SGX_SDK=/opt/intel/sgxsdk
SHLVL=1
LD_LIBRARY_PATH=:/opt/intel/sgxsdk/sdk_libs
PATH=/project/bin:/usr/local/go/bin:/opt/intel/sgxsdk.extras/external/toolset/ubuntu20.04:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/intel/sgxsdk/bin:/opt/intel/sgxsdk/bin/x64
CI=true
GOPATH=/project
_=/usr/bin/env
OLDPWD=/project/src/github.com/hyperledger/fabric-private-chaincode

Let me know if I am missing something.

azizkayumov commented 2 years ago

Previously, I didn't notice that these 2 tests were failing and probably I skipped to the hello world tutorial immediately.

mbrandenburger commented 2 years ago

@azizkayumov Aha! It seems that the FPC build (make -C $FPC_PATH) failed. In particular, the attestation API did not build due to ....:

Test project /project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build
    Start 1: utest
1/3 Test #1: utest ............................   Passed    0.50 sec
    Start 2: ttest
2/3 Test #2: ttest ............................***Exception: Illegal  0.23 sec

    Start 3: attested_evidence_test.sh
3/3 Test #3: attested_evidence_test.sh ........***Failed    0.16 sec
attested_evidence_test.sh: Testing simulated attestation
./attested_evidence_test.sh: line 27:  4880 Illegal instruction     (core dumped) ./get_attestation_app

We have seen similar issues trying to compile FPC on a recent MacBook with the ARM-based M1 ship.

@bvavala Bruno, should this code compile on his platform?

OS (distro+version): Ubuntu 20.04 VM running on Ubuntu 20.04.3 PC without SGX (AMD® Ryzen 5 3600 6-core processor × 12)

bvavala commented 2 years ago

@azizkayumov thanks for reporting this.

Would you please do the following in common/crypto: (I am assuming the project has been compiled once, so the dependencies of crypto are already met; if not, just recompile FPC from scratch first)

  1. export a debug variable export PDO_DEBUG_BUILD=1
  2. recompile crypto make clean; make, here the compilation should succeed, and the tests fail, as you reported
  3. cd _build/tests
  4. post the output of ./ttest and ./attested_evidence_test.sh

Also, you should be able to easily run gdb on the two executables (namely ttest and get_attestation_app) and determine what instruction is actually failing.

azizkayumov commented 2 years ago

Hello @bvavala.

Thank you for your help.

Here is the env:

root@ec:/project/src/github.com/hyperledger/fabric-private-chaincode/common/cry
pto/_build/tests# env
NANOPB_PATH=/usr/local/nanopb/
PKG_CONFIG_PATH=:/opt/intel/sgxsdk/pkgconfig
HOSTNAME=ec
OPENSSL_VERSION=
SGX_MODE=SIM
PWD=/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build/tests
SGXSSL_VERSION=
PROTOC_CMD=/usr/local/proto3/bin/protoc
SGX_SSL=/opt/intel/sgxssl
FPC_VERSION=
HOME=/root
FPC_PATH=/project/src/github.com/hyperledger/fabric-private-chaincode
DOCKERD_FPC_PATH=/home/ec/go/src/github.com/hyperledger/fabric-private-chaincode/
PROTO_DIR=/usr/local/proto3
FABRIC_PATH=/project/src/github.com/hyperledger/fabric
NANOPB_VERSION=0.4.3
GOROOT=/usr/local/go
GO_VERSION=1.16.7
LESSCLOSE=/usr/bin/lesspipe %s %s
TERM=xterm
LESSOPEN=| /usr/bin/lesspipe %s
SGX_SDK=/opt/intel/sgxsdk
SHLVL=1
LD_LIBRARY_PATH=:/opt/intel/sgxsdk/sdk_libs
PATH=/project/bin:/usr/local/go/bin:/opt/intel/sgxsdk.extras/external/toolset/ubuntu20.04:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/intel/sgxsdk/bin:/opt/intel/sgxsdk/bin/x64
CI=true
PDO_DEBUG_BUILD=1
OLDPWD=/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build
GOPATH=/project
_=/usr/bin/env

Here is the log from running ttest:

root@ec:/project/src/github.com/hyperledger/fabric-private-chaincode/common/cry
pto/_build/tests# ./ttest 
[LOG 0] Test TRUSTED Common API.
Illegal instruction (core dumped)
root@ec:/project/src/github.com/hyperledger/fabric-private-chaincode/common/cry
pto/_build/tests# gdb ./ttest 
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./ttest...
(No debugging symbols found in ./ttest)
(gdb) run
Starting program: /project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build/tests/ttest 
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[LOG 0] Test TRUSTED Common API.

Program received signal SIGILL, Illegal instruction.
0x00007f46aa18b5ce in ?? ()
(gdb) 

And here is the result of running get_attestation_app:

root@ec:/project/src/github.com/hyperledger/fabric-private-chaincode/common/cry
pto/_build/tests# ./get_attestation_app  
INFO  (/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/attestation-api/test/get_attestation_app/app/main.cpp:47) Testing init attestation

Illegal instruction (core dumped)
root@ec:/project/src/github.com/hyperledger/fabric-private-chaincode/common/cry
pto/_build/tests# gdb ./get_attestation_app 
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./get_attestation_app...
(No debugging symbols found in ./get_attestation_app)
(gdb) run
Starting program: /project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build/tests/get_attestation_app 
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
INFO  (/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/attestation-api/test/get_attestation_app/app/main.cpp:47) Testing init attestation

Program received signal SIGILL, Illegal instruction.
0x00007f44749d155e in ?? ()

Let me know if this can help.

bvavala commented 2 years ago

Thanks, but unfortunately it does not help.

Let's focus on get_attestation_app. We have to understand what's failing at 0x00007f44749d155e.

I have pushed #661 . It should add some more symbols for gdb. Please check it out and apply that to your code. You have to export FPC_DEBUG_ENABLED=true first. Then a make clean; make should be sufficient -- you can double check that "FPC Debug enabled" is displayed during the compilation. Now, hopefully gdb will tell you what instruction failed -- or maybe not, particularly for the pre-built sgx libraries. Also, you can call backtrace to see the stack frames, and layout asm to see the the instructions being executed.

Additionally, please post the output of ldd get_attestation_app which displays the shared libraries and their addresses; and please post again the address of the illegal instruction, which will likely change after you recompile.

azizkayumov commented 2 years ago

Sorry for the confusion. I forgot to make clean; make before. Here are what I followed:

  1. git pull origin pull/661/head
  2. export FPC_DEBUG_ENABLED=true
  3. make clean && make
  4. cd common/crypto/_build/tests
  5. ./get_attestation_app
  6. gdb get_attestation_app
  7. (gdb) run
  8. Got the same ??:
    
    (gdb) run
    Starting program: /project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/_build/tests/get_attestation_app 
    warning: Error disabling address space randomization: Operation not permitted
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    INFO  (/project/src/github.com/hyperledger/fabric-private-chaincode/common/crypto/attestation-api/test/get_attestation_app/app/main.cpp:47) Testing init attestation

Program received signal SIGILL, Illegal instruction. 0x00007f0f6c54555e in ?? ()

9. `(gdb) backtrace` output:

(gdb) backtrace

0 0x00007f0f6c54555e in ?? ()

1 0x00007f0f6c3844dc in ?? ()

2 0x0000000000000000 in ?? ()

10. `layout asm`:

┌─────────────────────────────────────────────────────────────────────────────┐ │ >0x7f0f6c54555e ud2 │ │ 0x7f0f6c545560 endbr64 │ │ 0x7f0f6c545564 mov %rdi,%rcx │ │ 0x7f0f6c545567 mov (%rcx),%rax │ │ 0x7f0f6c54556a push %rax │ │ 0x7f0f6c54556b mov 0x8(%rcx),%rax │ │ 0x7f0f6c54556f push %rax │ │ 0x7f0f6c545570 mov 0x20(%rcx),%rax │ │ 0x7f0f6c545574 sub $0x88,%rax │ │ 0x7f0f6c54557a mov 0x10(%rcx),%rdx │ │ 0x7f0f6c54557e mov 0x18(%rcx),%rbx │ │ 0x7f0f6c545582 mov 0x28(%rcx),%rbp │ │ 0x7f0f6c545586 mov 0x30(%rcx),%rsi │ │ 0x7f0f6c54558a mov 0x38(%rcx),%rdi │ │ 0x7f0f6c54558e mov 0x40(%rcx),%r8 │ │ 0x7f0f6c545592 mov 0x48(%rcx),%r9 │ │ 0x7f0f6c545596 mov 0x50(%rcx),%r10 │ └─────────────────────────────────────────────────────────────────────────────┘ multi-thre Thread 0x7f0f6d6da7 In: L?? PC: 0x7f0f6c54555e (gdb)

11. `ldd get_attestation_app`:

root@ec:/project/src/github.com/hyperledger/fabric-private-chaincode/common/cry pto/_build/tests# ldd get_attestation_app linux-vdso.so.1 (0x00007ffd25708000) libsgx_urts_sim.so => /opt/intel/sgxsdk/sdk_libs/libsgx_urts_sim.so (0x00007f1dea2fe000) libsgx_epid_sim.so => /opt/intel/sgxsdk/sdk_libs/libsgx_epid_sim.so (0x00007f1dea2f8000) libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1dea111000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1dea0f6000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1de9f04000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1de9efe000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1de9ed9000) libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f1de9c03000) libsgx_uae_service_sim.so => /opt/intel/sgxsdk/sdk_libs/libsgx_uae_service_sim.so (0x00007f1de9bfd000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1de9aae000) /lib64/ld-linux-x86-64.so.2 (0x00007f1dea331000)


I hope this helps. I will try to do the FPC simulation in Intel-based laptop as well (without SGX). I see that it is not reasonable to try the simulation on my AMD-based PC, since my use case requires IntelSGX in production anyways.  
bvavala commented 2 years ago

Thanks for reporting this. This shows that the program is not trying to execute an instruction that is only available on Intel processors. Unfortunately, the program is taking an error path, but it's not immediately apparent which one and why. A deeper analysis is needed. The laptop is likely the best alternative at this point.

azizkayumov commented 2 years ago

I tested on the Intel-based laptop, everything worked like a charm. Thank you for your help, and for your presentations on Youtube, they were really helpful in understanding your paper.