PPML attestation validation

[Erik-ly]

hi, I want to test the PPML attestation on docker, but the result is not as expected. The following is my test environment and steps, hoping to get help,thanks.

Environment info:

[root@node1 opt]# uname -sr
Linux 4.18.0-305.10.2.8.kux.x86_64
[root@node1 opt]# 
[root@node1 ~]# ls /dev | grep sgx
sgx_enclave
sgx_provision
[root@node1 ~]# 
[root@node1 ~]# docker images | grep bigdata
intelanalytics/bigdl-ppml-trusted-bigdata-gramine-noattest-32g              2.4.0-SNAPSHOT   2d15310fc806   3 weeks ago     5.59GB
[root@node1 ~]# 

SGX PRMRR Size is 64G.

Test Steps:

1. Sign SGX enclave 1 with private key 1

​ a. Generate private key 1

[root@node1 output]# cd /opt/BigDL/ppml/scripts
[root@node1 scripts]# ll
total 16
-rw------- 1 root root 2459 Aug 17 09:42 enclave-key.pem
-rw------- 1 root root 1635 Jul 11  2022 generate-keys.sh
-rw------- 1 root root  613 Jul 11  2022 generate-password.sh
drwx------ 2 root root  138 Aug 17 09:47 keys
drwx------ 2 root root   60 Aug 17 09:48 password
-rw------- 1 root root  645 Jul 11  2022 start-fl-server.sh
[root@node1 scripts]# openssl genrsa -3 -out private-key1.pem 3072
Generating RSA private key, 3072 bit long modulus (2 primes)
.................................................++++
...........................................++++
e is 3 (0x03)
[root@node1 scripts]# 
[root@node1 scripts]# ll
total 20
-rw------- 1 root root 2459 Aug 17 09:42 enclave-key.pem
-rw------- 1 root root 1635 Jul 11  2022 generate-keys.sh
-rw------- 1 root root  613 Jul 11  2022 generate-password.sh
drwx------ 2 root root  138 Aug 17 09:47 keys
drwx------ 2 root root   60 Aug 17 09:48 password
-rw------- 1 root root 2455 Aug 17 16:21 private-key1.pem
-rw------- 1 root root  645 Jul 11  2022 start-fl-server.sh
[root@node1 scripts]# 
[root@node1 scripts]# sha256sum private-key1.pem 
43ace69ef0c8a0870c5ec912707c7c7555e88ba2dfb6c45e20736c3cfacc1a6f  private-key1.pem
[root@node1 scripts]# 

b. Deploy SGX enclave 1 docker by private key 1

[root@node1 docker-graphene]# cat deploy-sgx-enclave-1-by-private-key-1.sh 
#!/bin/bash

export K8S_MASTER=k8s://$(kubectl cluster-info | grep 'https.*6443' -o -m 1)
echo The k8s master is $K8S_MASTER .
export ENCLAVE_KEY=/opt/BigDL/ppml/scripts/private-key1.pem
export DATA_PATH=/insight/data1/tpc-h-data/tpc-h-data-100g
export KEYS_PATH=/opt/BigDL/ppml/trusted-big-data-ml/python/docker-graphene/keys
export SECURE_PASSWORD_PATH=/opt/BigDL/ppml/trusted-big-data-ml/python/docker-graphene/password
export KUBECONFIG_PATH=/opt/BigDL/ppml/trusted-big-data-ml/python/docker-graphene/kubeconfig/kubeconfig
export LOCAL_IP=100.18.xx.xxx
export DOCKER_IMAGE=intelanalytics/bigdl-ppml-trusted-bigdata-gramine-noattest-32g:2.4.0-SNAPSHOT
sudo docker run -itd \
    --privileged \
    --net=host \
    --name=sgx-enclave-1 \
    --cpuset-cpus="0-4" \
    --oom-kill-disable \
    --device=/dev/sgx/enclave \
    --device=/dev/sgx/provision \
    -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket \
    -v $ENCLAVE_KEY:/root/.config/gramine/enclave-key.pem \
    -v $DATA_PATH:/ppml/trusted-big-data-ml/work/data \
    -v $KEYS_PATH:/ppml/trusted-big-data-ml/work/keys \
    -v $SECURE_PASSWORD_PATH:/ppml/trusted-big-data-ml/work/password \
    -v $KUBECONFIG_PATH:/root/.kube/config \
    -e RUNTIME_SPARK_MASTER=$K8S_MASTER \
    -e RUNTIME_K8S_SERVICE_ACCOUNT=spark \
    -e RUNTIME_K8S_SPARK_IMAGE=$DOCKER_IMAGE \
    -e RUNTIME_DRIVER_HOST=$LOCAL_IP \
    -e RUNTIME_DRIVER_PORT=54321 \
    -e RUNTIME_DRIVER_CORES=1 \
    -e RUNTIME_EXECUTOR_INSTANCES=1 \
    -e RUNTIME_EXECUTOR_CORES=8 \
    -e RUNTIME_EXECUTOR_MEMORY=1g \
    -e RUNTIME_TOTAL_EXECUTOR_CORES=4 \
    -e RUNTIME_DRIVER_CORES=4 \
    -e RUNTIME_DRIVER_MEMORY=1g \
    -e SGX_DRIVER_MEM=32g \
    -e SGX_DRIVER_JVM_MEM=8g \
    -e SGX_EXECUTOR_MEM=32g \
    -e SGX_EXECUTOR_JVM_MEM=12g \
    -e SGX_ENABLED=true \
    -e SGX_LOG_LEVEL=error \
    -e SPARK_MODE=client \
    -e LOCAL_IP=$LOCAL_IP \
    $DOCKER_IMAGE bash
[root@node1 docker-graphene]# 
[root@node1 docker-graphene]# sh deploy-sgx-enclave-1-by-private-key-1.sh 
The k8s master is k8s://https://100.18.xx.xxx:6443 .
WARNING: Disabling the OOM killer on containers without setting a '-m/--memory' limit may be dangerous.
WARNING: OOM killer is disabled for the container, but no memory limit is set, this can result in the system running out of resources.
4e06eaf434434e466301b24a70ad8e930a229efd4d145922f63d74a75155bd51
[root@node1 docker-graphene]# docker ps | grep sgx
4e06eaf43443   intelanalytics/bigdl-ppml-trusted-bigdata-gramine-noattest-32g:2.4.0-SNAPSHOT              "/opt/entrypoint.sh …"   13 seconds ago   Up 13 seconds             sgx-enclave-1
[root@node1 docker-graphene]# 

c. sign SGX enclave 1 with make command

[root@node1 ~]# docker exec -it sgx-enclave-1 bash
root@node1:/ppml# make SGX=1 THIS_DIR=/ppml G_SGX_SIZE=16G G_LOG_LEVEL=error
gramine-sgx-get-token --output bash.token --sig bash.sig
Attributes:
    mr_enclave:  50e34080d48c01bc41ddff7e0c900be0c3bb7e04104cec2b07d39a47b953cd3f
    mr_signer:   b2f16e6cddae4cb55350b22c087e79f756edbe518b781edd481db448b30a8244
    isv_prod_id: 1
    isv_svn:     3
    attr.flags:  0000000000000004
    attr.xfrm:   00000000000000e7
    mask.flags:  ffffffffffffffff
    mask.xfrm:   fffffffffff9ff1b
    misc_select: 00000000
    misc_mask:   ffffffff
    modulus:     81a35ff2f751c862de83de10c92e4555...
    exponent:    3
    signature:   a7bb0aa812fed8cfc6530ccc744a9706...
    date:        2023-08-04
root@node1:/ppml# 

check the enclave-key.pem(private-key1.pem)

root@node1:/ppml# ll /root/.config/gramine/enclave-key.pem
-rw------- 1 root root 2455 Aug 17 08:21 /root/.config/gramine/enclave-key.pem
root@node1:/ppml# sha256sum /root/.config/gramine/enclave-key.pem
43ace69ef0c8a0870c5ec912707c7c7555e88ba2dfb6c45e20736c3cfacc1a6f  /root/.config/gramine/enclave-key.pem
root@node1:/ppml# 

2. Sign SGX enclave 2 with private key 2

​ a. Generate private key 2

[root@node1 scripts]# openssl genrsa -3 -out private-key2.pem 3072
Generating RSA private key, 3072 bit long modulus (2 primes)
...........................++++
...........................++++
e is 3 (0x03)
[root@node1 scripts]# ll
total 24
-rw------- 1 root root 2459 Aug 17 09:42 enclave-key.pem
-rw------- 1 root root 1635 Jul 11  2022 generate-keys.sh
-rw------- 1 root root  613 Jul 11  2022 generate-password.sh
drwx------ 2 root root  138 Aug 17 09:47 keys
drwx------ 2 root root   60 Aug 17 09:48 password
-rw------- 1 root root 2455 Aug 17 16:21 private-key1.pem
-rw------- 1 root root 2459 Aug 17 16:40 private-key2.pem
-rw------- 1 root root  645 Jul 11  2022 start-fl-server.sh
[root@node1 scripts]# 
[root@node1 scripts]# sha256sum private-key2.pem 
b310b5dcf1a2cd5e1ad7548e6ecee77a983b1ab5a5544d36970178291223aa1a  private-key2.pem
[root@node1 scripts]# 

b. Deploy SGX enclave 2 docker by private key 2

[root@node1 docker-graphene]# cat deploy-sgx-enclave-2-by-private-key-2.sh 
#!/bin/bash

export K8S_MASTER=k8s://$(kubectl cluster-info | grep 'https.*6443' -o -m 1)
echo The k8s master is $K8S_MASTER .
export ENCLAVE_KEY=/opt/BigDL/ppml/scripts/private-key2.pem
export DATA_PATH=/insight/data1/tpc-h-data/tpc-h-data-100g
export KEYS_PATH=/opt/BigDL/ppml/scripts/keys
export SECURE_PASSWORD_PATH=/opt/BigDL/ppml/scripts/password
export KUBECONFIG_PATH=/opt/BigDL/ppml/trusted-big-data-ml/python/docker-graphene/kubeconfig/kubeconfig
export LOCAL_IP=100.18.xx.xxx
export DOCKER_IMAGE=intelanalytics/bigdl-ppml-trusted-bigdata-gramine-noattest-32g:2.4.0-SNAPSHOT
sudo docker run -itd \
    --privileged \
    --net=host \
    --name=sgx-enclave-2 \
    --cpuset-cpus="0-4" \
    --oom-kill-disable \
    --device=/dev/sgx/enclave \
    --device=/dev/sgx/provision \
    -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket \
    -v $ENCLAVE_KEY:/root/.config/gramine/enclave-key.pem \
    -v $DATA_PATH:/ppml/trusted-big-data-ml/work/data \
    -v $KEYS_PATH:/ppml/trusted-big-data-ml/work/keys \
    -v $SECURE_PASSWORD_PATH:/ppml/trusted-big-data-ml/work/password \
    -v $KUBECONFIG_PATH:/root/.kube/config \
    -e RUNTIME_SPARK_MASTER=$K8S_MASTER \
    -e RUNTIME_K8S_SERVICE_ACCOUNT=spark \
    -e RUNTIME_K8S_SPARK_IMAGE=$DOCKER_IMAGE \
    -e RUNTIME_DRIVER_HOST=$LOCAL_IP \
    -e RUNTIME_DRIVER_PORT=54321 \
    -e RUNTIME_DRIVER_CORES=1 \
    -e RUNTIME_EXECUTOR_INSTANCES=1 \
    -e RUNTIME_EXECUTOR_CORES=8 \
    -e RUNTIME_EXECUTOR_MEMORY=1g \
    -e RUNTIME_TOTAL_EXECUTOR_CORES=4 \
    -e RUNTIME_DRIVER_CORES=4 \
    -e RUNTIME_DRIVER_MEMORY=1g \
    -e SGX_DRIVER_MEM=32g \
    -e SGX_DRIVER_JVM_MEM=8g \
    -e SGX_EXECUTOR_MEM=32g \
    -e SGX_EXECUTOR_JVM_MEM=12g \
    -e SGX_ENABLED=true \
    -e SGX_LOG_LEVEL=error \
    -e SPARK_MODE=client \
    -e LOCAL_IP=$LOCAL_IP \
    $DOCKER_IMAGE bash
[root@node1 docker-graphene]# 
[root@node1 docker-graphene]# sh deploy-sgx-enclave-2-by-private-key-2.sh 
The k8s master is k8s://https://100.18.xx.xxx:6443 .
WARNING: Disabling the OOM killer on containers without setting a '-m/--memory' limit may be dangerous.
WARNING: OOM killer is disabled for the container, but no memory limit is set, this can result in the system running out of resources.
13d9c0365b5e114af95c846f0757d7b9d39f1db6865f34b17bbfd93e753526c4
[root@node1 docker-graphene]# docker ps | grep sgx
13d9c0365b5e   intelanalytics/bigdl-ppml-trusted-bigdata-gramine-noattest-32g:2.4.0-SNAPSHOT              "/opt/entrypoint.sh …"   10 seconds ago   Up 10 seconds             sgx-enclave-2
4e06eaf43443   intelanalytics/bigdl-ppml-trusted-bigdata-gramine-noattest-32g:2.4.0-SNAPSHOT              "/opt/entrypoint.sh …"   6 minutes ago    Up 6 minutes              sgx-enclave-1
[root@node1 docker-graphene]# 

c. sign SGX enclave 2 with make command

[root@node1 ~]# docker exec -it sgx-enclave-2 bash
root@node1:/ppml# make SGX=1 THIS_DIR=/ppml G_SGX_SIZE=16G G_LOG_LEVEL=error
gramine-sgx-get-token --output bash.token --sig bash.sig
Attributes:
    mr_enclave:  50e34080d48c01bc41ddff7e0c900be0c3bb7e04104cec2b07d39a47b953cd3f
    mr_signer:   b2f16e6cddae4cb55350b22c087e79f756edbe518b781edd481db448b30a8244
    isv_prod_id: 1
    isv_svn:     3
    attr.flags:  0000000000000004
    attr.xfrm:   00000000000000e7
    mask.flags:  ffffffffffffffff
    mask.xfrm:   fffffffffff9ff1b
    misc_select: 00000000
    misc_mask:   ffffffff
    modulus:     81a35ff2f751c862de83de10c92e4555...
    exponent:    3
    signature:   a7bb0aa812fed8cfc6530ccc744a9706...
    date:        2023-08-04
root@node1:/ppml# 

check the enclave-key.pem(private-key2.pem)

root@node1:/ppml# ll /root/.config/gramine/enclave-key.pem
-rw------- 1 root root 2459 Aug 17 08:40 /root/.config/gramine/enclave-key.pem
root@node1:/ppml# sha256sum ll /root/.config/gramine/enclave-key.pem
sha256sum: ll: No such file or directory
b310b5dcf1a2cd5e1ad7548e6ecee77a983b1ab5a5544d36970178291223aa1a  /root/.config/gramine/enclave-key.pem
root@node1:/ppml# 

Observe the output, mr_signer in container sgx-enclave-1 and container sgx-enclave-2 is the same.

I have verified that pem is different, the output of using the command "openssl rsa -in /root/.config/gramine/enclave-key.pem" is also different.

Please check the test steps and tell me the cause of the problem, thank you.

[xiangyuT]

The docker image intelanalytics/bigdl-ppml-trusted-bigdata-gramine-noattest-32g:2.4.0-SNAPSHOT is a reference image we have already built an enclave in it, which means if you want to make a SGX enclave by commands, you need to use intelanalytics/bigdl-ppml-trusted-bigdata-gramine-base:2.4.0-SNAPSHOT, or execute bash ./clean.sh in /ppml dir. Otherwise, the make command will show the mr_enclave and mr_signer of our built SGX enclave, and the enclave-key.pem you provided will not effect.

And mr_signer is the SHA-256 hash of the modulus of the enclave key. A method to check mr_signer is provided here:

1. Generate modulus of your private key

   openssl rsa -in enclave-key.pem -noout -modulus

   

You can verify the modulus with the output of make or gramine-sgx-get-token. It needs endian conversion in our example.

2. Calculate SHA-256 hash. Following is a Python example:

   
   import hashlib

hex_integer = "D1D2889EC4F695F93E74A868A5BFB1C390AA66C918F10178CB7F5D1BC4CE53F5BA2561E7E32E2D394587E3798209862CAE82A127E7FF9F6CA7B0B25E590D2863D6F6E86723C0B63B7E7EB7DBD539A0AB402F4078711C9ABEC5FC62745F337994EB53810260A92DEAF9BA8676CF2BDC209A58B665C285013948B14B56C2A40C807E28FDAC2CB7AE96ADE0AEBECB1A47E89A55E46C6CC9114C93EFCA67C4D5D2D7672FC1A6F6558ACA26539C982B6891A1CF53696A2E44F8641D77A83BABBD9CB104C69EFD5FFBBCB2DDDFF85D0D48AB3652B1B274D3DFF94E8045E39A1ECB1CE8FF6504FBE7C241A5BF735F13EC604B853E0A668746C7B0CDE2D7A050ADF2612BD090434F5D4D77B2615461FA5E87D3F1E3FCD80D7125B532A1F6238A0C44F8D15C1AD1E6759F5D2B536762ADF66D3227D312B41ACD64FA2EC0AF65B9014C30605C6A4FDDCBB23093605EC250C84418BFD959EEB76946C26F96C87D9A9AEEE44945F817AA905A96F5FBD92FF1E0202E434F8B77DD1C55DDD7B4E5926F12BEBADD" # the result you got from first step

byte_string = bytes.fromhex(hex_integer) hash_object = hashlib.sha256(byte_string[::-1]) # Endian conversion hash_value = hash_object.hexdigest()

print(hash_value)

You can check the value with `mr_signer` in Attributes.
<img width="410" alt="image" src="https://github.com/intel-analytics/BigDL/assets/109123695/fb9e164a-43f2-4695-8a82-16f32a195e5d">

[Erik-ly]

Thanks. I use intelanalytics/bigdl-ppml-trusted-bigdata-gramine-base:2.4.0-SNAPSHOT, and execute bash ./clean.sh in /ppml dir，then execute make cmd，get the different mr_signer.

when I execute openssl rsa -in enclave-key.pem -noout -modulus, output rsa:Unrecognized flag moudlus. can't get the modulus info.Do I need to execute other commands?

[xiangyuT]

Thanks. I use intelanalytics/bigdl-ppml-trusted-bigdata-gramine-base:2.4.0-SNAPSHOT, and execute bash ./clean.sh in /ppml dir，then execute make cmd，get the different mr_signer.

when I execute openssl rsa -in enclave-key.pem -noout -modulus, output rsa:Unrecognized flag moudlus. can't get the modulus info.Do I need to execute other commands?

The output you provided logs moudlus, which might be a mistaken spell of modulus. Could you check it in your commands?

If the openssl command won't effect, you can try another python code below: (based on cryptography==3.3.2)

from cryptography.hazmat.primitives import serialization
import hashlib
private_key_file = "/ppml/enclave-key.pem"

with open(private_key_file, "rb") as key_file:
    private_key = serialization.load_pem_private_key(key_file.read(), password=None)

modulus = private_key.private_numbers().public_numbers.n
modulus_hex = hex(modulus)[2:]
print("Private Key Modulus:", modulus_hex)

byte_string = bytes.fromhex(modulus_hex)
hash_object = hashlib.sha256(byte_string[::-1]) # Endian conversion
hash_value = hash_object.hexdigest()

print(hash_value)

[Erik-ly]

The output you provided logs moudlus, which might be a mistaken spell of modulus. Could you check it in your commands?

Sorry, it is indeed a misspelled command. I use the command openssl rsa -in /root/.config/gramine/enclave-key.pem -noout -modulus to get the Modulus value.

Then，I use the method Calculate SHA-256 hash., and I can get the same value as mr_signer.

Thank you.