search

gramineproject / gsc

Gramine Shielded Containers (Docker integration)
BSD 3-Clause "New" or "Revised" License
44 stars 37 forks source link

GSC failed to run signed Docker image #162

Open NandiniKJ opened 1 year ago

NandiniKJ commented 1 year ago

Description of the problem

We are trying to run a Docker image using Gramine shielded containers. We are successfully able to create the signed image but the container is throwing the below error when we try to run the signed docker image.

Screenshot 2023-07-17 at 12 04 20 PM

We have been following this link: https://gramine.readthedocs.io/projects/gsc/en/latest/

We were able to bring this application up with normal container deployment and were able to verify that the start-baffle-shiled.sh file exists. After converting to and running inside Gramine container I found that the opt folder is empty.

Could you help us with this, are we missing something here.

Steps to reproduce

Expected results

Actual results

mkow commented 1 year ago

Please don't paste screenshots of text, instead just copy and paste the text itself. It's hard to read and makes it impossible to copy and search in it.

dimakuv commented 1 year ago

After converting to and running inside Gramine container I found that the opt folder is empty.

This doesn't sound right. GSC doesn't do anything with the /opt directory. So I don't know why it becomes empty.

NandiniKJ commented 1 year ago

I am able to run the docker image without Gramine and it works fine. Please find the below output. After converting it to Gramine container should we mount the /opt path or change the entrypoint.sh. I'm I missing something here.

`SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/baffle/Release-Baffle.1.7.0.161/shield/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/baffle/Release-Baffle.1.7.0.161/shield/bm-connector.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2023-07-17 10:49:08,086 1 INFO CommonConfig:508 - baffle.config.bmShieldSyncID is defined, using value from system parameter 2023-07-17 10:49:08,090 1 INFO BMShieldClient:69 - Initializing BMShieldClient 2023-07-17 10:49:08,091 1 INFO BMShieldClient:760 - baffle.config.path is defined, using value /opt/baffle/Release-Baffle.1.7.0.161/shield 2023-07-17 10:49:08,091 1 INFO BMShieldClient:756 - baffle.ssl.path undefined, loading from default value at /opt/baffle/ssl 2023-07-17 10:49:08,091 1 INFO BMShieldClient:761 - baffle.config.bmShieldSyncID is defined, using value from system parameter 2023-07-17 10:49:08,201 1 INFO BMShieldClient:756 - baffle.config.shieldTag undefined, loading from default value at 2023-07-17 10:49:08,201 1 INFO BMShieldClient:760 - baffle.config.bm.ip is defined, using value dsb-manager-dsb-for-nandini.dsb-roks-vpc-412-c9b7119538b194dae4a1958742b244b0-0000.eu-de.containers.appdomain.cloud 2023-07-17 10:49:08,201 1 INFO BMShieldClient:756 - baffle.config.bm.port undefined, loading from default value at 443 2023-07-17 10:49:08,202 1 INFO BMShieldClient:756 - baffle.config.bm.tenant undefined, loading from default value at ibm 2023-07-17 10:49:08,202 1 INFO BMShieldClient:756 - baffle.config.bm.appID undefined, loading from default value at 64acfabd65fab6124f7872fa 2023-07-17 10:49:08,202 1 INFO BMShieldClient:756 - baffle.config.bm.shieldID undefined, loading from default value at null 2023-07-17 10:49:08,202 1 INFO BMRestClient:58 - Checking BM Connectivity using URL - https://dsb-manager-dsb-for-nandini.dsb-roks-vpc-412-c9b7119538b194dae4a1958742b244b0-0000.eu-de.containers.appdomain.cloud:443/ 2023-07-17 10:49:09,225 1 INFO BMShieldClient:795 - baffle.shield.ssl is defined, using value true 2023-07-17 10:49:09,376 1 INFO BMShieldClient:191 - Registration Successful ShieldID =64b3f41765fab6124f78731f 2023-07-17 10:49:09,377 1 INFO BMShieldClient:205 - Getting Shield Initial Configuration 2023-07-17 10:49:09,522 1 INFO BMShieldClient:318 - Downloading Config File : BafflePrivacySchema 2023-07-17 10:49:09,664 1 INFO BMShieldClient:318 - Downloading Config File : BaffleEntitySchema 2023-07-17 10:49:09,803 1 INFO BMShieldClient:318 - Downloading Config File : KmsConfig.properties 2023-07-17 10:49:09,955 1 INFO BMShieldClient:318 - Downloading Config File : BaffleCommonConfig 2023-07-17 10:49:10,101 1 INFO CommonConfig:1156 - Tier: 0 2023-07-17 10:49:10,102 1 INFO CommonConfig:1067 - Nothing to do for the deployment type local 2023-07-17 10:49:10,136 1 INFO KmsConfig:59 - KeyStore type - local 2023-07-17 10:49:10,136 1 INFO KmsConfig:63 - local : Configured application namespace - null 2023-07-17 10:49:10,136 1 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked 2023-07-17 10:49:10,139 1 INFO KmsConfig:59 - KeyStore type - local 2023-07-17 10:49:10,140 1 INFO KmsConfig:63 - local : Configured application namespace - null 2023-07-17 10:49:10,140 1 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked 2023-07-17 10:49:10,141 1 INFO BaffleSecretStoreUtil:587 - No SecretStores configured 2023-07-17 10:49:10,142 1 WARN CommonConfig:1324 - defaulting value of encMode as M_CTR 2023-07-17 10:49:10,142 1 INFO CommonConfig:465 - baffle.shield.ssl is defined, using value true 2023-07-17 10:49:10,143 1 INFO CommonConfig:507 - baffle.shield.keystore is defined, using value /opt/sslconfig/baffleshield-keystore.jks 2023-07-17 10:49:10,143 1 INFO CommonConfig:508 - baffle.shield.keystore.password is defined, using value from system parameter 2023-07-17 10:49:10,143 1 INFO CommonConfig:507 - baffle.shield.truststore is defined, using value /opt/sslconfig/baffleshield-keystore.jks 2023-07-17 10:49:10,143 1 INFO CommonConfig:508 - baffle.shield.truststore.password is defined, using value from system parameter 2023-07-17 10:49:10,143 1 INFO CommonConfig:507 - baffle.shield.ssl.tlsVersion is defined, using value TLSv1.2 2023-07-17 10:49:10,145 1 INFO ConfigReaderUtil:398 - File: /opt/baffle/Release-Baffle.1.7.0.161/shield/BaffleCommonConfig baffleHeartbeatInterval=30000 .... 2023-07-17 10:49:10,146 1 INFO KmsConfig:59 - KeyStore type - local 2023-07-17 10:49:10,146 1 INFO KmsConfig:63 - local : Configured application namespace - null 2023-07-17 10:49:10,146 1 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked 2023-07-17 10:49:10,146 1 INFO KmsConfig:59 - KeyStore type - local 2023-07-17 10:49:10,147 1 INFO KmsConfig:63 - local : Configured application namespace - null 2023-07-17 10:49:10,147 1 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked 2023-07-17 10:49:10,147 1 INFO KmsConfig:59 - KeyStore type - local 2023-07-17 10:49:10,147 1 INFO KmsConfig:63 - local : Configured application namespace - null 2023-07-17 10:49:10,147 1 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked 2023-07-17 10:49:10,149 1 INFO ConfigReaderUtil:398 - File: /opt/baffle/Release-Baffle.1.7.0.161/shield/KmsConfig.properties baffle_secret=*** kmsType=local

2023-07-17 10:49:10,175 1 INFO CommonConfig:507 - baffle.config.path is defined, using value /opt/baffle/Release-Baffle.1.7.0.161/shield 2023-07-17 10:49:10,176 1 INFO PrivacySchemaReaderToml:106 - Reading from BafflePrivacySchema 2023-07-17 10:49:10,187 1 INFO TomlPrivacySchemaReader:60 - Nothing to encrypt. Empty BafflePrivacySchema 2023-07-17 10:49:10,189 1 INFO PrivacySchemaReaderToml:106 - Reading from BafflePrivacySchema 2023-07-17 10:49:10,189 1 INFO TomlPrivacySchemaReader:60 - Nothing to encrypt. Empty BafflePrivacySchema 2023-07-17 10:49:10,190 1 INFO PrivacySchemaReaderToml:194 - Copying BafflePrivacySchema to .BafflePrivacySchema.Verified 2023-07-17 10:49:10,192 1 INFO BaffleMonitorMetric:42 - Baffle Health Metrics bean registration complete 2023-07-17 10:49:10,288 15 INFO BaffleShield:392 - **** 2023-07-17 10:49:10,289 15 INFO BaffleShield:393 - ** BAFFLESHIELD ** 2023-07-17 10:49:10,289 15 INFO BaffleShield:394 - **** 2023-07-17 10:49:10,289 15 INFO BaffleShield:395 - Deployment type: local 2023-07-17 10:49:10,295 15 INFO CommonConfig:756 - Baffle Release Version: Release-Baffle.1.7.0.161 2023-07-17 10:49:10,295 15 INFO BaffleShield:397 - Parser version: 2.6.5.4-baffle2 2023-07-17 10:49:10,296 15 INFO BaffleShield:398 - JVM bit size: 64 2023-07-17 10:49:10,296 15 INFO BaffleShield:399 - Free memory: 179432360 Total memory: 204472320 Max memory: 3674210304 2023-07-17 10:49:10,296 15 INFO BaffleShield:401 - nofile limit : 1048576 2023-07-17 10:49:10,296 15 INFO BaffleShield:403 - Using OpenSSL version: BoringSSL 2023-07-17 10:49:10,296 15 INFO BaffleShield:412 - Using BaffleCommonConfig at: /opt/baffle/Release-Baffle.1.7.0.161/shield 2023-07-17 10:49:10,296 15 INFO BaffleShield:413 - Using Credential store configuration at: /opt/baffle/Release-Baffle.1.7.0.161/shield/credstore 2023-07-17 10:49:10,297 15 INFO BaffleShield:417 - Proxying *:8444 to f637666e-dc0a-48ac-806d-8e6af4046a2c.c9v3nahd0oekcvsra2t0.databases.appdomain.cloud:31841 2023-07-17 10:49:10,297 15 INFO BaffleShield:437 - BaffleShutdownHook Thread Started 2023-07-17 10:49:10,428 15 INFO CommonConfig:507 - baffle.config.path is defined, using value /opt/baffle/Release-Baffle.1.7.0.161/shield 2023-07-17 10:49:10,428 15 INFO KmsConfig:59 - KeyStore type - local 2023-07-17 10:49:10,428 15 INFO KmsConfig:63 - local : Configured application namespace - null 2023-07-17 10:49:10,428 15 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked 2023-07-17 10:49:10,429 15 INFO ConfigReaderUtil:398 - File: /opt/baffle/Release-Baffle.1.7.0.161/shield/BafflePrivacySchema format = "TOML"

2023-07-17 10:49:10,429 15 INFO PrivacySchemaReaderToml:106 - Reading from BafflePrivacySchema 2023-07-17 10:49:10,430 15 INFO TomlPrivacySchemaReader:60 - Nothing to encrypt. Empty BafflePrivacySchema 2023-07-17 10:49:10,430 15 INFO TransformDB:460 - Privacy schema update: 2023-07-17 10:49:10,432 15 INFO PrivacySchemaHandler:92 - Starting BafflePrivacySchema Watcher on /opt/baffle/Release-Baffle.1.7.0.161/shield/BafflePrivacySchema `

dimakuv commented 1 year ago

Can you show the docker run command with which you run:

  1. the original app image,
  2. the GSC-generated image
NandiniKJ commented 1 year ago

Original app image command: docker run --rm -it -e BM_IP=dsb-manager-dsb-for-nandini.dsb-roks-vpc-412-c9b7119538b194dae4a1958742b244b0-0000.eu-de.containers.appdomain.cloud -e BM_SHIELD_SYNC_ID="*****" -e BM_SHIELD_TAG=dsb-shield-app1 -e BS_SSL=true -e BS_SSL_KEYSTORE_FILE=/opt/sslconfig/baffleshield-keystore.jks -e BS_SSL_KEYSTORE_PASSWORD=keystore -e BS_SSL_TRUSTSTORE_FILE=/opt/sslconfig/baffleshield-keystore.jks -e BS_SSL_TRUSTSTORE_PASSWORD=keystore -e BS_SSL_TLS_VERSION=TLSv1.2 -e KMS_CONFIG_PROPERTIES="{'baffle_secret':'*****','kmsType': 'local'}" icr.io/data-security-broker/dsb-shield-postgresql-v1:v1

GSC generated image:

docker run --rm -it -e BM_IP=dsb-manager-dsb-for-nandini.dsb-roks-vpc-412-c9b7119538b194dae4a1958742b244b0-0000.eu-de.containers.appdomain.cloud -e BM_SHIELD_SYNC_ID="*****" -e BM_SHIELD_TAG=dsb-shield-app1 -e BS_SSL=true -e BS_SSL_KEYSTORE_FILE=/opt/sslconfig/baffleshield-keystore.jks -e BS_SSL_KEYSTORE_PASSWORD=keystore -e BS_SSL_TRUSTSTORE_FILE=/opt/sslconfig/baffleshield-keystore.jks -e BS_SSL_TRUSTSTORE_PASSWORD=keystore -e BS_SSL_TLS_VERSION=TLSv1.2 -e KMS_CONFIG_PROPERTIES="{'baffle_secret':'*****','kmsType': 'local'}" --device=/dev/sgx_enclave -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket gsc-icr.io/data-security-broker/dsb-shield-postgresql-v1:v1

I even tried to run the gsc container command without passing the environment variables. docker run --device=/dev/sgx_enclave \ -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket \ gsc-icr.io/data-security-broker/dsb-shield-postgresql-v1:v1

dimakuv commented 1 year ago

Have you tried to enter this GSC-generated Docker image and take a look around? With a command like:

docker run -it --entrypoint /bin/bash gsc-icr.io/data-security-broker/dsb-shield-postgresql-v1:v1

When you enter the Bash session inside this GSC-generated image, you really don't see the /opt/ directory?

NandiniKJ commented 1 year ago

I tried to look inside the GSC Docker image, opt folder is there but its empty.

root@baremetal01-innovation-poc-sgx:~/gramine-poc/gsc-v2# docker run -it --entrypoint /bin/bash gsc-icr.io/data-security-broker/dsb-shield-postgresql-v1:v1 root@16ac5039e29b:/# ls bin boot dev etc gramine home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var root@16ac5039e29b:/# cd opt/ root@16ac5039e29b:/opt# ls root@16ac5039e29b:/opt#

dimakuv commented 1 year ago

And it's not empty in the original app Docker image? Are you sure? How is this possible...

NandiniKJ commented 1 year ago

Its not empty in the original Docker image. It has the start-baffle-shield.sh script in the defined path /opt/baffle/Release-Baffle.1.7.0.161/shield.

root@baremetal01-innovation-poc-sgx:~/gramine-poc/gsc-v2# docker exec -it 17e6f3692053 sh sh-4.4$ ls bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var sh-4.4$ cd opt/ sh-4.4$ ls baffle sslconfig sh-4.4$ cd baffle/ sh-4.4$ sh-4.4$ ls Release-Baffle.1.7.0.161 sh-4.4$ cd Release-Baffle.1.7.0.161/ sh-4.4$ ls migration shield sh-4.4$ cd shield/ sh-4.4$ ls BaffleCommonConfig BafflePrivacySchema baffle-shield-postgresql-final.jar bss_public.pem key_transfer log4j-baffleshield.properties log4j-slf4j-impl-2.17.2.jar start-baffle-shield.sh BaffleEntitySchema KmsConfig.properties bm-connector.jar credstore log4j-api-2.17.2.jar log4j-core-2.17.2.jar logs sh-4.4$

dimakuv commented 1 year ago

But you performed docker exec -it 17e6f3692053, i.e., you connected to an already-existing Docker container. Could it be that in this container, you or some script created the /opt/ contents?

What if you just do:

docker run -it --entrypoint /bin/bash icr.io/data-security-broker/dsb-shield-postgresql-v1:v1
NandiniKJ commented 1 year ago

I tried the above command which you gave, the existing image has the baffle folder inside opt.

root@baremetal01-innovation-poc-sgx:~/gramine-poc/gsc-v2# docker run -it --entrypoint /bin/bash icr.io/data-security-broker/dsb-shield-postgresql:v1 [baffle@b0c3229b9f8a /]$ ls bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var [baffle@b0c3229b9f8a /]$ cd opt/ [baffle@b0c3229b9f8a opt]$ ls baffle sslconfig [baffle@b0c3229b9f8a opt]$ cd baffle [baffle@b0c3229b9f8a baffle]$ ls Release-Baffle.1.7.0.161 [baffle@b0c3229b9f8a baffle]$ cd Release-Baffle.1.7.0.161/ [baffle@b0c3229b9f8a Release-Baffle.1.7.0.161]$ cd shield/ [baffle@b0c3229b9f8a shield]$ ls baffle-shield-postgresql-final.jar bss_public.pem log4j-api-2.17.2.jar log4j-core-2.17.2.jar start-baffle-shield.sh bm-connector.jar key_transfer log4j-baffleshield.properties log4j-slf4j-impl-2.17.2.jar [baffle@b0c3229b9f8a shield]$

dimakuv commented 1 year ago

This makes no sense to me...

Could you:

  1. Remove all Docker images, containers, etc. -- basically prune all Docker artifacts
  2. Remove the build/ directory in GSC -- prune all GSC-temporary data
  3. Re-run gsc build and gsc sign-image commands and attach the produced logs

In other words, re-do the whole GSC process from an absolutely clean state, and show us the logs.

NandiniKJ commented 1 year ago

I deleted all the Docker images and pruned it. Git cloned - https://github.com/gramineproject/gsc.git Then ran the following commands:

cd gsc cp config.yaml.template config.yaml openssl genrsa -3 -out enclave-key.pem 3072 ./gsc build --insecure-args icr.io/data-security-broker/dsb-shield-postgresql:v1 test/generic.manifest

This time the build failed in Step 9. Please find the build output below.

`Step 9/29 : RUN cd /gramine && meson setup build/ --prefix="/gramine/meson_build_output" --buildtype=release -Ddirect=enabled -Dsgx=enabled -Ddcap=enabled -Dsgx_driver=upstream -Dsgx_driver_include_path=/gramine/driver && ninja -C build && ninja -C build install

---> Running in 319df51f93e9 The Meson build system Version: 1.2.0 Source dir: /gramine Build dir: /gramine/build Build type: native build Project name: gramine Project version: 1.5post~UNRELEASED C compiler for the host machine: cc (gcc 9.4.0 "cc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0") C linker for the host machine: cc ld.bfd 2.34 C++ compiler for the host machine: c++ (gcc 9.4.0 "c++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0") C++ linker for the host machine: c++ ld.bfd 2.34 Host machine cpu family: x86_64 Host machine cpu: x86_64 Program check-no-reloc.sh found: YES (/gramine/scripts/check-no-reloc.sh) Program gen-pal-map.py found: YES (/gramine/scripts/gen-pal-map.py) Program get-python-platlib.py found: YES (/gramine/scripts/get-python-platlib.py) Program meson-clang-format.sh found: YES (/gramine/scripts/meson-clang-format.sh) Fetching value of define "GLIBC" : 2 Program objcopy found: YES (/usr/bin/objcopy) Program python3 found: YES (/usr/bin/python3) WARNING: You should add the boolean check kwarg to the run_command call. ....... cc common.o onefile.o fuzz_dtlsserver.o ../../tests/src/helpers.o ../../tests/src/bignum_helpers.o ../../tests/src/asn1_helpers.o ../../tests/src/psa_crypto_helpers.o ../../tests/src/psa_exercise_key.o ../../tests/src/threading_helpers.o ../../tests/src/random.o ../../tests/src/fake_external_rng_for_test.o ../../tests/src/certs.o ../../tests/src/drivers/test_driver_aead.o ../../tests/src/drivers/test_driver_asymmetric_encryption.o ../../tests/src/drivers/test_driver_pake.o ../../tests/src/drivers/test_driver_key_agreement.o ../../tests/src/drivers/test_driver_signature.o ../../tests/src/drivers/test_driver_key_management.o ../../tests/src/drivers/test_driver_cipher.o ../../tests/src/drivers/hash.o ../../tests/src/drivers/test_driver_mac.o ../../tests/src/drivers/platform_builtin_keys.o -L../../library -lmbedtls -lmbedx509 -lmbedcrypto -o fuzz_dtlsserver make[2]: Leaving directory '/gramine/build/subprojects/mbedtls-mbedtls-3.4.0/libmbedcrypto.a.p/programs/fuzz' make[1]: Leaving directory '/gramine/build/subprojects/mbedtls-mbedtls-3.4.0/libmbedcrypto.a.p/programs' make: Leaving directory '/gramine/build/subprojects/mbedtls-mbedtls-3.4.0/libmbedcrypto.a.p'

aneessahib commented 1 year ago

What you are now seeing is another new issue.

Please change this line to the below, and retry.

&& /usr/bin/python3 -B -m pip install 'tomli>=1.1.0' 'tomli-w>=0.4.0' 'meson==1.1.1'

NandiniKJ commented 1 year ago

I tried the above fix and the ninja issue is resolved, but got into another error.

`Step 12/29 : RUN apt-get update && env DEBIAN_FRONTEND=noninteractive apt-get install -y binutils expect libprotobuf-c-dev locales openssl python3 python3-cryptography python3-protobuf python3-pyelftools \python3-pip && /usr/bin/python3 -B -m pip install click jinja2 protobuf 'tomli>=1.1.0' 'tomli-w>=0.4.0' && apt-get remove -y python3-pip && apt-get autoremove -y && rm -rf /var/lib/apt/lists/*

---> Running in 030aed51b7a5 /bin/sh: apt-get: command not found

Failed to build unsigned graminized Docker image gsc-icr.io/data-security-broker/dsb-shield-postgresql:v1-unsigned.`

aneessahib commented 1 year ago

What's the distro of your base image?

NandiniKJ commented 1 year ago

Please find the details below:

DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS" PRETTY_NAME="Ubuntu 22.04.2 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.2 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL=https://www.ubuntu.com/ SUPPORT_URL=https://help.ubuntu.com/ BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/ PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy UBUNTU_CODENAME=jammy

Gramine does not yet support Ubuntu 22.04, though the support will be merged shortly, probably next week. In the meantime, you can try this PR…

https://github.com/gramineproject/gsc/pull/155

aneessahib commented 1 year ago

So are you already using the mentioned PR? #155 ?

NandiniKJ commented 1 year ago

Yes, even after using this PR I get the below error.

`Step 12/29 : RUN apt-get update && env DEBIAN_FRONTEND=noninteractive apt-get install -y binutils expect libprotobuf-c-dev locales openssl python3 python3-cryptography python3-protobuf python3-pyelftools \python3-pip && /usr/bin/python3 -B -m pip install click jinja2 protobuf 'tomli>=1.1.0' 'tomli-w>=0.4.0' && apt-get remove -y python3-pip && apt-get autoremove -y && rm -rf /var/lib/apt/lists/*

---> Running in 030aed51b7a5 /bin/sh: apt-get: command not found

Failed to build unsigned graminized Docker image gsc-icr.io/data-security-broker/dsb-shield-postgresql:v1-unsigned.`

aneessahib commented 1 year ago

please paste the contents of your config.yaml file here

NandiniKJ commented 1 year ago

Please find the contents below.

# Specify the OS distro that is used to build Gramine, i.e., the distro from where the Gramine build
# gets all tools and dependencies from. This distro should match the distro underlying the
# application's Docker image; otherwise the results may be unpredictable.
#
# Currently supported distros are:
# - ubuntu:20.04, ubuntu:21.04, ubuntu:22.04
# - debian:10, debian:11, debian:12
# - centos:8
Distro: "ubuntu:22.04"

# If the image has a specific registry, define it here.
# Empty by default; example value: "registry.access.redhat.com/ubi8".
Registry: ""

# If you're using your own fork and branch of Gramine, specify the GitHub link and the branch name
# below; typically, you want to keep the default values though.
#
# It is also possible to specify the prebuilt Gramine Docker image (that was built previously via
# the `gsc build-gramine` command). For this, remove Repository and Branch and instead write:
#   Image:      "<prebuilt Gramine Docker image>"
#
# GSC releases are guaranteed to work with corresponding Gramine releases (and GSC `master`
# branch is guaranteed to work with current Gramine `master` branch).
Gramine:
    Repository: "https://github.com/gramineproject/gramine.git"
    Branch:     "master"

# Specify the Intel SGX driver installed on your machine (more specifically, on the machine where
# the graminized Docker container will run); there are several variants of the SGX driver:
#
#   - upstream (in-kernel) driver: use empty values like below
#         Repository: ""
#         Branch:     ""
#
#   - DCAP out-of-tree driver: same as above, use empty values
#         Repository: ""
#         Branch:     ""
#
#   - legacy out-of-tree driver: use something like the below values, but adjust the branch name
#         Repository: "https://github.com/01org/linux-sgx-driver.git"
#         Branch:     "sgx_driver_1.9"
#
SGXDriver:
    Repository: ""
    Branch:     ""
aneessahib commented 1 year ago

Please take out all the redundant comments. What's the Python version of your base image, and GSC image

NandiniKJ commented 1 year ago

My base image doesn't have Python installed on it. I have attached the Distro of base image and GSC image.

Base image: `root@baremetal01-innovation-poc-sgx:~/gramine/gsc# docker run -it --entrypoint /bin/bash icr.io/data-security-broker/dsb-shield-postgresql:v1 [baffle@9943670ad863 /]$ cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="8.8 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.8" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.8 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.8 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.8" [baffle@9943670ad863 /]$ `

GSC image:

root@baremetal01-innovation-poc-sgx:~/gramine/gsc# docker run -it --entrypoint /bin/bash gsc-icr.io/data-security-broker/dsb-shield-postgresql:v1 root@0c8fde87c67b:/# cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.2 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.2 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy

aneessahib commented 1 year ago

Ok so that's the issue. Your base image distro is RHEL 8(and not Ubuntu 22.04 as mentioned earlier). The GSC supported distros are called out in the config.yaml file. For now, you can try setting the distro in the config.yaml to centos:8 and retry.

NandiniKJ commented 1 year ago

I tried to change the distro to centos:8 in config.yaml file and then build it using ./gsc build --insecure-args icr.io/data-security-broker/dsb-shield-postgresql:v1 test/generic.manifest command. Got the following error.

` ---> 4a357a2376f9 Step 10/24 : RUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-Linux- && sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Linux- && sed -i 's/enabled=0/enabled=1/g' /etc/yum.repos.d/CentOS-Linux-PowerTools.repo

---> Running in 776215230c73 sed: can't read /etc/yum.repos.d/CentOS-Linux-*: No such file or directory

Failed to build unsigned graminized Docker image gsc-icr.io/data-security-broker/dsb-shield-postgresql:v1-unsigned.`

aneessahib commented 1 year ago

GSC does not support RHEL at this point. But Gramine packages for RHEL are available, so you could consider building an image by installing RHEL packages. Another option is to move to one of the supported distros, does that work for you? If no other options, then we will have to look at taking this as a feature request. @dimakuv - any other ideas?

NandiniKJ commented 1 year ago

Can you explain how to build an image by installing RHEL packages. Will this option run inside GSC or Gramine directly.

Moving to other distros is not an option because this is a third party application.

aneessahib commented 1 year ago

We will revert with a simple script to run a helloworld program in a RHEL container with Gramine.

NandiniKJ commented 1 year ago

Ok thanks.

NandiniKJ commented 1 year ago

Just wanted to check - my host OS is ubuntu and the application base image is RHEL. Will this work wrt GSC ? If not then which image is the issue here and needs to be changed to what.

aneessahib commented 1 year ago
  1. Test GSC branch that has support for RHEL https://github.com/sahason/gsc/tree/sahason/gsc-rhel8-support
  2. Instructions to run a sample RHEL hello world application in Gramine https://github.com/sreeharikax/contrib/tree/rhel-changes/Examples/rhel-helloworld You can use either of the above
NandiniKJ commented 1 year ago

Hi @aneessahib, I tried the first approach and getting the below error. My base image is RHEL8.8 and Distro is Ubuntu 22.04.

` ---> 2cfddb5ede1e Step 10/26 : USER root

---> Using cache ---> 4a357a2376f9 Step 11/26 : RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm && dnf update -y

---> Running in 22e1c0046704 /bin/sh: dnf: command not found

Failed to build unsigned graminized Docker image gsc-icr.io/data-security-broker/dsb-shield-postgresql:v1-unsigned.`

aneessahib commented 1 year ago

Use Distro: "ubi:8.8" in config.yaml

NandiniKJ commented 1 year ago

I am using Distro: "ubi:8.8" in config.yaml file. I have taken the latest change of https://github.com/sahason/gsc/tree/sahason/gsc-rhel8-support

root@baremetal01-innovation-poc-sgx:~/gramine/gsc-rhel/gsc# ./gsc build --insecure-args icr.io/data-security-broker/dsb-shield-postgresql:v1 test/generic.manifest Building unsigned graminized Docker imagegsc-icr.io/data-security-broker/dsb-shield-postgresql:v1-unsignedfrom original application imageicr.io/data-security-broker/dsb-shield-postgresql:v1`... Step 1/26 : FROM registry.access.redhat.com/ubi8/ubi:8.8 AS gramine

---> e8e5725e8af3 Step 2/26 : RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm && dnf update -y

---> Using cache ---> a817ae11cce1 Step 3/26 : RUN rpm --import https://www.centos.org/keys/RPM-GPG-KEY-CentOS-Official && dnf config-manager --disableplugin subscription-manager --add-repo http://vault.centos.org/centos/8/BaseOS/x86_64/os && dnf config-manager --disableplugin subscription-manager --add-repo http://vault.centos.org/centos/8/AppStream/x86_64/os && dnf config-manager --disableplugin subscription-manager --add-repo http://vault.centos.org/centos/8/PowerTools/x86_64/os

---> Using cache ---> f8c346a24940 Step 4/26 : RUN dnf update -y && dnf install -y autoconf bison curl elfutils-libelf-devel epel-release flex gawk gcc-c++ git httpd libevent-devel make nasm ncurses-devel ninja-build openssl-devel patch pkg-config protobuf-c-compiler protobuf-c-devel protobuf-compiler protobuf-devel python3 python3-cryptography python3-pip python3-protobuf rpm-build wget && /usr/bin/python3 -B -m pip install 'tomli>=1.1.0' 'tomli-w>=0.4.0' 'meson>=0.56,!=1.2.*'

---> Using cache ---> a9bade71345b Step 5/26 : RUN git clone https://github.com/gramineproject/gramine.git /gramine

---> Using cache ---> b6d3f52030ac Step 6/26 : RUN cd /gramine && git fetch origin master && git checkout master

---> Using cache ---> 1d9808687f43 Step 7/26 : RUN mkdir -p /gramine/driver/asm && cd /gramine/driver/asm && wget --timeout=10 -O sgx.h https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/plain/arch/x86/include/uapi/asm/sgx.h?h=v5.11 && sha256sum sgx.h | grep -q a34a997ade42b61376b1c5d3d50f839fd28f2253fa047cb9c0e68a1b00477956

---> Using cache ---> 8b7ab995a091 Step 8/26 : RUN cd /gramine && meson setup build/ --prefix="/gramine/meson_build_output" --buildtype=release -Ddirect=enabled -Dsgx=enabled -Dsgx_driver=upstream -Dsgx_driver_include_path=/gramine/driver && ninja -C build && ninja -C build install

---> Using cache ---> a798406dd3d9 Step 9/26 : FROM icr.io/data-security-broker/dsb-shield-postgresql:v1

---> 2cfddb5ede1e Step 10/26 : USER root

---> Using cache ---> 4a357a2376f9 Step 11/26 : RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm && dnf update -y

---> Running in 21ebfabb5a62 /bin/sh: dnf: command not found

Failed to build unsigned graminized Docker image gsc-icr.io/data-security-broker/dsb-shield-postgresql:v1-unsigned. `

aneessahib commented 1 year ago

@sahason - pls take a look at this issue.

sahason commented 1 year ago

@NandiniKJ I have made some change. Could you please try with latest change?

NandiniKJ commented 1 year ago

Hi @sahason, I took the latest pull and got the below error.

` ---> Using cache ---> 4a357a2376f9 Step 11/27 : RUN yum install -y dnf

---> Running in 4d3900bced71 /bin/sh: yum: command not found`

sahason commented 1 year ago

@NandiniKJ What is the package manager of the rhel8 distro of your base image?

NandiniKJ commented 1 year ago

This is the base image details:

`root@baremetal01-innovation-poc-sgx:~/gramine/gsc# docker run -it --entrypoint /bin/bash icr.io/data-security-broker/dsb-shield-postgresql:v1 [baffle@9943670ad863 /]$ cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="8.8 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.8" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.8 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.8 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.8" [baffle@9943670ad863 /]$ `

aneessahib commented 1 year ago

@NandiniKJ - please try the second option as well (which is to install Gramine to your base image).

NandiniKJ commented 1 year ago

I tried the second option too. Got the below error:

root@baremetal01-innovation-poc-sgx:~/gramine/gsc-rhel/gsc# docker run -it --device /dev/sgx_enclave --env GSC_PAL=Linux --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --entrypoint /bin/bash icr.io/data-security-broker/dsb-shield-postgresql:v1 [baffle@11f344d7d6b3 /]$ dnf update -y bash: dnf: command not found [baffle@11f344d7d6b3 /]$

NandiniKJ commented 1 year ago

The package manager in the current image is microdnf.

aneessahib commented 1 year ago

@NandiniKJ - can you confirm what is the base RHEL distro that your base image uses? This information should be in your dockerfile. look for the FROM keyword

NandiniKJ commented 1 year ago

@aneessahib Base image uses UBI8.

aneessahib commented 1 year ago

Then it should have worked. @sahason had tested the GSC branch we gave you with https://hub.docker.com/r/redhat/ubi8 as well as registry.access.redhat.com/ubi8/ubi:8.8 . yum is the package manager in both.

NandiniKJ commented 1 year ago

In my base image microdnf is the package manager.

sahason commented 1 year ago

@NandiniKJ Could you please send the string following the FROM commands from your dockerfile? Is the base image based ubi8-minimal? As microdnf is the package manager of ubi8-minimal.

NandiniKJ commented 1 year ago

@aneessahib I do not have the Dockerfile as it is third party application. I have asked them to send the details. They must be using ubi8-minimal.

NandiniKJ commented 1 year ago

@aneessahib Please find the Dockerfile details.

image (5)

sahason commented 1 year ago

@NandiniKJ I have made some change. Could you please try with latest change?