GSC: Unable to deploy Emojivoto sample app using graphene GSC

prateeksahu commented 3 years ago

Description of the problem

Unable to deploy simple microservice application (emojivoto - https://github.com/BuoyantIO/emojivoto/) using graphene GSC.

Steps to reproduce

System: Azure VM DC8_v2 (8 cpu, 32G mem) Kernel: 5.11.16

Run GSC tool on specific container images - emojivoto-web, emojivoto-emoji-svc, emojivoto-voting-svc config.yaml

Distro: "ubuntu18.04"
Graphene:
    Repository: "https://github.com/oscarlab/graphene.git"
    Branch:     "2e737e69f076c60918f87d6829bb769925e75fec" # last working commit
SGXDriver:
    Repository: ""
    Branch:     ""

manifest (named image.manifest)

sgx.enclave_size = "2048M"
sgx.thread_num = 16

Steps to graphenize the images: docker pull docker.l5d.io/buoyantio/emojivoto-emoji-svc:v11 docker tag docker.l5d.io/buoyantio/emojivoto-emoji-svc:v11 emojivoto-emoji-svc:latest ./gsc build --insecure-args -c config.yaml emojivoto-emoji-svc image.manifest ./gsc sign-image emojivoto-emoji-svc ~/graphene/Pal/src/host/Linux-SGX/signer/enclave-key.pem docker tag gsc-emojivoto-emoji-svc:latest gsc-emojivoto-emoji-svc:v1

above steps were executed for all three images (voting, emoji and web)

kubernetes cluster - minikube (driver=none) deploy the application using kubectl kubectl apply -k kustomize/deployment

The yaml files under development/ were updated to use the gsc- containers built above.

Expected results

The application should run as expected

Actual results

The pods crash repeatedly with environment variable errors. Eg -

ccVM@ccVM ~/emojivoto_linkerd (main)> kubectl logs web-64d4995c4-2vs5t -n emojivoto
+ '[' -z '' ']'
+ GSC_PAL=Linux-SGX
+ /graphene/python/graphene-sgx-get-token -output /entrypoint.token -sig /entrypoint.sig
Attributes:
    mr_enclave:  9517bfccaa1ba1e8f98b384c7b7b8a04f12d25e4b5bf046c64d569f2e5862a1e
    mr_signer:   0297a497500d4dd6d5195bad08a2a3a5955c50a7844d2d7279bc2f9d7b5e7a79
    isv_prod_id: 0
    isv_svn:     0
    attr.flags:  0600000000000000
    attr.xfrm:   1f00000000000000
    misc_select: 00000000
    misc_mask:   00000000
    modulus:     e33560469e9ffd020042ea0870ea0de6...
    exponent:    3
    signature:   d4f1d21d121d46ae4e1649dc580b5510...
    date:        2021-07-22
+ /graphene/Runtime/pal-Linux-SGX /graphene/Runtime/libpal-Linux-SGX.so init /entrypoint -c emojivoto-web ''
error: Using insecure argv source. Graphene will continue application execution, but this configuration must not be used in production!
2021/08/02 07:03:09 WEB_PORT (currently []) EMOJISVC_HOST (currently []) and VOTINGSVC_HOST (currently []) INDEX_BUNDLE (currently []) environment variables must me set.

dimakuv commented 3 years ago

GSC currently hard-codes the environment variables based on the initial values in the original Docker image. See starting from here: https://github.com/oscarlab/graphene/blob/master/Tools/gsc/gsc.py#L93

To support your use-case, GSC needs a way to propagate environment-specified envvars into the Graphene SGX enclave (in your case, Kubernetes sets these envvars and propagates them to each Docker container). There is currently no such functionality in Graphene and in GSC, though we can add a switch to "insecurely propagate environment variables from the host" to GSC -- this would be a simply workaround but obviously insecure.

Alternative solution is to use Marblerun, which actually runs the "pre-main" logic to populate the envvars with correct values given by Kubernetes. See https://www.marblerun.sh/.

TLDR: You hit a legitimate limitation of GSC, and there is currently no solution other than using Marblerun... Thanks for reporting this, we'll need to do something about envvars in GSC.

prateeksahu commented 3 years ago

Thanks @dimakuv I have used marblerun as well, and I was actually trying to run graphene with the sample app that marblerun uses so that I can study the pros and cons of the two. Can you help me try the insecure way of env var propagation? Since this is a very initial research experiment and not a production level deployment, I might be okay with it.

dimakuv commented 3 years ago

Check this https://graphene.readthedocs.io/en/latest/manifest-syntax.html#environment-variables.

I guess it would be enough to add loader.insecure__use_host_env = true to this file: https://github.com/oscarlab/graphene/blob/848f9617acfec9fd719871a3adfd3e38ad94ea40/Tools/gsc/templates/entrypoint.manifest.template

This is of course only for a research experiment.

prateeksahu commented 3 years ago

Thanks @dimakuv for the quick support. This helped me navigate past the blocker. I am now able to run the application.

For future readers, the line to be added is loader.insecure__use_host_env = 1 and it can be added to your own manifest file (image.manifest in my case) or the entrypoint.manifest.template as suggested above. I decided to add it to my own so as to keep the original files clean.

gramineproject / graphene