search

edgebitio / enclaver

Open source toolkit created to enable easy adoption of software enclaves
https://edgebit.io/enclaver
Apache License 2.0
138 stars 14 forks source link

enclaver cannot reach S3 from within enclave #197

Open pavankad opened 2 months ago

pavankad commented 2 months ago

I am trying to connect to S3 bucket to download file. KMS proxy works fine. But, it is not able to reach S3 bucket. As shown below in the logs, s3 is being proxied thru http://localhost:10000(I have not speficied this anywhere).

My use case is similar to the below github repo that you published. Any help is greatly appreciated.

https://github.com/edgebitio/no-fly-list/blob/main/server.py

I have following enclaver.yaml config

version: v1 name: "depa" target: "depa:enclave-latest" sources: app: "depa:latest" defaults: memory_mb: 2048 kms_proxy: listen_port: 9999 egress: allow:

eyakubovich commented 1 month ago

Your allow list for S3 is missing a region, I think:

- s3.amazonaws.com

I think the S3 addresses are of the form s3.REGION.amazonaws.com. Just change it to s3.*.amazonaws.com

robszumski commented 1 month ago

Whoops didn't intend to close this issue, but let us know if that fixes your issue.

pavankad commented 1 month ago

thanks for responding. Changing S3 address to "s3.*.amazonaws.com" did not help. I get the same error as before. Is it trying to connect to S3 proxy listening on 10000 on host ec2 instance?

[ec2-user@ip-172-31-16-159 aws_depa_training]$ sudo enclaver run --publish 8000:8001 depa:enclave-latest INFO enclaver::run > starting egress proxy on vsock port 17002 INFO enclaver::vsock > Listening on vsock port 17002 INFO enclaver::run > starting enclave INFO enclaver::run > started enclave i-03731652486c31676-enc192478cd0b21d3f INFO enclaver::run > starting ingress proxy on port 8001 INFO enclaver::run > waiting for enclave to boot to stream logs INFO enclaver::run > connected to enclave, starting log stream INFO enclave > INFO enclaver::vsock > Listening on vsock port 17001 INFO enclave > INFO odyn::enclave > Bringing up loopback interface INFO enclave > INFO odyn::enclave > Seeding /dev/random with entropy from nsm device INFO enclave > INFO odyn > Enclave initialized INFO enclave > INFO odyn::egress > Starting egress INFO enclave > INFO odyn::ingress > Starting TCP ingress on port 8001 INFO enclave > INFO enclaver::vsock > Listening on vsock port 8001 INFO enclave > INFO odyn::kms_proxy > Starting KMS proxy INFO enclave > INFO odyn::kms_proxy > Generating public/private keypair INFO enclave > INFO odyn::kms_proxy > Fetching credentials from IMDSv2 INFO enclave > INFO odyn::kms_proxy > Credentials fetched INFO enclave > INFO odyn > Starting ["python3", "-m", "flask", "run", "--host=0.0.0.0", "--port=8001"] INFO enclave > Serving Flask app '/app/server.py' INFO enclave > Debug mode: off INFO enclave > WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. INFO enclave > Running on all addresses (0.0.0.0) INFO enclave > Running on http://127.0.0.1:8001 INFO enclave > * Running on http://127.0.0.1:8001 INFO enclave > Press CTRL+C to quit

Here are the open ports, I don't find 10000, or even vsock ports 17002

[ec2-user@ip-172-31-16-159 aws_depa_training]$ sudo lsof -nP -iTCP -sTCP:LISTEN COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rpcbind 2160 rpc 8u IPv4 16237 0t0 TCP :111 (LISTEN) rpcbind 2160 rpc 11u IPv6 16240 0t0 TCP :111 (LISTEN) master 2621 root 13u IPv4 15135 0t0 TCP 127.0.0.1:25 (LISTEN) sshd 2808 root 3u IPv4 19769 0t0 TCP :22 (LISTEN) sshd 2808 root 4u IPv6 19771 0t0 TCP :22 (LISTEN) container 9479 root 11u IPv4 2191916 0t0 TCP 127.0.0.1:40533 (LISTEN)

eyakubovich commented 1 month ago

port 10000 is a TCP port inside the enclave. It gets advertised as http_proxy env variable. Your first log excerpt contained "401 Unauthorized" but this one does not. Are you still seeing it? I'm not seeing any errors in the last log you posted.

There is a vsock listening on the host but it's inside of a container (enclaver run runs a docker container) so it's hard see with regular tooling.

pavankad commented 1 month ago

thanks for those answers. Sorry I missed the logs, Following is the log. It is the same error

[ec2-user@ip-172-31-16-159 aws_depa_training]$ sudo enclaver run --publish 8000:8001 depa:enclave-latest INFO enclaver::run > starting egress proxy on vsock port 17002 INFO enclaver::vsock > Listening on vsock port 17002 INFO enclaver::run > starting enclave INFO enclaver::run > started enclave i-03731652486c31676-enc192478cd0b21d3f INFO enclaver::run > starting ingress proxy on port 8001 INFO enclaver::run > waiting for enclave to boot to stream logs INFO enclaver::run > connected to enclave, starting log stream INFO enclave > INFO enclaver::vsock > Listening on vsock port 17001 INFO enclave > INFO odyn::enclave > Bringing up loopback interface INFO enclave > INFO odyn::enclave > Seeding /dev/random with entropy from nsm device INFO enclave > INFO odyn > Enclave initialized INFO enclave > INFO odyn::egress > Starting egress INFO enclave > INFO odyn::ingress > Starting TCP ingress on port 8001 INFO enclave > INFO enclaver::vsock > Listening on vsock port 8001 INFO enclave > INFO odyn::kms_proxy > Starting KMS proxy INFO enclave > INFO odyn::kms_proxy > Generating public/private keypair INFO enclave > INFO odyn::kms_proxy > Fetching credentials from IMDSv2 INFO enclave > INFO odyn::kms_proxy > Credentials fetched INFO enclave > INFO odyn > Starting ["python3", "-m", "flask", "run", "--host=0.0.0.0", "--port=8001"] INFO enclave > Serving Flask app '/app/server.py' INFO enclave > Debug mode: off INFO enclave > WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. INFO enclave > Running on all addresses (0.0.0.0) INFO enclave > Running on http://127.0.0.1:8001 INFO enclave > * Running on http://127.0.0.1:8001 INFO enclave > Press CTRL+C to quit

INFO enclave > Starting server... INFO enclave > Here are params received>>>>>>>> INFO enclave > Instance Profil name KMS_instance INFO enclave > us-east-1

INFO enclave > Could not read file from S3 INFO enclave > Failed to connect to proxy URL: "http://127.0.0.1:10000/" INFO enclave > [2024-10-01 10:08:10,050] ERROR in app: Exception on /config [POST] INFO enclave > Traceback (most recent call last): INFO enclave > File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 712, in urlopen INFO enclave > self._prepare_proxy(conn) INFO enclave > File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1014, in _prepare_proxy INFO enclave > conn.connect() INFO enclave > File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 374, in connect INFO enclave > self._tunnel() INFO enclave > File "/usr/local/lib/python3.8/http/client.py", line 905, in _tunnel INFO enclave > raise OSError("Tunnel connection failed: %d %s" % (code, INFO enclave > OSError: Tunnel connection failed: 401 Unauthorized INFO enclave > INFO enclave > During handling of the above exception, another exception occurred: INFO enclave > INFO enclave > Traceback (most recent call last): INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/httpsession.py", line 464, in send INFO enclave > urllib_response = conn.urlopen( INFO enclave > File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 801, in urlopen INFO enclave > retries = retries.increment( INFO enclave > File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 527, in increment INFO enclave > raise six.reraise(type(error), error, _stacktrace) INFO enclave > File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise INFO enclave > raise value.with_traceback(tb) INFO enclave > File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 712, in urlopen INFO enclave > self._prepare_proxy(conn) INFO enclave > File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1014, in _prepare_proxy INFO enclave > conn.connect() INFO enclave > File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 374, in connect INFO enclave > self._tunnel() INFO enclave > File "/usr/local/lib/python3.8/http/client.py", line 905, in _tunnel INFO enclave > raise OSError("Tunnel connection failed: %d %s" % (code, INFO enclave > urllib3.exceptions.ProxyError: ('Cannot connect to proxy.', OSError('Tunnel connection failed: 401 Unauthorized')) INFO enclave > INFO enclave > During handling of the above exception, another exception occurred: INFO enclave > INFO enclave > Traceback (most recent call last): INFO enclave > File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2529, in wsgi_app INFO enclave > response = self.full_dispatch_request() INFO enclave > File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1825, in full_dispatch_request INFO enclave > rv = self.handle_user_exception(e) INFO enclave > File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1823, in full_dispatch_request INFO enclave > rv = self.dispatch_request() INFO enclave > File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1799, in dispatch_request INFO enclave > return self.ensure_sync(self.view_functions[rule.endpoint])(view_args) INFO enclave > File "/app/server.py", line 186, in get_config INFO enclave > encrypted_data_from_s3 = read_file_from_s3(s3_client, s3_bucket, filename) INFO enclave > File "/app/server.py", line 142, in read_file_from_s3 INFO enclave > response = s3_client.get_object(Bucket=bucket_name, Key=file_key) INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 553, in _api_call INFO enclave > return self._make_api_call(operation_name, kwargs) INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 989, in _make_api_call INFO enclave > http, parsed_response = self._make_request( INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 1015, in _make_request INFO enclave > return self._endpoint.make_request(operation_model, request_dict) INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 119, in make_request INFO enclave > return self._send_request(request_dict, operation_model) INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 202, in _send_request INFO enclave > while self._needs_retry( INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 354, in _needs_retry INFO enclave > responses = self._event_emitter.emit( INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 412, in emit INFO enclave > return self._emitter.emit(aliased_event_name, kwargs) INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 256, in emit INFO enclave > return self._emit(event_name, kwargs) INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 239, in _emit INFO enclave > response = handler(kwargs) INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 207, in call INFO enclave > if self._checker(checker_kwargs): INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 284, in call INFO enclave > should_retry = self._should_retry( INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 320, in _should_retry INFO enclave > return self._checker(attempt_number, response, caught_exception) INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 363, in call INFO enclave > checker_response = checker( INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 247, in call INFO enclave > return self._check_caught_exception( INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception INFO enclave > raise caught_exception INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 281, in _do_get_response INFO enclave > http_response = self._send(request) INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 377, in _send INFO enclave > return self.http_session.send(request) INFO enclave > File "/usr/local/lib/python3.8/site-packages/botocore/httpsession.py", line 495, in send INFO enclave > raise ProxyConnectionError( INFO enclave > botocore.exceptions.ProxyConnectionError: Failed to connect to proxy URL: "http://127.0.0.1:10000/" INFO enclave > 127.0.0.1 - - [01/Oct/2024 10:08:10] "POST /config HTTP/1.1" 500 -

pavankad commented 1 month ago

Following is the code to access S3 bucket. S3 bucket has public access, Also code works running in Docker. It is simple code, 1. get data key from KMS, 2.read pre-encrypted file in S3 bucket, 3. decrypt the file using data key fetched from KMS and do processing on the file.

I have tested the code accessing google website, that worked too, which means egress proxy works. But, not sure why it is not working for S3.

    try:
        # Produce attestion and get data key from KMS 
        kms_client = get_kms_client(region, credentials['access_key_id'], credentials['secret_access_key'], credentials['token'])
        response = kms_client.decrypt(KeyId=KMS_KEY1_ID, CiphertextBlob=base64.b64decode(data_key))
        plain_text = base64.b64encode(response['Plaintext'])
        response = {'key':plain_text}
        print(response)
    except Exception as err:
        print("Could not decrypt key")
        raise

    # Step 2: Read the encrypted file from S3
    try:
        print(region)
        #s3_client = get_s3_client(region, credentials['access_key_id'], credentials['secret_access_key'], credentials['token'])
        #encrypted_data_from_s3 = read_file_from_s3(s3_client, s3_bucket, filename)
        s3 = boto3.resource('s3')
        obj = s3.Object(s3_bucket, filename)
        encrypted_data_from_s3 = obj.get()['Body'].read()
        #url = 'http://www.google.com'
        #resp = requests.get(url)
        #print(resp.text) # Printing response

    except Exception as err:
        print("Could not read file from S3")
        print(str(err))
        raise

    # Step 3: Decrypt the file data
    try:
        decrypted_data = decrypt_data(plain_text, encrypted_data_from_s3)
        print(decrypted_data)
    except Exception as err:
        print("could not decrypt S3 data")
        raise

Here is the enclaver.yaml

version: v1 name: "depa" target: "depa:enclave-latest" sources: app: "depa:latest" defaults: memory_mb: 2048 kms_proxy: listen_port: 9999 egress: allow:

BTW, under egress:allow: S3 has "s3.*.amazonaws.com", not showing above.

Another question is INFO enclave > raise OSError("Tunnel connection failed: %d %s" % (code, INFO enclave > OSError: Tunnel connection failed: 401 Unauthorized

what does this mean, is it tunnel connection b/w enclave and host?

robszumski commented 1 month ago

Still debugging on our side, but try with '**.amazonaws.com' which is a little relaxed, but should get you a demo that works.

egress:
  allow:
    - kms.*.amazonaws.com
    - '**.amazonaws.com'
    - 169.254.169.254
pavankad commented 1 month ago

it works with the above change, thanks so much @robszumski . For now, I can get the data coming in and going out. I will wait for your update to enable this specific to S3.