[SGX] RFC: Support for Intel Project Amber Attestation

[bigdata-memory]

Intel Project Amber Attestation Service

Project Amber creates the industry’s first independent Trust Authority. The program is a SaaS (Software as a Service) implementation of a Trust Authority that provides remote verification of the trustworthiness of a compute asset based on attestation, policy, and reputation/risk data. Initially, Project Amber will verify the trustworthiness of trusted execution environments (TEEs), but the vision extends to much broader device verification, like IPUs, GPUs, platform roots of trust, and beyond. Project Amber provides REST APIs for users to manage their profile, attestation policies, and attestation of their workload. Intel is working towards standardization of the APIs and the data formats for the attestation reports (tokens), policies and evidence.

There are two fundamental use cases Project Amber supports

• "Basic" is the scenario in which the developer/customer just wants to verify that their application is running in a valid enclave. • "Key Release" is the scenario in which the developer/customer wants to decrypt sensitive data within a TEE.

Design Considerations: 1) Minimizing the burden on user side while not sacrificing usability and flexibility 2) Modularizing reusable functions and making them composable to build 3) Abstracting the security needs from low-level technical details 4) Generalizing the interface with variety of trust authorities

Project Amber provides client SDK for application developers to integrate with their application to communicate with Project Amber SaaS. However, for developers who use Gramine, we design a user-friendly interface that will be provided by Gramine without requiring application owner to change their code or just minor changes needed depend on particular usage scenario.

Proposed changes for Gramine Our proposal shifts all burden of communication with project Amber SaaS (with Project Amber’s appraise/ and keys/ endpoints) to the Gramine-SGX trusted runtime (tRT).

This means that user application just needs to do minor changes or even w/o any changes to their applications.

There are three modes from which our users can take most benefits out of this design.

Useful links: • Intel Blazes Security And Software Trails At Intel VisionIntel Blazes Security And Software Trails At Intel Vision - Forbes • Intel’s Project Amber provides security for confidential computing - AIExpress • Intel’s Greg Lavender On Project Amber, Securing The Digital Future, Quantum Computing And ‘Y2Q’ - Shane Snider/CRN • Intel Vision 2022: AI Gaudi2 Chip, Accenture Collaboration And More Products Unveiled, Shane Snider/CRN • Intel bets big on security as a service for confidential computing - Shweta Sharma/CSO Online • Intel’s Project Amber intros a service-based trust authority - Matt Hamblen/FierceElectronics • Intel Warns of Data Security Crisis Posed by Quantum Computers - HPCWire • The reinvention of Intel - Adam Shepherd/IT Pro • Intel reveals multiple new processors – and more - Stephan Withers/iTWire • Intel announces Project Amber, with the goal of ensuring independent trust - News7G • Intel Announces Project Amber, With The Goal Of Independent Trust Assurance - Andrew McCollum/NewsUpdate • Intel Announces "Project Amber" For Verifying The Trustworthiness Of Clouds - Michael Larabel/Phoronix • Intel's Project Amber Preserves Third-Party Attestation for Multi-Cloud - Nancy Liu/SDxCentral • Daily Roundup: Intel CEO Says Four 'Superpowers' Drive Global Problem-Solving - Julia King/SDXCentral • Intel announces confidential computing-as-a-service ‘Project Amber’ - Joe Uchill/SC Magazine • Intel’s ‘Project Amber’ offers remote verification of trustworthiness - Duncan Riley/SiliconANGLE • Intel Vision 2022 live: New Alder Lake CPUs, data center GPUs, HPC advances and more - Joel Khalili/TechRadar • Syndications: Wilson’s Media • New Intel service tackles a fundamental cybersecurity problem – Joel Khalili/TechRadar • Intel announces Project Amber, with the goal of independent trust assurance - Brian Stone/TechRepublic • Intel unveils new AI accelerators, partnerships at Intel Vision event, Staff/Telecompaper • Intel Poaches Open Source Execs from Netflix, Apple to Boost Linux Efforts, Agam Shah/The New Stack • How Intel and AMD hope to win the cloud security game - Dylan Martin/The Register
• Intel’s Project Amber provides security for confidential computing - Dean Takahashi/VentureBeat • Syndications: AI Express, BusinessTraverse, First FAQ • Intel announces new security as-a-service to deliver confidential computing - Tim Keary/VentureBeat • Project Amber: Intel apresenta segurança-como-serviço para garantir confiança em computação - tudoceluar • Intel VisiON: Intel maakt beveiligde communicatie tussen verschillende clouds en servers eenvoudiger - Frank Everaardt/HardwareInfo • Intel has announced Project Amber to ensure independent trust - Shubhransh/knownews

Sample of international coverage:
• Intel anuncia Proyecto Amber para computación confidencial y fiable - Diarioti
• Intel announces launch of ‘independent trust authority’ - Vasanth Srinivasan/The Hindu Business Line • Intel introduces Project Amber for cloud-to-edge and on-premises trust assurance - Rio Rancho Observer • Intel (INTC) Unveils Remote Verification Platform ‘Project Amber’ - Amit Chowdhry/Pulse2 • Intel Project Amber tanıtıldı - donanimhaber • Intel presenta Project Amber, per certificare la sicurezza dei dati cloud e on premise - Paolo Corsini/HardwareUpgrade • Intel Luncurkan Project Amber - Imam Suhartadi/Investor

[boryspoplawski]

Why cannot this be a simple library preloaded into the application (and somehow run, depending on the app runtime, e.g. in C it could juts define a global constructor to be run by libc runtime) that would use our /dev/attestation/* interfaces?

[bigdata-memory]

Why cannot this be a simple library preloaded into the application (and somehow run, depending on the app runtime, e.g. in C it could juts define a global constructor to be run by libc runtime) that would use our /dev/attestation/* interfaces?

It could be that way, however, it might make things complicated for users to deal with. Furthermore, as 3 usage scenarios above presented. Regarding the case 1, the way you described could be suitable for it but extra efforts needed for configuration and maintain. For case 2, the application would need a Project Amber client library to satisfy it (probably written in different programming languages) and user-level application code integration required. The last case would not be covered. In other words, the way we designed, can reduce the user-level error-prone code, attack surface, total maintenance burdens as well as be easily extended for other trust authorities. Therefore, we fully leverages the capability of Gramine-SGX to make things more secure and easier for end-users and application developers.

[boryspoplawski]

It could be that way, however, it might make things complicated for users to deal with.

How exactly?

Regarding the case 1, the way you described could be suitable for it but extra efforts needed for configuration and maintain.

No, this is exactly the same - you have something you called "premain", which could handle all of this, instead of embedding the code into Gramine.

For case 2, the application would need a Project Amber client library to satisfy it (probably written in different programming languages) and user-level application code integration required.

Since the application would already have to be "Project Amber" aware and would use the new "/dev/something" Gramine API, it could very well use a different API, i.e. either our /dev/attestation/* directly, or use a library that would wrap those in something suitable for "Project Amber", whatever that would be.

In other words, the way we designed, can reduce the user-level error-prone code

I would say it's the opposite - having some well defined API will be better for end users than special /dev files.

attack surface

???

total maintenance burdens

You just shift that into Gramine.

as well as be easily extended for other trust authorities.

To the contrary, extending Gramine will be harder than extending your lib using Gramine API.

[dimakuv]

Thanks @bigdata-memory for the detailed description and the links! I'll need to go through these links to understand the Amber design and APIs specifically (which I wanted to do anyway).

At a first glance, Project Amber's integration with Gramine looks very similar to the integrations that we've done in RA-TLS:

• with EPID (also a REST-based web service, Intel IAS)
• with DCAP (this one is different, the actual verification happens through a special Intel SGX DCAP shared library)
• with MAA (also a REST-based web service, MS Azure Attestation Provider, see https://github.com/gramineproject/gramine/pull/652 and https://github.com/gramineproject/gramine/issues/626)

All these RA-TLS attestation flows are implemented as shared libraries with a two-functions C API: https://github.com/gramineproject/gramine/blob/master/tools/sgx/ra-tls/ra_tls.h

Plus we have the Secret Provisioning library that hides the RA-TLS flows and exposes what you call "Premain" and "Dynamic" flows. Again, as a shared library with a C API: https://github.com/gramineproject/gramine/blob/master/tools/sgx/ra-tls/secret_prov.h

What @boryspoplawski seems to suggest is to implement integration with Project Amber in exactly the same way as we did in RA-TLS and Secret Prov to this date.

However, I also see two valid points in @bigdata-memory's comments:

1. For user experience, it is easier to expose API via /dev/ pseudo-files rather than a C API. Of course, all languages have adapters to interoperate with C libraries, but obviously reading/writing normal files is a much more "native and simple" API for e.g. Java or Python programmers.
2. The current two-function API of RA-TLS is not enough for advanced usages. Sometimes the user may legitimately want to look at e.g. the attestation collateral (e.g. CRLs). Or as in case with MAA and Amber, the user may want to get a copy of the JSON Web Token (JWT). This is currently completely hidden by the generic RA-TLS API.

So what I'm saying is: there are some pain points that we should probably resolve somehow. Gramine generally needs a better API for many different attestation schemes. We need to have a proper discussion on this topic.

[boryspoplawski]

What @boryspoplawski seems to suggest is to implement integration with Project Amber in exactly the same way as we did in RA-TLS and Secret Prov to this date.

Exactly.

1. For user experience, it is easier to expose API via /dev/ pseudo-files rather than a C API. Of course, all languages have adapters to interoperate with C libraries, but obviously reading/writing normal files is a much more "native and simple" API for e.g. Java or Python programmers.

I disagree. There could be language-specific libraries with different APIs, just like there would have been language specific libraries handling /dev files. No difference, if you want to have runtime/language specific stuff.

2. The current two-function API of RA-TLS is not enough for advanced usages. Sometimes the user may legitimately want to look at e.g. the attestation collateral (e.g. CRLs). Or as in case with MAA and Amber, the user may want to get a copy of the JSON Web Token (JWT). This is currently completely hidden by the generic RA-TLS API.

But why are you imposing this limitation that this needs to have the very same API as RA-TLS? It could have 100 functions, however complex you want. All of these goes into an "attestation backend" specific library.

So what I'm saying is: there are some pain points that we should probably resolve somehow. Gramine generally needs a better API for many different attestation schemes. We need to have a proper discussion on this topic.

Disagree. Gramine needs one API for exposing SGX specific stuff (like SGX quotas etc) and Gramine specific stuff (like encrypted files keys). Everything else should be written on top of it - like a glue connecting Gramine with external attestation service. I believe RA-TLS does exactly this atm.

[bigdata-memory]

The amber in the pathname could be replaced with a generic name if desired, and an additional setting item should be added to manifest.template for indicating which backend trust authority to talk.

[boryspoplawski]

The amber in the pathname could be replaced with a generic name if desired,

I don't think we need any new pathnames at all. @bigdata-memory could you respond to my previous comments?

[bigdata-memory]

It could be that way, however, it might make things complicated for users to deal with.

How exactly? As the way to pre-load a custom library is not frequently used in native running or development environment for most end users and developers. In addition, there might be some limitations imposed by user environments in which don't allow preload taking effect.

attack surface

??? In majority cases, the attack surface was introduced by user applications. if we can reduce the code size of them, then the attack surface can be reduced accordingly.

[boryspoplawski]

It could be that way, however, it might make things complicated for users to deal with.

How exactly?

As the way to pre-load a custom library is not frequently used in native running or development environment for most end users and developers. In addition, there might be some limitations imposed by user environments in which don't allow preload taking effect.

This makes no sense. How is using files from /dev/ any better? You must either make your app directly use them or preload some code. It doesn't make anything more complicated - it's completely orthogonal thing.

attack surface

???

In majority cases, the attack surface was introduced by user applications. if we can reduce the code size of them, then the attack surface can be reduced accordingly.

This doesn't make sense either. You just shift code from some library into Gramine - if anything you increase attack surface of Gramine, while keeping overall (Gramine+app) the same.

[monavij]

I still need to catch up fully on this thread, but I would say until we can achieve some level of standardization for attestation APIs in the industry, we will stick to preloaded libraries. That way the burden to maintain those will be on the attestation backends and not Gramine.

[dimakuv]

Looking at this again, I have some questions to @bigdata-memory:

1. It seems that the only difference between Case 1 ("Static Mode") and Case 3 ("Fix Mode") is: Case 1 runs the "premain" logic only once, before application startup, and there is no way to periodically renew the token. In Case 3, renewing the token is supposed to be done by Gramine periodically.

   1. What is the use case for renewing the token? What happens if the token is not renewed? What happens after the token is renewed?
   2. How does this renewal process happen? Is it the Project Amber service sending a request "hey, Gramine/app, renew the token"? Or is Gramine/app supposed to periodically ask the Project Amber service about the token?

2. In Case 1 ("Static Mode"), you say that "Cons: need application-specific premain". But is it really so? Can't the premain be sufficiently generic (just like you would write a generic code in Case 3).

[bigdata-memory]

Looking at this again, I have some questions to @bigdata-memory:

1. It seems that the only difference between Case 1 ("Static Mode") and Case 3 ("Fix Mode") is: Case 1 runs the "premain" logic only once, before application startup, and there is no way to periodically renew the token. In Case 3, renewing the token is supposed to be done by Gramine periodically.

Yes, you are right, and the Case 3 ("Fix mode") would be closer to TDX paradigm

1. What is the use case for renewing the token? What happens if the token is not renewed? What happens after the token is renewed?

Typically, token has an expiration to mitigate security risks. if the token get expired, it will be nullified and unacceptable for next transactions. if renewed, the refresh token will be valid one requesting permission to access confidential resources.

2. How does this renewal process happen? Is it the Project Amber service sending a request "hey, Gramine/app, renew the token"? Or is Gramine/app supposed to periodically ask the Project Amber service about the token?

The Gramine/app should check the expiration of token every time before using it. However, the client side can set the grace period ahead of deadline to minimize the impact on latency of transaction.

2. In Case 1 ("Static Mode"), you say that "Cons: need application-specific premain". But is it really so? Can't the premain be sufficiently generic (just like you would write a generic code in Case 3).

Yes, it is possible to be generic enough to cover majority of cases as Case 3 does, however, when users stray from the happy path, the Case 1 comes to rescue as it allows customing/extending the premain to special needs with ease.

[monavij]

As Dmitrii described all of the above can be done with premain or preloaded lib type approach, including the token renewel. There is no need to embed all this logic in Gramine.

[dimakuv]

@bigdata-memory Do we want to discuss this Amber proposal in one of the upcoming Gramine meetings (on Tuesdays)?

[bigdata-memory]

@dimakuv Thank you for this suggestion, however, I had a conflicting customer meeting that overlapped in the second half hour. So we can discuss it for a limited time, if you think it's okay. Thanks.

[monavij]

We discussed this in Gramine meeting today and discussion is captured in notes - https://github.com/gramineproject/gramine/discussions/1059. But the general consensus was that the task of re-fetching the token can be done in a worker thread in LD-PRELOADED library, as that library is fully aware of when the token expires and when it needs to be re-fetched. The attestation interfaces exposed by Gramine are generic enough to build any interfaces on top.

[mythi]

FWIW, I experimented with the idea a bit (without Amber itself still though) to see how the refresh part could be done.

I created an LD_PRELOAD hook that initializes a token and sets a "token expires" env variable. Users then now when a new token is needed and a getenv("AMBER_TOKEN") would return a new one.

The rest of the Amber params (URI, CA bits etc), can be set via loader.env. the same way was with secret provisioning libs.

$ /usr/local/bin/gramine-sgx busybox
Gramine is starting. Parsing TOML manifest file, this may take some time... 
-----------------------------------------------------------------------------------------------------------------------                                                                                            
Gramine detected the following insecure configurations: 

  - sgx.debug = true                           (this is a debug enclave)

Gramine will continue application execution, but this configuration must not be used in production!
----------------------------------------------------------------------------------------------------------------------- 

LD_PRELOAD constructor: AMBER_TOKEN env initialized to TOKEN1691409707954 from localhost:8080.
initial token: TOKEN1691409707954
sleep until AMBER_TOKEN_EXPIRATION
getenv("AMBER_TOKEN") to refresh: TOKEN1691409707954
sleep until AMBER_TOKEN_EXPIRATION
getenv("AMBER_TOKEN") to refresh: TOKEN1691409727954 
...

[bigdata-memory]

@mythi Thank you for the experiments. It is good to know that the getenv(...) can get updated environment variable values at runtime. Additionally, to keep the token refreshed periodically (5mins), we also need to keep the token refreshing thread alive in the preloaded library, in case that unmanaged thread failed for some reasons, the workload service has to be restarted. so it would be some other stable or reliability issues linked to this approach.

[mythi]

Additionally, to keep the token refreshed periodically (5mins), we also need to keep the token refreshing thread alive in the preloaded library,

@bigdata-memory How it is done in #1567 with the /dev/amber/renew flow? A quick look suggests to me it's just a blocking write that processes the update and then the app needs to read the token/secret pseudo file. Is that correct?

I think it would be good to take one step back and first define the user/app flows where Amber support is needed.

KBS (Amber token is used as a bearer token / POST data): the token is opaque to the app. KBS returns a wrapping key that Gramine can use to mount encrypted fs or it can be made available through SECRET_STRING or similar. One time operation throughout the app lifecycle so no token (nor secret/key) updates needed.

The RA-TLS+ MAA equivalent use-case also does not deal with Amber token inside Gramine. Gramine just passes SGX DCAP quotes and the relying part verifies the quote with Amber and gets a token back.

The app needs a token with some runtime/user data included. In my preload-test the setup always returned a fresh/valid token and apps could check from another env var when the token is going to be expired to know when to read. If this is not sufficient, we can always help to build more language specific wrappers (e.g., python modules) for the Amber functionality same way it's done with the Amber client Go module and TDX.

IMO, the best approach would be to just start enabling these use cases one by one on top of what vanilla Gramine offers today. It does not have to be all or nothing approach because then no progress is made.

[bigdata-memory]

@mythi We can of course offer various solutions with corresponding limitations to our customers. However, if we want to make our solution applicable to production environment, we must evaluate those limitations (security, stability, usability, scalability...) comprehensively ahead. IMHO, our customers may less likely adopt immature solutions but It is fine for PoC anyway. Thanks.