RFC: Support for Local Attestation (LA-TLS) between Enclaves

[vijaydhanraj]

Description of the problem

One of the cloud use cases is to modularize secure applications running in a system into different enclaves. This is to minimize the TCB and thereby reduce loss if some parts of the code are compromised. For example, one enclave may receive some personal user data and may interact with another enclave to service the user request. Due to the confidentiality nature of the data, both enclaves need to attest to each other before sharing data. But currently, Gramine supports local attestation only between a parent process and its forked child.

Creating this issue to identify potential pitfalls with this request and come up with a solid design to enable this feature.

[mkow]

This is to minimize the TCB and thereby reduce loss if some parts of the code are compromised. [...]

This whole paragraph seems to be based on incorrect assumptions - in Gramine all processes trust each other, if you compromise one of them then all of them are compromised, and this assumption is rather impossible in practice to remove (Gramine is +/- a distributed OS, making everything distrusting each other would make the design super hard, if not impossible).

Knowing parts of the original request, I think you wanted to actually write about local attestation between different Gramine instances, not inside one? If so, then that idea seemed insecure, because the connections between enclaves could be swapped (i.e. the specific "network" of enclaves is not attested), and had quite huge practical problems like breaking the cycles (both enclaves can't hardcode each other's hashes at the same time).

[DL8]

If so, then that idea seemed insecure, because the connections between enclaves could be swapped (i.e. the specific "network" of enclaves is not attested), and had quite huge practical problems like breaking the cycles (both enclaves can't hardcode each other's hashes at the same time).

You don't have to hard code MRENCLAVE in both enclaves: you could, for example, use MRSIGNER, ISVPRODID, ISVEXTPRODID and ISVFAMILYID (or a subset) to identify the other enclave (the first is derived from the signing public key, the others are signed in SIGSTRUCT). This will require some preliminary work to figure out the other enclave's MRENCLAVE, but at least the circular dependency is eliminated.

Support for Local Attestation (LA-TLS)

I don't think TLS is really needed here: using EREPORTs, we have a mechanism to authenticate both enclaves and have an integrity-protected channel, so the overhead introduced by TLS handshake is not really needed. Also, there is not much need to agree on a cipher suite. Instead, the report data can be used to perform a key exchange using some SIGMA protocol.

[vijaydhanraj]

You don't have to hard code MRENCLAVE in both enclaves: you could, for example, use MRSIGNER

Yes, agree and in addition, maybe can also generate and verify report data over K_e | tag where K_e is the Diffie-Hellman session key. (Note: This is done currently in Gramine as part of LA when a child is forked from a parent).

I don't think TLS is really needed here

The reason for having a TLS channel is to securely share a master key between the enclaves used for pipes' encryption as well as passing any other secrets/encryption key to the other enclave.

[DL8]

You don't have to hard code MRENCLAVE in both enclaves: you could, for example, use MRSIGNER

Yes, agree and in addition, maybe can also generate and verify report data over K_e | tag where K_e is the Diffie-Hellman session key. (Note: This is done currently in Gramine as part of LA when a child is forked from a parent).

I don't know the details of the protocol in case of fork, but it may be reusable.

I don't think TLS is really needed here

The reason for having a TLS channel is to securely share a master key between the enclaves used for pipes' encryption as well as passing any other secrets/encryption key to the other enclave.

We do want to establish a secure channel, but TLS is an overkill in this specific case, because by using EREPORT we have identity and integrity guarantees.

[mkow]

We do want to establish a secure channel, but TLS is an overkill in this specific case

I agree, and it's similar with other places where we use TLS in Gramine, but AFAIR we picked TLS because we couldn't find a good library implementing encrypted + integrity-checked byte streams with preshared keys and we didn't want to write a custom one.

you could, for example, use MRSIGNER, ISVPRODID, ISVEXTPRODID and ISVFAMILYID (or a subset) to identify the other enclave

@DL8: But how you then actually differentiate between one of your enclaves? You need to know who you're talking to, to verify whether the untrusted host didn't shuffle connections between your enclaves (i.e. connected your nginx process to the database instead of PHP process and now they're exchanging some garbage as trusted data). My point is rather high-level: you need to actually know the application you want to connect to, you can't leave this choice to the untrusted host, even if it's limited to the apps from that specific deployment.

[...] maybe can also generate and verify report data over K_e | tag where K_e is the Diffie-Hellman session key. (Note: This is done currently in Gramine as part of LA when a child is forked from a parent).

@vijaydhanraj: I don't see how it's relevant anyhow here.

[vijaydhanraj]

I don't see how it's relevant anyhow here.

I wanted to add a unique per enclave element in the report (report data) that could be cross verified by the other party. With just MRSigner we don't have this uniqueness between the enclaves signed by the same author.

[DL8]

you could, for example, use MRSIGNER, ISVPRODID, ISVEXTPRODID and ISVFAMILYID (or a subset) to identify the other enclave

@DL8: But how you then actually differentiate between one of your enclaves? You need to know who you're talking to, to verify whether the untrusted host didn't shuffle connections between your enclaves (i.e. connected your nginx process to the database instead of PHP process and now they're exchanging some garbage as trusted data). My point is rather high-level: you need to actually know the application you want to connect to, you can't leave this choice to the untrusted host, even if it's limited to the apps from that specific deployment.

As enclave developer, you should define a different ISVPRODID (and possibly additional KSS fields) per enclave. In the example you provided, developer can define the following values for ISVPRODID:

• 0 - nginx server
• 1 - database
• 2 - PHP process

Since ISVPRODID is signed in SIGSTRUCT, it reflects the type of the enclave that was launched, except that it doesn't change across builds/versions.

[mkow]

except that it doesn't change across builds/versions

Hmm, but you need to also check the version and refuse to communicate with old ones (with security bugs). But I guess you wanted to say "builds", not "builds/versions"?

[DL8]

Hmm, but you need to also check the version and refuse to communicate with old ones (with security bugs).

For that purpose the ISVSVN and CONFIGSVN fields are in the report. It is the enclaves' responsibility to decide whether or not to accept the communication. Once accepted, I would expect enclave developer to add version information/handshake

As for functional versions:

But I guess you wanted to say "builds", not "builds/versions"?

I meant "builds/versions". Once both enclaves attested each other including their security versions, they should negotiate the actual version as part of their protocol. It's possible to use product ID fields to indicate the software version, but in my opinion it's a bit of an abuse of the mechanism and is harder to maintain

[boryspoplawski]

Hmm, but you need to also check the version and refuse to communicate with old ones (with security bugs).

For that purpose the ISVSVN and CONFIGSVN fields are in the report. It is the enclaves' responsibility to decide whether or not to accept the communication. Once accepted, I would expect enclave developer to add version information/handshake

As for functional versions:

But I guess you wanted to say "builds", not "builds/versions"?

I meant "builds/versions". Once both enclaves attested each other including their security versions, they should negotiate the actual version as part of their protocol. It's possible to use product ID fields to indicate the software version, but in my opinion it's a bit of an abuse of the mechanism and is harder to maintain

Wait, if you require enclaves to negotiate stuff as part of the protocol, it's enough to use MRSIGNER. Rest of the information can be exchanges via some secure channel established once attestation was successful, there is no need for supporting HW features for that.

[vijaydhanraj]

So, for two enclaves to mutually attest each other signed by the same author, we need

1. MRsigner to establish evidence.
2. ISVPRODID to establish identity (to ensure we are not interacting with other enclaves signed by the same author). And to further check versions, enclaves can use ISVSVN and CONFIGSVN.

Would this be fair to say?

[boryspoplawski]

@vijaydhanraj I don't think you need 2 at all - after 1 you have a secure channel and can exchange any identity information via it. You can trust the other endpoint of that channel, because it's all the same MRSIGNER. Since there was some assumption you need to exchange some information anyway (as @DL8 said), I see no point in 2.

[DL8]

@vijaydhanraj I don't think you need 2 at all - after 1 you have a secure channel and can exchange any identity information via it. You can trust the other endpoint of that channel, because it's all the same MRSIGNER. Since there was some assumption you need to exchange some information anyway (as @DL8 said), I see no point in 2.

I think you misunderstood me: my point from my comment above was that attestation guarantees only security version of the enclaves. Actual version negotiation, if needed, is part of the application's protocol and is out of scope for attestation.

Regarding the fields to use, MRSIGNER alone is not enough: it only identifies who signed the enclave, but there is absolutely no guarantee about the actual enclave's identity. The same key can be used to sign multiple enclaves, which may be entirely different and unrelated.

So, for two enclaves to mutually attest each other signed by the same author, we need

1. MRsigner to establish evidence.
2. ISVPRODID to establish identity (to ensure we are not interacting with other enclaves signed by the same author). And to further check versions, enclaves can use ISVSVN and CONFIGSVN.

Would this be fair to say?

I thought about it again, and I am not sure we can force any rules whether or not the report should be accepted. For example, if we force both enclaves to have the same MRSIGNER, it will block a developer that chose to trust 3rd party enclaves signed by selected other MRSIGNERs. To be pragmatic, I would expect local attestation to get a report verification callback, which will be invoked after the MAC verification. I would expect its signature to be something like this: bool verify_report_contents_callback(const sgx_report_t *report, void *user_data);

The enclave developer is expected implement the logic that decides whether or not to trust the other enclave. I would expect a reasonable enclave to verify the following:

1. MRSIGNER: the identity of the signer
2. ISVPRODID and ISVSVN: the identity of the enclave and its security version
3. Additional KSS fields if needed and supported

[vijaydhanraj]

I thought about it again, and I am not sure we can force any rules whether or not the report should be accepted. For example, if we force both enclaves to have the same MRSIGNER, it will block a developer that chose to trust 3rd party enclaves signed by selected other MRSIGNERs.

Thanks @DL8 but the point is to avoid trusted third parties (TTPs) and get attested locally. If we were to have any such TTPs, then one can simply use remote attestation, right?

[BFuhry]

Note that my answer is based on the following assumptions:

1. Enclave A with functionality AF wants to attest enclave B with functionality BF
2. Enclave A fully trusts the developer of enclave B
3. Enclave A knows MRSIGNER of enclave B

If the best design we can come up with anyhow requires some negotiation step between the enclaves, I fully agree with the statement of @boryspoplawski:

Wait, if you require enclaves to negotiate stuff as part of the protocol, it's enough to use MRSIGNER. Rest of the information can be exchanges via some secure channel established once attestation was successful, there is no need for supporting HW features for that.

Involving other fields like ISVPRODID, ISVSVN, additional KSS fields, etc. seem not necessary in this case. Instead, Enclave A does local attestation with enclave B, checks the MRSIGNER, and establishes a secure channel. Afterwards, enclave B can just "say" to enclave A over the secure channel: I provide functionality BF. Even better, enclave B can say: I provide functionality BF in version BV. As a result, A would be sure that it talks to the enclave with the desired functionality and it could decide if the version is recent enough or not. Obviously, the same process can be done in the other direction for mutual trust.

Theoretically, B could lie about its functionality and version. However, this would contradict assumption 2. Furthermore, without assumption 2, ISVPRODID, ISVSVN, additional KSS fields, etc. would also not help as the developer of enclave B could lie in these fields as well.

An additional negotiation step might not even be necessary if BF and BV can be part of the report that B sends to A.

[vijaydhanraj]

Yes, true this Enclave A fully trusts the developer of enclave B this is critical for attestation using MRsigner.

@boryspoplawski @mkow @monavij But in cases where such trust cannot be expected, can we have an encrypted directory specified in the manifest between two enclaves that want to locally attest each other? Then both enclaves can share some randomly generated unique ID in this directory which can be compared with the report that is exchanged between the enclaves.

[DL8]

Thanks @DL8 but the point is to avoid trusted third parties (TTPs) and get attested locally. If we were to have any such TTPs, then one can simply use remote attestation, right?

If a another signer is trusted, I would expect it to be known at build time to have its MRSIGNER embedded it in the code. Therefore, a trusted third party (sort of a certificate authority) is not required.

Wait, if you require enclaves to negotiate stuff as part of the protocol, it's enough to use MRSIGNER. Rest of the information can be exchanges via some secure channel established once attestation was successful, there is no need for supporting HW features for that.

Involving other fields like ISVPRODID, ISVSVN, additional KSS fields, etc. seem not necessary in this case. Instead, Enclave A does local attestation with enclave B, checks the MRSIGNER, and establishes a secure channel. Afterwards, enclave B can just "say" to enclave A over the secure channel: I provide functionality BF. Even better, enclave B can say: I provide functionality BF in version BV. As a result, A would be sure that it talks to the enclave with the desired functionality and it could decide if the version is recent enough or not. Obviously, the same process can be done in the other direction for mutual trust.

While enclave may choose to use only MRSIGNER to verify the other enclave, I see several problems with it:

1. Not checking ISVSVN (possibly CONFIGSVN) is a security bug: if security bugs are found in the enclave, the SVN is expected to be incremented once those were fixed. Therefore, if not checked, B might accept a vulnerable version of A
2. ISVPRODID, ISVEXTPRODID and ISVFAMILYID are already part of the report. Not considering them may lead to a maintenance headache: there may be several entirely different and unrelated enclaves signed by the same signer. Not checking those fields will force the developer to implement another layer of authentication for all enclaves, to determine if we're talking to B and not C
3. I find it a questionable practice to continue with the key exchange when we can already know that it's going to be rejected

Theoretically, B could lie about its functionality and version. However, this would contradict assumption 2. Furthermore, without assumption 2, ISVPRODID, ISVSVN, additional KSS fields, etc. would also not help as the developer of enclave B could lie in these fields as well.

ISVSVN, ISVPRODID and extended product ID fields are signed in SIGSTRUCT. Therefore, B lying about its functionality and security version is possible in one of the following cases:

1. The private key was stolen
2. A software bug

An additional negotiation step might not even be necessary if BF and BV can be part of the report that B sends to A.

That's basically my argument: we don't need an additional negotiation step because we already have relevant fields in the report.

Yes, true this Enclave A fully trusts the developer of enclave B this is critical for attestation using MRsigner.

This is up to the developer to decide. It can be in build time or in any other way.

[monavij]

We are mixing multiple things here. So let me try and clarify

1. With remote attestation you can verify that you are running on a good up to date Intel system and running in an enclave
2. With local attestation you can just verify that you are running in an enclave on the same machine (it might be out of date)
3. In both cases we need a Trusted party that can verify your MRENCLAVE. That can be a trusted third party or you can be your own trusted party. As long as you know who you are going to trust you can make sense out of MRENCLAVE value.

With this premise we can now build a solution that can enable encrypted unix domain sockets between two enclaves running on the same machine.

1. We already know we can use MRENCLAVE as the ID of the enclave. So we just create a signed white list of MRENCAVE values and include the public key of the signer in your manifest.
2. We could put this public key in KSS as well, in case you have different trusted parties on different machines, your MRENCLAVE value remains the same.
3. If there are just two enclaves then they can also include each others MRENCLAve value in KSS field.

[DL8]

We are mixing multiple things here. So let me try and clarify

1. With remote attestation you can verify that you are running on a good up to date Intel system and running in an enclave
2. With local attestation you can just verify that you are running in an enclave on the same machine (it might be out of date)
3. In both cases we need a Trusted party that can verify your MRENCLAVE. That can be a trusted third party or you can be your own trusted party. As long as you know who you are going to trust you can make sense out of MRENCLAVE value.

I don't agree with statement 3. As I see it, there are two main steps in the attestation:

1. Integrity check: enclave verifies the integrity of the report (EGETKEY and MAC). If this succeeded, we know the report came from an enclave running on the same platform, and that its contents indeed reflect its identity
2. Content check: enclave decides whether or not to accept the report. A trusted third party is not required: for example, the enclave may decide based on hard-coded expectations from MRSIGNER, ISVPRODID and ISVSVN

With this premise we can now build a solution that can enable encrypted unix domain sockets between two enclaves running on the same machine.

1. We already know we can use MRENCLAVE as the ID of the enclave. So we just create a signed white list of MRENCAVE values and include the public key of the signer in your manifest.
2. We could put this public key in KSS as well, in case you have different trusted parties on different machines, your MRENCLAVE value remains the same.
3. If there are just two enclaves then they can also include each others MRENCLAve value in KSS field.

Using MRENCLAVE will be very hard to maintain. For example, you would potentially have to update the list every time one enclave in the ecosystem is modified. For example, let's say we have at least two enclaves: A and B. If A was updated, you would have to add its new MRENCLAVE to the list and keep the old value to maintain compatibility. Moreover, you would have to maintain different lists per build configuration (e.g. debug vs. release).

Even if other fields are used instead, I still see some problems with the external list approach:

1. Revocation: how is an enclave revoked? a new signed list may be provided, but what prevents an attacker from giving the enclave a previous version of the list?
2. Flexibility: enclave may behave differently under certain circumstances. For example, let's say we have enclave A that communicates an in-memory key-value store B and a DB server C. If it wants to establish connection with B but gets a report from C, it would want to reject the connection. In the list approach this would not be possible. It might get even more complicated if different enclave configurations are supported via KSS
3. KSS fields are a scarce resource and may impact key derivation. Therefore, other potential use cases must be carefully examined before taking a decision to reserve part of them for Gramine usage
4. There is no guarantee that the other enclave uses Gramine

In conclusion, I think there are too many unknowns to get a proper decision. Therefore, once the report is verified, the decision must remain the responsibility of the application itself. Gramine can provide examples or built-in callbacks, but forcing a mechanism might not be the right thing to do.

[boryspoplawski]

For example, you would potentially have to update the list every time one enclave in the ecosystem is modified. For example, let's say we have at least two enclaves: A and B. If A was updated, you would have to add its new MRENCLAVE to the list and keep the old value to maintain compatibility. Moreover, you would have to maintain different lists per build configuration (e.g. debug vs. release).

The very same thing can be said about KSS or any other ID(s) based system. You need a way to differentiate between configs, it doesn't really matter what you threat as an ID.

Revocation: how is an enclave revoked? a new signed list may be provided, but what prevents an attacker from giving the enclave a previous version of the list?

The list would have to come from a trusted third part, presumably as a part of remote attestation (or via some secure channel established with remote attestation). This doesn't have to be a list, it can be any format you want it to be.

There is no guarantee that the other enclave uses Gramine

There is no point in attestation then, from Gramine point of view. App can do whatever it wants and attest the remote end, Gramine should not interfere (and hence there's no problem with this).

Generally speaking I firmly believe that Gramine should only act as and interface for HW features (just provide an easy to use API, like /dev/ files) and allow the app to do whatever it wants. Whether it uses MRENCLVE, KSS, anything, it does not matter - it's up to the app. We could provide some examples like we do with RA-TLS, but again, that would be only an example usage of the API. This wouldn't support the seamless encryption of UNIX domain sockets though (but I never believed this is a good idea).

[monavij]

Ok what is the problem with what I proposed above? Where you have a trusted party that maintains the MRENCLAVEs? Having seamless encryption of unix domain sockets is an imp feature and there is a secure way to do it.

[DL8]

The very same thing can be said about KSS or any other ID(s) based system. You need a way to differentiate between configs, it doesn't really matter what you threat as an ID.

Yes, but it's much easier to maintain and verify those fields, because they are not bound to change with each build. I would expect that for each enclave, developer defines a different constant for product ID in the manifest, and it will not change often (if at all). This can be set with sgx.isvprodid and other fields can be added for KSS support in the future. sgx.isvsvn will be used to define the security version of the enclave, so other parties can reject prior vulnerable versions.

There is no guarantee that the other enclave uses Gramine

There is no point in attestation then, from Gramine point of view. App can do whatever it wants and attest the remote end, Gramine should not interfere (and hence there's no problem with this).

Generally speaking I firmly believe that Gramine should only act as and interface for HW features (just provide an easy to use API, like /dev/ files) and allow the app to do whatever it wants. Whether it uses MRENCLVE, KSS, anything, it does not matter - it's up to the app. We could provide some examples like we do with RA-TLS, but again, that would be only an example usage of the API.

Sounds like we pretty much agree. As for usage, I would expect Gramine to provide a handle to verify the integrity of the report. The app is then expected to check the report's contents and decide.

Ok what is the problem with what I proposed above? Where you have a trusted party that maintains the MRENCLAVEs?

Technically speaking, local attestation is done after both enclaves verified each others' reports. I believe that in most cases, there is enough data in the report for the enclave to reach a decision without using MRENCLAVE, so forcing a trusted party will increase the TCB without a real justification.

Having seamless encryption of unix domain sockets is an imp feature and there is a secure way to do it.

How to establish a secure channel is an important topic that hasn't been discussed so far.

I have no opinion regarding this feature, but if implemented, I believe it should have (at least) the following in the handshake:

1. Communicating enclaves exchange reports with key exchange material (depending on the protocol)
2. In Gramine: each receiving enclave verify the integrity of the other's report
3. In the application: each receiving enclave verifies the content of the report and chooses whether or not to accept the connection

[boryspoplawski]

Ok what is the problem with what I proposed above? Where you have a trusted party that maintains the MRENCLAVEs? Having seamless encryption of unix domain sockets is an imp feature and there is a secure way to do it.

@monavij The problem is that Gramine itself would need to do the attestation, not only expose API/interfaces for the app to do that. /dev/ files wouldn't be enough, you would need some configuration of which enclaves can connect to which enclave on which UNIX domain socket i.e. an extensive configuration that would have to be tightly coupled with Gramine and would offer no flexibility, e.g. if we proceed with MRENCLAVE that would mean the config cannot be provided at build time, but have to arrive via secure channel with RA, if we go with MRSIGNER + KSS/ISV* fields then users would have to use them in their software, MRENCLAVE wouldn't be enough. Supporting both would make it even more complex.

How to establish a secure channel is an important topic that hasn't been discussed so far.

@DL8 For this we have code already, we do local attestation on fork() (but in our case child and parent have exactly same MRENCLAVE). It's just a matter of deciding who to trust/attest, the rest shouldn't be a problem

[monavij]

@boryspoplawski - Agree with you that for unix domain socket the onus is on Gramine to do the establishment of secure channel. Also we would like to make that mechanism flexible and let application decide how they want to establish trust in the other local enclaves. Not sure if that will make Gramine more complex or just more flexible. KSS is just another tool in our toolbox. We should add support for KSS in Gramine and allow applications to add whatever they want to use that field for. Gramine does not need to worry what they use it for. My main point is that we have mechanisms for being able to do secure Unix domain sockets.

[DL8]

@boryspoplawski - Agree with you that for unix domain socket the onus is on Gramine to do the establishment of secure channel. Also we would like to make that mechanism flexible and let application decide how they want to establish trust in the other local enclaves. Not sure if that will make Gramine more complex or just more flexible. KSS is just another tool in our toolbox. We should add support for KSS in Gramine and allow applications to add whatever they want to use that field for. Gramine does not need to worry what they use it for. My main point is that we have mechanisms for being able to do secure Unix domain sockets.

Sounds like an existing mechanism can be reused for this purpose, which may be the right thing to do. However, "raw attestation" interface may be needed as well, in case the other enclave doesn't use Gramine and another protocol is required. Regarding KSS, note that local attestation can be achieved without it, so it should not be a dependency. Also note that even if it is supported in the platform, it must be explicitly enabled in SECS attributes. The attributes are also part of the report, so the verifying enclave may make a decision based on KSS status.

[vijaydhanraj]

To summarize the scope:

1. For raw attestation:

   a. Gramine already exposes raw attestation interfaces via /dev/attestation. b. Local attestation between parent and forked child is already supported. c. For Local Attestation between 2 different enclaves running on the same host using Gramine LibOS:

   • No changes are needed in core Gramine LibOS.
   • Gramine can provide a reference library for local attestation similar to RA-TLS for remote attestation. (onus is one application to decide what they choose to trust)
     • MRsigner, ISVPRODID and ISVSVN should be sufficient.
     • The apps can use the library to exchange reports and verify mac.
     • Application can use the verified report to check PRODID to ensure it is communicating with the intended enclave.
     • If needed the application can also use reportdata to securely pass information between the enclaves. (From SDM: REPORTDATA is a 64-Byte data structure that is provided by the enclave and included in the REPORT. It can be used to securely pass information from the enclave to the target enclave)

   d. For Local Attestation between 2 different enclaves running on the same host using different LibOS:

   • Should be possible to use the same steps as mentioned in case 2 but some standardization will be required between different LibOSes. (Or LibOSes can choose to support a custom interface)

2. Seamless communication over Unix Domain Sockets between Parent and child enclave using Gramine. - Not sure if this is currently supported but have all things in place to enable it. @boryspoplawski can you please confirm?

3. Communication over Unix Domain Sockets between enclaves using Gramine from same signing authority. a. Pre-establish the two enclaves that may communicate over UDS. May need manifest changes something like,

   • Set the pathname [sun_path] for both enclaves to use.
   • Encrypted path where each enclave tells Gramine that is done doing LA of the other enclave. Gramine can check this before proceeding to establish a UDS channel.)

   b. Application should use the LA library from Gramine. c. Application verifies the identity of the other enclave. d. Application sets up a secure channel and exchanges an encryption wrap key. d. Application writes to the encrypted path that it is done with LA and trusts the other enclave. e. Gramine checks the encrypted path and sets up UDS communication.

   Or another option where Gramine can trigger the local attestation transparently, but may need more information may need to be added to the manifest like who takes the server/client role? and what is the expected PRODID of the participating enclaves etc.

4. Communication over Unix Domain Sockets between enclaves using Gramine from different signing authority. - Not supported. Need same MRsigner

5. Communication over Unix Domain Sockets between Gramine and other LibOSes - Not supported for now. Need some standardization in the near future)

Thoughts/comments, please.

[DL8]

To summarize the scope:

1. For raw attestation: a. Gramine already exposes raw attestation interfaces via /dev/attestation.

What do you mean by "raw attestation"? As I understand it, it's the use case where the app uses Gramine only to verify a given report's integrity. I would expect Gramine to provide higher level APIs for secure channel establishment, such that if both enclaves use Gramine, these can be used and Gramine will take care of the key exchange details.

c. For Local Attestation between 2 different enclaves running on the same host using Gramine LibOS:

• No changes are needed in core Gramine LibOS.

• Gramine can provide a reference library for local attestation similar to RA-TLS for remote attestation. (onus is one application to decide what they choose to trust)

  • MRsigner, ISVPRODID and ISVSVN should be sufficient.
  • The apps can use the library to exchange reports and verify mac.
  • Application can use the verified report to check PRODID to ensure it is communicating with the intended enclave.
  • If needed the application can also use reportdata to securely pass information between the enclaves. (From SDM: REPORTDATA is a 64-Byte data structure that is provided by the enclave and included in the REPORT. It can be used to securely pass information from the enclave to the target enclave)

Note that once MAC is verified, the application may use all of the report's fields to decide. Gramine should provide a callback to let the app verify the report's contents and act accordingly. If we provide examples, I would expect the most standard flow to verify ISVPRODID, ISVSVN and MRSIGNER.

d. For Local Attestation between 2 different enclaves running on the same host using different LibOS:

• Should be possible to use the same steps as mentioned in case 2 but some standardization will be required between different LibOSes. (Or LibOSes can choose to support a custom interface)

The exact protocol may be defined at the application level (either explicitly or with some common 3rd party library). Note that raw attestation API fits this use case:

1. The two enclaves establish a communication channel (be it pipes, sockets, shared memory or any other medium)
2. Enclaves exchange reports with custom REPORTDATA (contents are defined by their protocol)
3. Gramine-based enclave uses report verification API to verify the other report's integrity
4. If passed, Gramine enclave verifies the content of the report and decides whether or not to accept it
5. If passed, Gramine enclave uses data from REPORTDATA to establish a secure channel

3. Communication over Unix Domain Sockets between enclaves using Gramine from same signing authority. a. Pre-establish the two enclaves that may communicate over UDS. May need manifest changes something like,

   • Set the pathname [sun_path] for both enclaves to use.
   • Encrypted path where each enclave tells Gramine that is done doing LA of the other enclave. Gramine can check this before proceeding to establish a UDS channel.)

   b. Application should use the LA library from Gramine. c. Application verifies the identity of the other enclave. d. Application sets up a secure channel and exchanges an encryption wrap key. d. Application writes to the encrypted path that it is done with LA and trusts the other enclave. e. Gramine checks the encrypted path and sets up UDS communication.

Are you sure two sockets are required? Once key exchange is done, the same socket may be used to communicate.

Or another option where Gramine can trigger the local attestation transparently, but may need more information may need to be added to the manifest like who takes the server/client role? and what is the expected PRODID of the participating enclaves etc.

I don't think it's a good idea to do it in the manifest, because there are other fields to consider and the actual decision may depend on runtime circumstances.

4. Communication over Unix Domain Sockets between enclaves using Gramine from different signing authority. - Not supported. Need same MRsigner

Why limit this mechanism to the same signing authority? If the app provides a callback, we don't need this restriction.

[vijaydhanraj]

What do you mean by "raw attestation"?

Please see here the low-level interfaces exposed by Gramine. This is what I mean by raw attestation. https://gramine.readthedocs.io/en/stable/attestation.html#low-level-dev-attestation-interface And yes, the idea is to create a library that will verify the integrity of the report by checking the mac but whether to expose the entire report, or limit it to just a few fields, establish a secure channel and other APIs related to user interface need to be finalized with maintainers.

Note that once MAC is verified, the application may use all of the report's fields to decide. Gramine should provide a callback to let the app verify the report's contents and act accordingly. If we provide examples, I would expect the most standard flow to verify ISVPRODID, ISVSVN and MRSIGNER

Yes, it makes sense.

d. For Local Attestation between 2 different enclaves running on the same host using different LibOS:

• Should be possible to use the same steps as mentioned in case 2 but some standardization will be required between different LibOSes. (Or LibOSes can choose to support a custom interface)

The exact protocol may be defined at the application level (either explicitly or with some common 3rd party library). Note that raw attestation API fits this use case:

1. The two enclaves establish a communication channel (be it pipes, sockets, shared memory or any other medium)
2. Enclaves exchange reports with custom REPORTDATA (contents are defined by their protocol)
3. Gramine-based enclave uses report verification API to verify the other report's integrity
4. If passed, Gramine enclave verifies the content of the report and decides whether or not to accept it
5. If passed, Gramine enclave uses data from REPORTDATA to establish a secure channel

Please note the library is built based on low-level raw attestation interfaces (/dev/attestation) and when using with other LibOSes would need to be customized. This is what I meant by the comment.

Are you sure two sockets are required? Once key exchange is done, the same socket may be used to communicate.

The key exchange will not use a UDS as this is what we want to set up after the local attestation and other verification steps.

Or another option where Gramine can trigger the local attestation transparently, but may need more information may need to be added to the manifest like who takes the server/client role? and what is the expected PRODID of the participating enclaves etc.

I don't think it's a good idea to do it in the manifest

Agree this will be a little tightly coupled with Gramine but the idea behind the proposal is to make the end-user's life easier. With this proposal the user needs to set the reportdata, PRODID they expect in the manifest and Gramine can transparently check the integrity (from MAC), verify the MRsigner and other ISV fields to ensure the other enclave is trusted and indeed the expected one. Gramine, can go ahead and then set up a UDS. But this imposes a restriction where the enclave identification is by `MRSigner/ISV` fields.

because there are other fields to consider and the actual decision may depend on runtime circumstances.

Can you elaborate more on this?

Why limit this mechanism to the same signing authority? If the app provides a callback, we don't need this restriction.

We may need this if we were to go with the above option. If not, then yes, this restriction doesn't apply.

[DL8]

What do you mean by "raw attestation"?

Please see here the low-level interfaces exposed by Gramine. This is what I mean by raw attestation. https://gramine.readthedocs.io/en/stable/attestation.html#low-level-dev-attestation-interface And yes, the idea is to create a library that will verify the integrity of the report by checking the mac but whether to expose the entire report, or limit it to just a few fields, establish a secure channel and other APIs related to user interface need to be finalized with maintainers.

Thanks for the reference. For the low-level attestation API, I would expect a handle called /dev/attestation/verify_report to be added. The app is expected to use it as follows:

1. Write the contents of the report to verify
2. Read a single byte to get the MAC verification results:
   • 0 - fail
   • 1 - success

Note that for the scope of this handle I assume the app already has the report at hand (how to get it is out of scope). @mkow - I think this API could be useful added regardless of local attestation. Should it be a separate feature request?

d. For Local Attestation between 2 different enclaves running on the same host using different LibOS:

• Should be possible to use the same steps as mentioned in case 2 but some standardization will be required between different LibOSes. (Or LibOSes can choose to support a custom interface)

The exact protocol may be defined at the application level (either explicitly or with some common 3rd party library). Note that raw attestation API fits this use case:

1. The two enclaves establish a communication channel (be it pipes, sockets, shared memory or any other medium)
2. Enclaves exchange reports with custom REPORTDATA (contents are defined by their protocol)
3. Gramine-based enclave uses report verification API to verify the other report's integrity
4. If passed, Gramine enclave verifies the content of the report and decides whether or not to accept it
5. If passed, Gramine enclave uses data from REPORTDATA to establish a secure channel

Please note the library is built based on low-level raw attestation interfaces (/dev/attestation) and when using with other LibOSes would need to be customized. This is what I meant by the comment.

Yes, the non-Gramine-based enclave will have a different abstraction (or bare-bone Assembly), but this is out of scope for us.

Are you sure two sockets are required? Once key exchange is done, the same socket may be used to communicate.

The key exchange will not use a UDS as this is what we want to set up after the local attestation and other verification steps.

A UDS is a communication channel between processes. Local attestation and key exchange are done on top of that to establish a secure channel over it (or any other IPC mechanism). It doesn't really matter whether the handshake is performed on the same insecure channel or a different one, as long as in the end both enclaves use the same scheme and keys. This is analogous to TLS, where a secure channel is established over an insecure TCP/IP connection (and then the same socket is used to continue the communication).

Or another option where Gramine can trigger the local attestation transparently, but may need more information may need to be added to the manifest like who takes the server/client role? and what is the expected PRODID of the participating enclaves etc.

I don't think it's a good idea to do it in the manifest

Agree this will be a little tightly coupled with Gramine but the idea behind the proposal is to make the end-user's life easier. With this proposal the user needs to set the reportdata, PRODID they expect in the manifest and Gramine can transparently check the integrity (from MAC), verify the MRsigner and other ISV fields to ensure the other enclave is trusted and indeed the expected one. Gramine, can go ahead and then set up a UDS. But this imposes a restriction where the enclave identification is by `MRSigner/ISV` fields.

Well, it could make is slightly easier for the developer to predefine a set of accepted enclaves. However, it might make SVN increment slightly more of a headache: for example, let's say that enclave A is used by enclave B and C, all of them are signed by the same key (same MRSIGNER). If I understand correctly, the expected (minimal?) ISVSVN of A will have to be in the manifest of B and C along with its ISV* identifiers. Now, if a security update is applied to A, its ISVSVN is incremented in its manifest. With this solution, it will have to be incremented in B's and C's manifests, otherwise they may be able to communicate with a vulnerable version of A. On the other hand, the alternative may be that developer does that manually. On the technical side, note that REPORTDATA is the field that is used for the actual enclave communication, so it should not be added to the manifest (in this case I would expect Gramine to use it as part of the the key exchange protocol).

because there are other fields to consider and the actual decision may depend on runtime circumstances.

Can you elaborate more on this?

Let's say we have an enclave that is expected to communicate with a DB server (e.g. SQL server) and a key-value store (e.g. memcached), both running in enclave and all have the same MRSIGNER and proper ISVPRODID. If the enclave tries to connect to the DB server but gets a quote from the memcached server, it will make more sense to reject the connection, even though it is a trusted enclave. If it tries to connect to the DB and gets a DB, it will make sense to accept. I don't know if it's a real use case though, but it sounds possible.

[dimakuv]

Let me add more chaos to this issue :)

1. The name "LA-TLS" is a bit misleading, and I regret coining it. The original thought was "This looks very much like RA-TLS", but now I see significant differences:

   • RA-TLS builds on top of TLS code of the native app. In other words, native app already encrypts its communication channels with TLS, and RA-TLS simply "augments" the TLS handshake with additional SGX-specific verification steps.
   • LA-TLS builds on top of plaintext-communications code of the native app. Applications always use UDS (aka AF_UNIX) without TLS. Thus, LA-TLS must not only augment the TLS handshake, but actually wrap UDS communications in TLS first.
   • I don't have a better name for it now, so I'll continue calling it "LA-TLS".

2. There are two things to be implemented for LA-TLS:

   1. The wrapping of UDS sessions in TLS. This is similar to how we do it here (for pipes and one-Gramine-instance UDSes): https://github.com/gramineproject/gramine/blob/48f4660a4b5e3e469922bb8cef7e99a336c06b59/pal/src/host/linux-sgx/pal_pipes.c
   2. The SGX LA-based TLS handshake. This is similar to how we do it here (during fork, for the parent and the child to establish a TLS channel): https://github.com/gramineproject/gramine/blob/48f4660a4b5e3e469922bb8cef7e99a336c06b59/pal/src/host/linux-sgx/pal_process.c

3. I agree with @boryspoplawski and @mkow that the policies to decide which SGX enclaves to trust are hard to embed in core Gramine (either as some manifest options as some hard-coded policies). It was easy with forking because there we applied a strict policy of same-MRENCLAVE: https://github.com/gramineproject/gramine/blob/48f4660a4b5e3e469922bb8cef7e99a336c06b59/pal/src/host/linux-sgx/pal_process.c#L111-L127

However, it is really hard to decide how to trust SGX enclaves in the general case. In RA-TLS, we have a special callback: https://github.com/gramineproject/gramine/blob/cf43bbca4c48456378b34621a1a271c6b3bd8148/tools/sgx/ra-tls/ra_tls.h#L67-L82

Maybe we could introduce a similar callback in LA-TLS, just like @DL8 suggested.

My main problem currently: How do we combine the "wrapping of UDS sessions in TLS" (must be implemented in core Gramine) and the "SGX LA-based TLS handshake" (should be implemented in a shared lib on top of core Gramine)? It feels like we'll need to introduce several callbacks from Gramine to the app to make LA-TLS workable. And this is not a good design.

[boryspoplawski]

I think we are mixing two different features here, which is confusing. A) Local attestation of unrelated enclaves - this should be all on user app / library. Gramine only exports /dev/ interfaces on top of which app can do the attestation (just like RA-TLS). I think it's non-controversial feature that we could have - after all we shift all attestation burden / deciding who to trust to the user app (we only provide interfaces to SGX related stuff). B) Seamless encryption of UNIX domain sockets. We have that for enclaves in one Gramine instance and the request is to support that for unrelated Gramine instances. This is a bit controversial, because it would require Gramine itself to do the local attestation - and to know who to trust we would need to introduce complex configuration / policy.

Btw @DL8 I think you are confusing what we mean by UDS. We mean sockets that are fully emulated inside Gramine, we do not do LA or other comms over them, we just need to establish a key for seamlessly encrypting them.

[DL8]

A) Local attestation of unrelated enclaves - this should be all on user app / library. Gramine only exports /dev/ interfaces on top of which app can do the attestation (just like RA-TLS). I think it's non-controversial feature that we could have - after all we shift all attestation burden / deciding who to trust to the user app (we only provide interfaces to SGX related stuff).

I agree, and I believe it can already be achieved with existing low-level attestation API.

B) Seamless encryption of UNIX domain sockets. We have that for enclaves in one Gramine instance and the request is to support that for unrelated Gramine instances. This is a bit controversial, because it would require Gramine itself to do the local attestation - and to know who to trust we would need to introduce complex configuration / policy.

Btw @DL8 I think you are confusing what we mean by UDS. We mean sockets that are fully emulated inside Gramine, we do not do LA or other comms over them, we just need to establish a key for seamlessly encrypting them.

Thanks for the clarification. Is the UDS an actual socket file on the filesystem, or is it emulated in Gramine sandbox? If emulated, I understand why LA doesn't happen (because it's already happened on fork()). If the intention is to add an option to create an actual socket file on the filesystem with seamless encryption, I agree that it's controversial also because of compatibility:

1. The app has already defined its protocol. Adding another layer in the middle will most likely break compatibility with other clients/servers
2. Connecting with non-SGX entities: depending on the app, it might be needed (e.g. an HTTPS server that serves clients even without SGX)
3. A key exchange and LA will have to be implemented/reused. In the absence of a standard, it won't be compatible with other LibOSes

In current version, is it possible in Gramine to communicate with Unix sockets in the filesystem? If not, a pragmatic solution may be to add an option for that. If corresponding settings are to be added to the manifest, they should have an insecure__ prefix and a disclaimer that it's the app's responsibility to secure the channel. Low-level attestation APIs may be good enough to let the app perform LA with the connecting party.

[boryspoplawski]

@DL8

in Gramine sandbox?

Gramine is not a sandbox.

Is the UDS an actual socket file on the filesystem, or is it emulated in Gramine sandbox?

It's emulated in Gramine. In PAL API (think of it as a backend) we have something called PAL pipes which we use to implement UNIX domain sockets and which on SGX Linux PAL are coincidentally implementes AS UNIX domain sockets, seamlessly encrypted. The encryption key is shared between all enclaves originating from the same source (attested via LA on fork). We do not support connecting to these sockets from outside of Gramine and we do not plan to. We also do not support Gramine communicating with UNIX domain sockets created by the host OS. There were requests to support connecting these sockets between different Gramine instances and it's part of this discussion (whether to do it and how).

[dimakuv]

Ok, I think me, @boryspoplawski and @DL8 are on the same page now.

I think we are mixing two different features here, which is confusing.

Agree. Maybe we should start discussing these two features separately. I'll try to discuss them separately from now on.

A) Local attestation of unrelated enclaves - this should be all on user app / library. Gramine only exports /dev/ interfaces on top of which app can do the attestation (just like RA-TLS). I think it's non-controversial feature that we could have - after all we shift all attestation burden / deciding who to trust to the user app (we only provide interfaces to SGX related stuff).

Yes, I also think it is a non-controversial feature and can be easily implemented, very analogous to RA-TLS. Actually, this feature (feature A in Borys's classification) is what I would call LA-TLS. This feature should be applied for AF_UNIX sockets that already use TLS in the native application.

B) Seamless encryption of UNIX domain sockets. We have that for enclaves in one Gramine instance and the request is to support that for unrelated Gramine instances. This is a bit controversial, because it would require Gramine itself to do the local attestation - and to know who to trust we would need to introduce complex configuration / policy.

Yes, I would call it AF_UNIX network shield (or AF_UNIX seamless encryption), instead of LA-TLS. This is indeed controversial. I disagree though that it "would require Gramine itself to do the local attestation" -- Gramine can expose an API to specify user-defined callbacks, and call these callbacks during the TLS handshake of these seamless-encrypted sessions. (No idea how to create such an API, since then the Gramine kernel would call back into the app, which is not something people do or expect.)

1. The app has already defined its protocol. Adding another layer in the middle will most likely break compatibility with other clients/servers

Yes, agreed. Both peers must talk the exact same protocol, i.e, both peers must be the same/compatible Gramine instances.

2. Connecting with non-SGX entities: depending on the app, it might be needed (e.g. an HTTPS server that serves clients even without SGX)

Yes, agreed. To support selective AF_UNIX seamless encryption, the user app can be restricted to create AF_UNIX paths under some special path. This is quite an ugly requirement though.

3. A key exchange and LA will have to be implemented/reused. In the absence of a standard, it won't be compatible with other LibOSes

Yes, that's the same as point 1 basically.

[dimakuv]

I know wonder, maybe we should just go with Feature A only (LA-TLS), and expose it as @DL8 suggests:

If corresponding settings are to be added to the manifest, they should have an insecure__ prefix and a disclaimer that it's the app's responsibility to secure the channel. Low-level attestation APIs may be good enough to let the app perform LA with the connecting party.

This would enable developers to use AF_UNIX sockets with SGX Local Attestation between independent Gramine instances, but the onus of wrapping these AF_UNIX connections in TLS sessions will be on the developers (not on Gramine).

[boryspoplawski]

Actually, this feature (feature A in Borys's classification) is what I would call LA-TLS. This feature should be applied for AF_UNIX sockets that already use TLS in the native application.

Why do you mix some UNIX domain sockets here? This is completely not important how the two enclaves communicate and can only bring confusion.

Yes, I would call it AF_UNIX network shield (or AF_UNIX seamless encryption), instead of LA-TLS.

I don't think anyone called this LA-TLS. It's just that it requires LA and hence the confusion.

Gramine can expose an API to specify user-defined callbacks, and call these callbacks during the TLS handshake of these seamless-encrypted sessions. (No idea how to create such an API, since then the Gramine kernel would call back into the app, which is not something people do or expect.)

No, please no user app/space callbacks. What we could have is /dev/gramine/unix_domain_sockets_keys special file, which you can open and write a struct into:

struct unix_domain_socket_key {
    char path[MAX_UDS_PATH_LEN == 256?];
    uint8_t key[64];
};
fd = open("/dev/gramine/unix_domain_sockets_keys", O_WRONLY);
struct unix_domain_socket_key key_struct = {
    .path = "/path/to/my/socket",
};
memcpy(&key_struct.key, some_key, key_size);
write(fd, &key_struct, sizeof(key_struct));

This way it's up to the user app to do the attestation and establish they shared key, so it nicely connects with problem A) and doesn't require extensive configuration in Gramine (it's all on the user app). Potential problem: we somehow need to set the encrytption key from LibOS, which can get ugly because e.g. these are generic PAL PIPEs and e.g. are not encrypted on Linux.

[dimakuv]

Why do you mix some UNIX domain sockets here? This is completely not important how the two enclaves communicate and can only bring confusion.

By LA-TLS, I mean a very concrete implementation of the "SGX-aware TLS library" on top of Gramine-SGX. Just like the case with RA-TLS.

Yes, LA-TLS should not be mixed with UNIX domain sockets in general -- any TLS session (no matter which transport layer it uses) can be converted to an LA-TLS session. But for all practical reasons, UNIX domain sockets (UDSes) are the only candidates currently to use LA-TLS.

I don't think anyone called this LA-TLS. It's just that it requires LA and hence the confusion.

What do you mean by "nobody called this LA-TLS"? This term is there in the title of this GitHub issue.

What we could have is /dev/gramine/unix_domain_sockets_keys special file ...

I like @boryspoplawski's proposal a lot! Indeed, then we can continue using TLS-PSK algo (just like we do currently for pipes in Linux-SGX PAL). And then we don't need LA-TLS (as I envisioned it, with some callbacks for fake X.509 certificates), it will be purely a shared-secret-establishment SGX Local Attestation flow.

[boryspoplawski]

What do you mean by "nobody called this LA-TLS"? This term is there in the title of this GitHub issue.

And the original post didn't mention seamless encryption of UNIX domain sockets at all. It was brought and mixed in later.

[dimakuv]

One unsolved problem is: how does the Gramine enclave know to which other Gramine enclave to connect? In other words, for Local Attestation, each enclave needs some well-known listening socket, and each other enclave must know this well-known listening socket (to be able to connect to it).

Taking our examples of Nginx, PHP, and MySQL database. Each runs in its own Gramine enclave. Each needs to perform Local Attestation to the other two. Currently (afair) Gramine enclaves do not have a well-known listening socket (there is a socket used by the parent Gramine enclave that listens for children, but afair it's not "well-known" but rather anonymous).

I can imagine a following solution, implemented purely in what I call the LA-TLS shared library:

1. Each enclave app upon startup calls la_tls_start_listening_tcpip_socket(/*port=*/11111). The port is hard-coded through an envvar in the manifest file (LA_TLS_LISTENING_TCPIP_PORT="11111").
2. LA-TLS creates this listening TCP/IP socket in a separate thread and waits on it.
3. If this app wants to connect to some peer enclave, it calls la_tls_connect_to_tcpip_socket(/*port=*/11112, &exchanged_session_key). The port is hard-coded through an envvar in the manifest file (LA_TLS_CONNECTING_TCPIP_PORTS="11112;11113"). The app iterates over the specified ports.
4. LA-TLS performs SGX local attestation with each of the peer enclaves, via TCP/IP connections, and establishes a session key with each of them.
5. The app now uses the code snippet proposed by @boryspoplawski above, and binds each session key to each UNIX Domain Socket (UDS) path.
6. After this, the app can connect(uds_path), and Gramine will consult /dev/gramine/unix_domain_sockets_keys, find this UDS path and its corresponding key, and wrap this UDS into a TLS-PSK session with this key.
7. Done! Now the app can talk to other apps (in other Gramine enclaves) over the UDSes, and this communications will be transparently encrypted by Gramine.

This scheme looks pretty cumbersome, so ideas are welcome.

[dimakuv]

Summary of discussion during the Gramine core meeting on 23. August

Dmitrii: the main unsolved question now is "how do enclaves find each other? e.g. via TCP/IP ports". The idea of finding another enclave through the IP port is generic enough. It maps to Enclave-CC use case: the agent enclave has a hard-coded IP port, and the app enclave connects to this port and performs SGX local attestation.

Borys: the port can also be chosen in the user-defined callback: la_tls_connect_to_tcpip_socket(port, &measurement_callback, &exchanged_session_key).

Mona: we also want to have LA-Secret-Prov library that can be preloaded and used for unmodified apps, in addition to the proposed LA-TLS library that requires app modifications. (This is analogous to how RA-TLS and SecretProv currently work.)

Borys: in the PoC for these features, there should be also production-ready callbacks with measurement verifications. In other words, the PoC should have some real-world usages, not toy examples.

Michal: why not start a Bash script from the same parent enclave, and then all UDSes are automatically SGX locally attested and TLS-encrypted?

• Counter use case: enclaves need to have different MRENCLAVEs.
• Counter use case: one "super-powerful" enclave (that has access to a lot of files) and many "small and less trusted" enclaves (have access to a few files).
• Counter use case: Docker containers in a Kubernetes Pod.

Michal and Borys: doubt that local-attestation policies can be easily constructed -- will still require some kind of whitelist, which gets complicated, and then the obvious solution boils down to: let's just use RA-TLS. Michal and Borys want to see the detailed end-to-end example which necessitates the need for LA-TLS.

Dmitrii: What if we still use RA-TLS to preshare the master encryption key, and then use Borys's proposed code snippet and add core-Gramine changes to enable UDSes between independent Gramine enclaves (via seamless TLS-PSK sessions).

• Borys: but there still must be the "main" Gramine enclave that distributes the master encryption key, otherwise three enclaves connecting to the same UDS path will use different keys.
• Dmitrii: we assume that only two enclaves communicate on one UDS path.