search

aws / aws-nitro-enclaves-cli

Tooling for Nitro Enclave Management
Apache License 2.0
127 stars 81 forks source link

Maximum resources possible for allocation #263

Open mdamgaard-biolib opened 3 years ago

mdamgaard-biolib commented 3 years ago

I have searched through the documentation and made some experiments. Yet i have not figured out what the maximum amount of vCPUs and memory i could allocate to an enclave is.

Have you done these test at some point and reached a value or is it more like above some point behaviour becomes "undefined"?

For example: An r5.24xlarge ec2 instance boasts 96vCPUs and 728GB RAM, so i was wondering how many of those resources i could realistically use in an enclave.

exFalso commented 2 years ago

I'd recommend looking at the nitro-cli-config logic for allocating enclave CPUs, in particular the following function and its comment: https://github.com/aws/aws-nitro-enclaves-cli/blob/8f6ed740b05225512d86163f8b02292668c4b056/bootstrap/nitro-cli-config#L544-L549

So, vcpu 0 must be retained for host usage + one whole NUMA must be allocated(and at most one) for the enclave. For smaller VMs I have tried there have been 2 NUMAs, and the vCPUs are evenly distributed between them. This means that at most half of the overall vCPUs can be allocated to enclave usage.

To check the vCPU - NUMA assignment you can use lscpu -p=cpu,node. Note that this will only display online CPUs. Once CPUs are allocated to an enclave they are brought offline, so they won't be displayed.

lx3-g commented 1 year ago

For r5.24xlarge ec2 instance that has 728GB RAM, when in /etc/nitro_enclaves/allocator.yaml I set memory_mib to 367572 (a little less than half the RAM) and then run systemctl start nitro-enclaves-allocator.service It works fine. When I set it to 397572 (a little over the half), it fails with the error below. What is the percentage of the overall memory that the enclave machine can use? Is there a hard limit, or is it always percentage?

● nitro-enclaves-allocator.service - Nitro Enclaves Resource Allocator
   Loaded: loaded (/usr/lib/systemd/system/nitro-enclaves-allocator.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2023-10-30 17:00:43 UTC; 11s ago
  Process: 19705 ExecStart=/usr/bin/nitro-enclaves-allocator (code=exited, status=1/FAILURE)
 Main PID: 19705 (code=exited, status=1/FAILURE)

Oct 30 17:00:30 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: Will try to reserve 667572 MB of memory on node 1.
Oct 30 17:00:30 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: Configuring the huge page memory...
Oct 30 17:00:34 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: - Reserved 368 pages of type: 1048576kB.
Oct 30 17:00:34 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: - Reserved 2677 pages of type: 2048kB.
Oct 30 17:00:34 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: Memory configuration failed, rolling back memory reservations...
Oct 30 17:00:43 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: Error: Failed to configure entire amount of requested memory. This indicates insufficient system resources.
Oct 30 17:00:43 ip-10-0-235-54.ec2.internal systemd[1]: nitro-enclaves-allocator.service: main process exited, code=exited, status=1/FAILURE
Oct 30 17:00:43 ip-10-0-235-54.ec2.internal systemd[1]: Failed to start Nitro Enclaves Resource Allocator.
Oct 30 17:00:43 ip-10-0-235-54.ec2.internal systemd[1]: Unit nitro-enclaves-allocator.service entered failed state.
Oct 30 17:00:43 ip-10-0-235-54.ec2.internal systemd[1]: nitro-enclaves-allocator.service failed.