search

Optum / dce

Disposable Cloud Environment
Apache License 2.0
304 stars 80 forks source link

Budget limit not working for more than 25 accounts #482

Open victormachado-ada-tech opened 3 months ago

victormachado-ada-tech commented 3 months ago

Version information DCE: v0.34.1 OS: aws/codebuild/amazonlinux2-x86_64-standard:4.0 Go: 1.18 Terraform: 0.12.31

Describe the bug I followed the setup of Sandbox Accounts for Events. For some reason the fan_out_update_lease_status lambda only triggers for 25 leased accounts. In practice, most of our leased accounts are not being automatically deactivated after the expiration date or budget limit.

To Reproduce

  1. Lease 50 accounts with some budget.
  2. Wait for the cloudwatch logs of fan_out_update_lease_status lambda to find an account that does not show up on the logs.
  3. On one of those accounts, allocate some resource that would cost over the budget.
  4. The account is not deactivated.

Expected behavior The account should be deactivated after spending more than the provided budget.

Additional context Example logs (at this time there were actually 80 active leases):

2024-03-07T17:45:13.385-03:00   INIT_START Runtime Version: go:1.v26 Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:30052276b0b7733e82eddf1f0942de1022c7dfbc0ca93cfc121c868194868dec
    2024-03-07T17:45:13.551-03:00   2024/03/07 20:45:13 Using AWS region "us-east-1" to create session...
    2024-03-07T17:45:13.552-03:00   2024/03/07 20:45:13 Already added SQS service
    2024-03-07T17:45:13.553-03:00   2024/03/07 20:45:13 Already added SNS service
    2024-03-07T17:45:13.553-03:00   2024/03/07 20:45:13 Already added CloudWatch Events service
    2024-03-07T17:45:13.553-03:00   2024/03/07 20:45:13 Already added Eventer service
    2024-03-07T17:45:13.553-03:00   2024/03/07 20:45:13 Already added DynamoDB service
    2024-03-07T17:45:13.556-03:00   START RequestId: 2ec2e089-59ab-49d9-b3f9-9f2a50e4036f Version: $LATEST
    2024-03-07T17:45:14.202-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease idMRqdxAzL__xxxxxxxx+com @ 7071xxxxxxxx
    2024-03-07T17:45:14.349-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease sZ3cXNQSwO__xxxxxxxx+com @ 1184xxxxxxxx
    2024-03-07T17:45:14.370-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease idMRqdxAzL__xxxxxxxx+com @ 7765xxxxxxxx
    2024-03-07T17:45:14.394-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease idMRqdxAzL__xxxxxxxx+com @ 0376xxxxxxxx
    2024-03-07T17:45:14.425-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease 0STGWTTwnh__xxxxxxxx+com @ 4373xxxxxxxx
    2024-03-07T17:45:14.462-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease idMRqdxAzL__xxxxxxxx+com @ 2230xxxxxxxx
    2024-03-07T17:45:14.482-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease sZ3cXNQSwO__xxxxxxxx+com @ 3761xxxxxxxx
    2024-03-07T17:45:14.505-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease idMRqdxAzL__xxxxxxxx+com @ 5435xxxxxxxx
    2024-03-07T17:45:14.532-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease sZ3cXNQSwO__xxxxxxxx+com @ 8427xxxxxxxx
    2024-03-07T17:45:14.569-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease 0STGWTTwnh__xxxxxxxx+com @ 1275xxxxxxxx
    2024-03-07T17:45:14.590-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease 0STGWTTwnh__xxxxxxxx+com @ 7004xxxxxxxx
    2024-03-07T17:45:14.611-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease 8svhDEntzB__xxxxxxxx+com @ 1741xxxxxxxx
    2024-03-07T17:45:14.638-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease sZ3cXNQSwO__xxxxxxxx+com @ 3941xxxxxxxx
    2024-03-07T17:45:14.766-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease 8svhDEntzB__xxxxxxxx+com @ 0644xxxxxxxx
    2024-03-07T17:45:14.782-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease 0STGWTTwnh__xxxxxxxx+com @ 3227xxxxxxxx
    2024-03-07T17:45:14.803-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease idMRqdxAzL__xxxxxxxx+com @ 5783xxxxxxxx
    2024-03-07T17:45:14.819-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease 8svhDEntzB__xxxxxxxx+com @ 4148xxxxxxxx
    2024-03-07T17:45:14.843-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease 0STGWTTwnh__xxxxxxxx+com @ 3772xxxxxxxx
    2024-03-07T17:45:14.865-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease idMRqdxAzL__xxxxxxxx+com @ 2919xxxxxxxx
    2024-03-07T17:45:14.882-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease sZ3cXNQSwO__xxxxxxxx+com @ 8874xxxxxxxx
    2024-03-07T17:45:14.906-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease 0STGWTTwnh__xxxxxxxx+com @ 8796xxxxxxxx
    2024-03-07T17:45:14.928-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease idMRqdxAzL__xxxxxxxx+com @ 1643xxxxxxxx
    2024-03-07T17:45:14.948-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease idMRqdxAzL__xxxxxxxx+com @ 1025xxxxxxxx
    2024-03-07T17:45:14.972-03:00   2024/03/07 20:45:14 Invoking lambda update_lease_status-dce with lease idMRqdxAzL__xxxxxxxx+com @ 8958xxxxxxxx
    2024-03-07T17:45:15.000-03:00   2024/03/07 20:45:15 Invoking lambda update_lease_status-dce with lease idMRqdxAzL__xxxxxxxx+com @ 7341xxxxxxxx
    2024-03-07T17:45:15.030-03:00   END RequestId: 2ec2e089-59ab-49d9-b3f9-9f2a50e4036f
    2024-03-07T17:45:15.030-03:00   REPORT RequestId: 2ec2e089-59ab-49d9-b3f9-9f2a50e4036f Duration: 1473.98 ms Billed Duration: 1474 ms Memory Size: 128 MB Max Memory Used: 55 MB Init Duration: 170.28 ms
victormachado-ada-tech commented 3 months ago

So by further investigation, I suspect there is a problem on the pagination script.

On this file, they set a default Limit of 25, which explains why I was getting only 25 leases on the logs.

And here is the function that is called to list the leases.

But the problem is not the Limit itself, but the pagination algorithm which is not working. I would be glad if someone could fix this.

Meanwhile, if someone come across the same problem, I could workaround it by setting and environment variable LIMIT=100 on the fan_out_update_lease_status lambda. But it might not work for lots of leases because DynamoDB has a limit of 1MB read per query.