[Bug] Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its minimum value of [x]

[rams3sh]

I have been trying to run matano in a fresh personal AWS account after having it tried it an another account with extended lambda limits to see if there exists any additional configuration / request for quota increase. I hit upon this error with matano.

Details below.

Version : matano/0.0.0 linux-x64 node-v14.18.1 Note: This is the nightly build as of today.

Snippet Error from Terminal Below :

rams3sh@monastery:~/Garage/matano$ matano init
━━━ Matano: Get started Wizard ━━━

Welcome to the Matano init wizard. This will get you started with Matano.
Follow the prompts to get started. You can always change these values later.

✔ Which AWS Region to deploy to? · us-east-1
✔ What is the AWS Account ID to deploy to? · XXXXXXXXXXXXX
✔ Do you have an existing matano directory? (y/N) · false
  I will generate a Matano directory in the current directory.
✔ What is the name of the directory to generate?(use . for current directory) · .
✔ Generated Matano directory at /home/rams3sh/Garage/matano.
✔ Successfully initialized your account.
⠦ Now deploying Matano to your AWS account... 
›   Error: An error occurred: Command failed with exit code 1: /usr/local/matano-cli/cdk deploy DPMainStack --require-approval never --app /usr/local/matano-cli/matano-cdk 
...

 ›   Failed resources:
 ›   MatanoDPMainStack | 7:51:57 PM | CREATE_FAILED        | AWS::Lambda::Function            | DPMainStack/LakeWriter/AlertsFunction (LakeWriterAlertsFunctionCB567D9B) 
 ›   Resource handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its minimum value of [50].
 ›    (Service: Lambda, Status Code: 400, Request ID: c990af9b-a3e6-4328-a3c7-4f0b01967c4f)" (RequestToken: b6eb4fad-441b-2493-86c5-5c29b6969a6f, HandlerErrorCode: 
 ›   InvalidRequest)
 ›   
 ›    ❌  DPMainStack (MatanoDPMainStack) failed: Error: The stack named MatanoDPMainStack failed creation, it may need to be manually deleted from the AWS console: 
 ›   ROLLBACK_COMPLETE: Resource handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its 
 ›   minimum value of [50]. (Service: Lambda, Status Code: 400, Request ID: c990af9b-a3e6-4328-a3c7-4f0b01967c4f)" (RequestToken: b6eb4fad-441b-2493-86c5-5c29b6969a6f, 
 ›   HandlerErrorCode: InvalidRequest)
 ›       at FullCloudFormationDeployment.monitorDeployment (/snapshot/node_modules/aws-cdk/lib/api/deploy-stack.ts:505:13)
 ›       at runMicrotasks (<anonymous>)
 ›       at processTicksAndRejections (internal/process/task_queues.js:95:5)
 ›       at deployStack2 (/snapshot/node_modules/aws-cdk/lib/cdk-toolkit.ts:265:24)
 ›       at /snapshot/node_modules/aws-cdk/lib/deploy.ts:39:11
 ›       at run (/snapshot/node_modules/p-queue/dist/index.js:163:29)
 ›   
 ›    ❌ Deployment failed: Error: Stack Deployments Failed: Error: The stack named MatanoDPMainStack failed creation, it may need to be manually deleted from the AWS console:
 ›    ROLLBACK_COMPLETE: Resource handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its 
 ›   minimum value of [50]. (Service: Lambda, Status Code: 400, Request ID: c990af9b-a3e6-4328-a3c7-4f0b01967c4f)" (RequestToken: b6eb4fad-441b-2493-86c5-5c29b6969a6f, 
 ›   HandlerErrorCode: InvalidRequest)
 ›       at deployStacks (/snapshot/node_modules/aws-cdk/lib/deploy.ts:61:11)
 ›       at runMicrotasks (<anonymous>)
 ›       at processTicksAndRejections (internal/process/task_queues.js:95:5)
 ›       at CdkToolkit.deploy (/snapshot/node_modules/aws-cdk/lib/cdk-toolkit.ts:339:7)
 ›       at initCommandLine (/snapshot/node_modules/aws-cdk/lib/cli.ts:374:12)
 ›
 ›   Stack Deployments Failed: Error: The stack named MatanoDPMainStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource 
 ›   handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its minimum value of [50]. 
 ›   (Service: Lambda, Status Code: 400, Request ID: c990af9b-a3e6-4328-a3c7-4f0b01967c4f)" (RequestToken: b6eb4fad-441b-2493-86c5-5c29b6969a6f, HandlerErrorCode: 
 ›   InvalidRequest)
 ›   Created temporary directory for configuration files: /tmp/mtnconfigv9yADs/config
 ›   arn:aws:cloudformation:us-east-1:XXXXXXXXXX:stack/MatanoDPCommonStack/cebd94d0-7e14-11ed-9855-0e5a30013c2f

Lambda Quotas :

rams3sh@monastery:~/Garage/matano$ aws lambda get-account-settings
{
    "AccountLimit": {
        "TotalCodeSize": 80530636800,
        "CodeSizeUnzipped": 262144000,
        "CodeSizeZipped": 52428800,
        "ConcurrentExecutions": 50,
        "UnreservedConcurrentExecutions": 50
    },
    "AccountUsage": {
        "TotalCodeSize": 1337,
        "FunctionCount": 1
    }
}

Please let me know how to proceed from here.

Also, should I have to mandatory increase the lambda quota since it has a separate pricing ? Can there be any option not to have this concurrency enabled as part of matano deployment ? This will be helpful for experimentation use cases such as the current scenario like mine where I don't expect to have production scale events.

Further, such cases (in general) can be part of some kind of cli argument where the user has an option to explicitly disable such recommended production settings which may not be required for a staging / experimentation.

[Samrose-Ahmed]

Thanks for opening an issue. This is a known issue (should be documented).

Yes, unfortunately Lambda sets a very low default account concurrency. They will quickly increase your quota upon request.

Can you clarify what you mean by additional pricing? There is no cost to having your account concurrency increased and we are using reserved concurrency which is free (unlike provisioned concurrency which has a cost).

As far as disabling concurrency, we would have to be careful as we actually use reserved concurrency to ensure a specific function is single threaded to avoid concurrency issues with Glue and Iceberg. If concurrency was disabled, depending on the scale of data, it could result in errors since concurrent executions could result in too many concurrent updates on a Glue Table.

[rams3sh]

Thanks for getting back quick.

Can you clarify what you mean by additional pricing? There is no cost to having your account concurrency increased and we are using reserved concurrency which is free (unlike provisioned concurrency which has a cost).

Sorry for my ignorance. I typed it out of my head as I was thinking of provisioned concurrency while the error was actually about reserved concurrency.

My issue / feature to disable does not make much sense now as there is no additional pricing as you said for reserved concurrency. Also can you let me know what would be the minimum number that is to be requested for increasing the reserved concurrency quota for deploying matano.

It would be also great if this is documented as part of installation steps.

[Samrose-Ahmed]

No problem, they are easy to confuse :)

It will technically be sufficient to request 51 or 100, but you can probably just request 1000 since that's the default per the AWS docs (ref), new accounts just have a lowered concurrency.

I'll make sure to update the docs.

[rams3sh]

Thanks.

Also when you document , kindly include the below details that your referred link suggests to be put as part of the support case.

This will help others to use it as is (as a template request) and raise a case instead of figuring out justification by themselves for experimentation / initial deployment as these details are something that a user cannot determine before he / she even deploys and tests matano.

    Concurrency limit requested
    Anticipated average number of requests that your function will receive per second
    Anticipated highest number of requests that your function will receive per second
    Anticipated runtime duration
    Function memory size
    Invocation type (event or request-response)
    Event source
    Load test results that explain the reason why the quota increase is needed
    Any additional information that will help us better understand your use case

[shaeqahmed]

Thank you so much Ram for the bug report! Let's add @rams3sh for the contributions. cc @all-contributors