brefphp / bref

Serverless PHP on AWS Lambda
https://bref.sh
MIT License
3.05k stars 364 forks source link

Long running CLI commands repeat every 2 minutes #1775

Closed duncan-c closed 1 month ago

duncan-c commented 3 months ago

Description:

We're using Bref with Laravel, and we have some artisan commands which take some time to complete (usually database operations).

We've seen that after a command has been running for two minutes it is triggered again (but the original one continues running), and then after another two minutes, it's triggered again, and so on, etc. This can continue infinitely, and can cause some real problems if the commands are doing database manipulations which then conflict (for example I just added a lot of duplicate data into production!).

As soon as the call to serverless bref:cli is manually stopped (or an error causes it to stop) the commands stop repeating.

How to reproduce:

Create an artisan command which just waits 5 minutes and stops:

$this->info('Command started!');
for ($i = 0; $i < 5; $i++) {
    sleep(60); // Wait for 1 minute
    $this->info(($i+1).' minute(s) have passed.');
}
$this->info('Command completed!');

Then deploy and run the command: serverless bref:cli --args="wait-5-mins"

The command will never complete itself, and the Lambda logs will show the command starting every 2 minutes:

image

Bref Version: 2.1.16

duncan-c commented 1 month ago

Turns out this is because the AWS CLI timeout is 2 minutes by default. Setting a larger (or 0) value in the AWS_CLIENT_TIMEOUT environment variable fixes this issue.

https://github.com/serverless/serverless/issues/10935