Laravel queue driver

[barryvdh]

So this might be out-of-scope, but with the bootstrap method etc, I was thinking of a Laravel queue driver. Where each job is just the event basically (as jobs are stored as json mostly anyways)

• Bootstrap start the queue worker (passthru("cd app && /opt/bin/php -c/opt/php.ini artisan queue:work lambda");)
  • In the application code, jobs are dispatched by invoking the function with the payload in the event (eg dispatch(new SomeJob)->onQueue('lambda');), using the aws php API.
• Queue worker on Lambda is responsible for popping jobs, which is just calling the Lambda API endpoint to get a new event
  • After job is done, output is sent to the Success/Fail endpoints

So the queue worker should always be idling until a job is incoming, which should work fast (eg. no need to Bootstrap laravel/start CLI command). And Laravel has methods for checking for memory etc + easy dispatch.

[mnapoli]

Would that make more sense to use SQS as a queue driver? That way we have durability, dead letter queues, logging, monitoring, etc.

[pmayet]

+1 @mnapoli with the trigger SQS/Lambda

[barryvdh]

But that would trigger a new process, not using the same Laravel queue loop right?

[pmayet]

Exactly !

For SQS with JSON payload : https://github.com/dusterio/laravel-plain-sqs

[mnapoli]

And that makes much more sense to me. Maybe there is some misunderstanding here. Here is how it works with SQS (though I haven't tried it yet so…):

• Laravel HTTP app ("function A") sends job message to SQS
• SQS triggers another lambda ("function B") that receives the message in the $event variable -> it should process it

The Laravel queue loop doesn't make sense on Lambda since the code is executed on demand (when there is a message in SQS).

[mnapoli]

Actually see https://github.com/mnapoli/bref/issues/109#issuecomment-449310816 for a (hacky) way to do it.

Maybe there is some "adapter" to do for Laravel to ease running one job in a lambda.

ping @thiagomarini

[deleugpn]

I think mnapoli suggestion is on the right track here. AFAIK, you cannot rely on the event loop anymore. Even if you manage to get it working, it will probably be expensive as you wouldn't pay in batches of 100ms and it would still shutdown when the lambda max time reached. Having an SQS triggering a Lambda that runs the code would be highly scalable and easy to manage. Each job get it's own instance of the lambda and finishes. One thing I thought that could work with bref 0.2 is this:

• Web App pushes to SQS
• SQS triggers lambda but not as a worker, just as an event. The message is still in the queue.
• Lambda runs php artisan queue:work --once
• The message gets picked, worked and finishes.

[barryvdh]

Yes that would probably be more reliable. But the event loop could just keep waiting for a new event, in the meantime it's frozen by aws. That would avoid the bootstrapping if Laravel itself and keep the application and MySQL alive in the process.

[mnapoli]

Keep in mind that when SQS invokes a lambda it can send multiple jobs, not just one.

I think we need to let go the artisan command entirely. I'll be speaking about Bref 0.3 only:

• we can use the PHP base layer
• then we can provide either a layer or encourage users to write a custom bootstrap file to override the base bootstrap
• this bootstrap would take the jobs from the event data, boot Laravel and make Laravel process all the jobs
• it would then loop and wait for the next invocation

Booting Laravel once could be done, but that means shared global state, memory leaks, etc. Also maintaining the db connection isn't that great because of timeouts and because it eats the connection pool fairly quickly. Instead MySQL connections need to be closed/reopened every time.

Anyway this is an implementation detail that only affects performances, and here performances will be quite great because of how scalable Lambda is: as soon as there are messages in the queue there will be lambda booted to process the jobs. I'm not too worried about that.

[barryvdh]

Yeah okay, but connection/state management is what Laravel already does when running it as a normal worker deamon, so that shouldn't be a problem. But I guess reliability and ease of use should be preferred over performance indeed.

[deleugpn]

Just to fully contextualize your answer @mnapoli, what you seem to be describing is:

• A message lands on SQS Queue (doesn't matter how the message gets pushed at this point)
• SQS triggers a Lambda execution (not as a lambda worker) so the messages are still there
• The bootstrap calls the Lambda API which has an event waiting
• It bootstraps Laravel and run a custom script that would pull messages from the queue and work them.

If this is what you mean, I see one caveat. Imagine the App is pushing 12 messages to the Queue. That would trigger 12 events for Lambda. We don't know how many containers Amazon would start for that, but we don't care either. Let's pretend Amazon starts 3 containers and each handle their 1st event. At this point we have 3 workers that will start pulling messages from the queue and will only stop once the queue is empty. Now the queue is empty but we still have 9 Lambda Events. The 3 bootstrapped containers will run 3 Lambda events each that will bootstrap Laravel and ask to work an empty queue, which could be quite wasteful.

There's also the scenario where a Lambda worker starts and in the 15 minutes limite the queue doesn't get cleared, which could lead to a job being killed mid-process. We know it will be retried, so it's not the end of the world, but I think it's nice to have that in mind.

[mnapoli]

@deleugpn this is correct until this point:

• It bootstraps Laravel and run a custom script that would pull messages from the queue and work them.

Instead the information about the SQS messages (the 12 of them, or maybe just 3, or any number) are contained in the $event variable.

So what should happen instead is that the bootstrap file of the lambda application should boot Laravel and process all the messages that were sent in the $event variable.

Once that it's done the bootstrap can then loop and ask for the next event to process.

[deleugpn]

I see, okay, so in that case I believe this is wrong:

• ~SQS triggers a Lambda execution (not as a lambda worker) so the messages are still there~

If I'm not mistaken, what we want here is an SQS that triggers a Lambda Worker, meaning each message gets automatically worked by a Lambda, so Laravel doesn't pull the Queue. Instead the message gets injected into the Lambda and automatically consumed.

Result:

• A message lands on SQS Queue (doesn't matter how the message gets pushed at this point)
• SQS triggers a Lambda Worker
• The bootstrap calls the Lambda API which has an event waiting containing the Message.
• It bootstraps Laravel and processes the Message.

[mnapoli]

Yes, this is the good part: AWS takes care of distributing the messages for us. We "just" have to process them when they arrive. We don't poll anything anymore, which is good.

[barryvdh]

It's probably very similar to what I did with https://github.com/barryvdh/laravel-async-queue Which does:

• Use regular queue to store the event (eg. databse driver in my case)
• Fire an artisan command to fetch that job based on the ID passed in the commandline and process it like a regular job. (eg. php artisan queue:process-job <id 123> --queue=<queue>)

In AWS there would be and additional step between them, which is fire an AWS event. (Which SQS perhaps takes care of itself).

[mnapoli]

Which SQS perhaps takes care of itself

Yes:

• here is how to configure AWS SAM
• here is the event data when it is invoked by SQS

That's basically it. Like I said it is currently done manually, see https://github.com/mnapoli/bref/issues/109#issuecomment-449310816 We can however make it easier with a little bridge for Laravel.

[barryvdh]

Maybe we should wait a bit for Taylor to see what he comes up with :)

https://twitter.com/taylorotwell/status/1081527294420836352

I’ve solved laravel queues on lambda and to do it right requires a few core changes I’ve made internally. More to come soon. 👍

[deleugpn]

That was a hell of an exciting tweet, indeed. I'm going to try and convince my company to waste as little as possible with background workers optimization for as long as we can to see what's coming. Lambda workers will definitely be amazing.

[pmayet]

I have make some test. I have deploy a Lambda with Bref 0.3 and Lumen Application. I have a driver sqs-plain (dusterio/laravel-plain-sqs) to push job with JSON payload. I configure my Lambda to trigger event from SQS.

SQS => Lambda

You can specify for your function how mush job you want to process from 1 to 10 maximum with the BatchSize param

Events:
    QueueMessages:
    Type: SQS
    Properties:
        Queue: 'arn:aws:sqs:eu-west-1:AWS_ACCOUNT_ID:SQS_QUEUE_NAME'
        BatchSize: 1

[deleugpn]

@pmayet did you manage to get anything working at all?

I'm trying to set that up and I wrote this simple worker.php file:

<?php

require __DIR__ . '/vendor/autoload.php';

echo 'here';

lambda(function (array $event) {
    echo 'inside';

    print_r($event);

    dd('died inside');

    return;
});

dd('died outside');

And then I configured my SAM model like following:

  Worker:
    Type: AWS::Serverless::Function
    Properties:
      Role: !ImportValue LambdaExecutionRoleArn
      CodeUri: .
      Handler: ./worker.php
      Timeout: 300
      MemorySize: 1024
      Runtime: provided
      Layers:
        - arn:aws:lambda:eu-west-1:209497400698:layer:php-72:1
        - arn:aws:lambda:eu-west-1:209497400698:layer:console:1
      VpcConfig:
        SecurityGroupIds: [!ImportValue AllowAllAddressesContainerSecurityGroup]
        SubnetIds: !Split [',', !ImportValue PrivateSubnets]

  EventSourceMapping:
    Type: AWS::Lambda::EventSourceMapping
    Properties:
      BatchSize: 10
      Enabled: true
      EventSourceArn: !GetAtt [Queue, Arn]
      FunctionName: !GetAtt Worker.Arn

  Queue:
    Type: AWS::SQS::Queue
    Properties:
      VisibilityTimeout: 600
      MessageRetentionPeriod: 604800

Every time I push a message to the queue, it gets worked by lambda, but I don't see anything in the logs at all.

14:06:55
START RequestId: cfbe464a-1ec6-5499-a7c5-12916b5bf30e Version: $LATEST
14:06:55
END RequestId: cfbe464a-1ec6-5499-a7c5-12916b5bf30e
14:06:55
REPORT RequestId: cfbe464a-1ec6-5499-a7c5-12916b5bf30e  Init Duration: 189.72 ms    Duration: 41.53 ms  Billed Duration: 300 ms Memory Size: 1024 MB    Max Memory Used: 54 MB

maybe @mnapoli have any suggestions?

[deleugpn]

I got it to work by removing the console layer and only using the php layer.

[deleugpn]

After a lot of attempts, I am happy to announce I just successfully achieved this. The caveat is having to write a custom WorkCommand that extends from Illuminate\Queue\Console\WorkCommand (no big deal) and having to edit the original artisan command (a bit disappointing). The setup that I used here was 2 lambda functions deployed from the same project. Upon committing to GitHub, AWS CodePipeline fetches the code, run AWS CodeBuild to package the SAM template and deploy the lambdas.

One Lambda uses the console layer and invokes artisan by using the cli attribute on the event. Unfortunately, I could not get logging to work with this layer at all. There's never anything on CloudWatch using echo, Monolog stderr or Monolog errorlog. I guess this layer was designed to be invoked from the bref cli so anything that we echo from the invocation gets sent to the output. However, my goal is to have AWS Cloud Events to trigger this layer on a cron schedule, so having CloudWatch logs would be a big help.

The second lambda uses only the php function layer and points to artisan as it's handler. Unfortunately, we cannot have space (`) on a Handler's name, so I got creative by using an environment variable. Here's the modification I did onartisan` file:

$kernel = $app->make(Illuminate\Contracts\Console\Kernel::class);

$command = getenv('ARTISAN_COMMAND');

$status = $kernel->handle(
    $input = new Symfony\Component\Console\Input\ArgvInput($command ? ['lambda', $command] : null),
    new Symfony\Component\Console\Output\ConsoleOutput
);

This way I can set an environment variable on SAM to make this lambda automatically run php artisan queue:lambda.

Finally, the WorkLambdaCommand simply extends from the Laravel's original one and overrides the runWorker method.

    protected function runWorker($connection, $queue)
    {
        $this->worker->setCache($this->laravel['cache']->driver());

        lambda(function ($event) {
            $job = new LambdaJob($this->laravel, $event['Records'][0]);

            $this->worker->process('lambda', $job, $this->gatherWorkerOptions());
        });
    }

I implemented a custom Illuminate\Contracts\Queue\Job on the LambdaJob class and made this lambda subscribe to an SQS. When the first lambda pushes a message to the queue, the second one gets triggered automatically and run the command above. The message gets worked and the output gets sent to CloudWatch correctly (php layer).

21:01:05 START RequestId: 0e979cce-8ce8-5cae-854e-c82f9059a750 Version: $LATEST
21:01:05 [2019-02-15 20:01:05] [98698bee-df40-4865-aac7-6af8cfa21bd0] Processing: App\Console\Commands\TestJob
21:01:05 this is an echo [2019-02-15 20:01:05] production.INFO: successfully logged
21:01:05 [2019-02-15 20:01:05] [98698bee-df40-4865-aac7-6af8cfa21bd0] Processed: App\Console\Commands\TestJob
21:01:05 END RequestId: 0e979cce-8ce8-5cae-854e-c82f9059a750
21:01:05 REPORT RequestId: 0e979cce-8ce8-5cae-854e-c82f9059a750 Init Duration: 591.83 ms    Duration: 163.73 ms Billed Duration: 800 ms Memory Size: 1024 MB    Max Memory Used: 76 MB

Huge accomplishment for me, I hope to blog about this soon.

[barryvdh]

In case you missed it: https://github.com/laravel/vapor-core

[jasperf]

Would be good to have this Vapor queue part added to https://bref.sh/docs/frameworks/laravel.html . Also good to update the demo package to 6.0 as some Vapor tweaks were added to it.

I would like to try to use Bref as an alternative to Vapor or Firevel. But then we do need queue workers to work besides things like an RDS database, Redis and such.

[barryvdh]

Isn't is possible to use vapor-core in Bref? It's open source and uses a similar runtime (I think)

[mnapoli]

Let's reopen that, I want to support Symfony Messenger (already working on it) and Bref Queues natively in Bref.

I already had a look at Vapor's runtime but I don't understand why it's built the way it is (which sounds inefficient to me): it fetches messages from SQS.

This isn't the design of SQS->Lambda at all. The SQS events in Lambda already contain the jobs to execute. We don't need to use Laravel or any of its code to deal with polling, failures, retry, etc. SQS and Lambda do that for us.

I think Taylor went that way in Vapor to keep everything working with the way most of the code in Laravel Queue is currently written. It still runs with a cron AFAICT.

We can use the native AWS way instead.

[barryvdh]

I don't think it runs on a cron, but I could imagine that SQS is used as a source only, to keep the logic the same as the other queue drivers.

[mnapoli]

I'm not sure because the scheduler command still runs every minute, so why not do the same for the worker (on Vapor I mean). Or else how would it work?

[barryvdh]

The cron is for the scheduler component, usually not for the queue workers. They are normally running as a long running process. But they can still use SQS to trigger the lambda to run on a new job, but just handle it from the application itself?

[mnapoli]

Oh I see, I'm trying to picture it in my head with the concurrency and everything 🤔

Anyway I think it would be safer, and simpler, to stay with the native AWS Lambda behavior. That also allows everyone to take advantage of all tools that work with Lambda (dead letter queues, retry counts, etc.).

[grigdodon]

@mnapoli great stuff with bref! I am in a kind of desperate situation here though. The question is: can Bref interpret and process SQS messages out of the box or I need to fire the jobs manually? I am using Laravel.

[mnapoli]

@grigdodon yes this is possible (I do that with Symfony for example). There is unfortunately no documentation or library for this at the moment.

[grigdodon]

@mnapoli do you think you could give me some guidance with this? what are the mechanics? thanks!

[mnapoli]

Unfortunately I haven't had time to document anything yet, nor write the Laravel bridge (I worked only on the Symfony implementation for the moment for a client). If your company can sponsor this, you can also get in touch at contact@null.tc and we can work together to make this happen.

[christoph-kluge]

Going to join conversation here. I've worked yesterday and today on a SQS adapter for bref for laravel which acts as a snap-in replacement w/o changing anything in your config, except of adding a new service provider. Besides that no additional configuration changes are required.

Any Idea when #521 is going to be merged?

[grigdodon]

@christoph-kluge you think you could share the SQS adapter?

[christoph-kluge]

@grigdodon sure. I was just asking about #521 in order to see if it make sense to prepare a package for "old" behavior of it it makes sense to go for the new one.

// Edit#1: I will create a new repo and move my 3 pieces of code there. Will share the link here once ready.

[christoph-kluge]

@grigdodon feel free to check this package: https://github.com/christoph-kluge/bref-sqs-laravel . It should work almost out of the box after adding it to composer. You need add the artisan.php and adopt extend your serverless.yml with the environment. In case something is missing or unclear please create an issue within my repo.

[aran112000]

Hi all,

Just checking in to see if there's been any progress on an official way to process SQS events via Bref (for Laravel)?

Whilst the bref-sqs-laravel package @christoph-kluge has suggested works, it still uses the current polling behaviour of queue:work by the looks of things and we'd much rather have a way where SQS invokes lambda with a batch instead for this project.

[mnapoli]

Yes, in fact there has been progress, just a few days ago 🙂

Here is the repository: https://github.com/brefphp/laravel-bridge The package is not published on Packagist yet, I'm looking for beta testers. Any feedback you can provide is welcome.

This package does not follow the "Laravel way" with the polling queue:work command. Instead, it follows the "Lambda way" by directly deserializing and running jobs that SQS sends to Lambda.

[christoph-kluge]

Hi @aran112000 is there any special feature you're looking for?

It's not polling like a typical queue worker. The events are pushed into the lambda as a batch. You can configure the batch-size in your SQS Event Source. With the serverless framework you can just define size: 2 and from here it's taking the batch size of max. of 2 items and pushes them into the internals of laravel. In fact it looks like it's polling but it's not.

This was for me the most suitable way to reduce the maintenance level and to prevent the duplication of code. Laravel has it's internals and I think it belongs there. I didn't wanted to duplicated too much of them.

As @mnapoli mentioned correctly: there was some progress during the last days. I do plan to support typed handlers (#521) which got released with version 0.5.14.

[thtg88]

Hi @mnapoli I'd like to give the Laravel SQS Bridge a try if you still need beta testers? :-) do you have some tips on how to include it in a Laravel Bref app? Thanks

[barryvdh]

If you sponsor him, you will get early access. (2 sponsors to go before it's public) https://github.com/sponsors/mnapoli

[thtg88]

Oh I hadn't realised it was available that way! Sponsored ✅

[mnapoli]

Thank you @thtg88! I have given you access to the project and documentation.

[mnapoli]

Awesome, I reached 30 sponsors, the project is now fully open source here: https://github.com/brefphp/laravel-bridge

I have updated the Bref documentation in #668

[thtg88]

That's brilliant @mnapoli I've just tested and it works a treat! Thanks for all your efforts :)

[barryvdh]

So what is the biggest difference with how Vapor does it? This doesn't do an additional request for the message, but re-use the actual event?

[mnapoli]

@barryvdh exactly yes. It is implemented the way SQS + Lambda works, i.e. how it's done in all other languages, and how it is recommended to be implemented by AWS.

That means that any documentation or tutorial you can read online applies here.

The way I like to sum it up is: this package adapts Laravel to Lambda, instead of adapting Lambda to Laravel. I understand both approaches, but for infrastructure-related problems, I think it's best to align with AWS.