task queue support

[kiahmed]

Not really an issue with code, but need some design guideline on function features. I have a requirement where I have to pass data from firebase storage and realtime db to an on premise process to finish the job. I wanted to implement a pull task queue in between for better asynchronous and traffic load distribution as well. How can a firebase function call a task queue and vice versa? Any sample would be really helpful

How to reproduce these conditions

Sample name or URL where you found the bug

Failing Function code used (including require/import commands at the top)

Steps to set up and reproduce

Sample data pasted or attached as JSON (not an image)

Security rules used

Debug output

Errors in the console logs

Screenshots

Expected behavior

Actual behavior

[prescottprue]

@kiahmed I haven't really needed to "queue" jobs persay since functions handle automatically scalling, but we have setup request/response collections to setup a "microservice" pattern for all of our functions. That way we can tell when a specific request is in progress or was completed.

Part of it was based on the patterns set up by firebase-queue.

[kiahmed]

@prescottprue thanks for chiming in. Well, here is my situation I was thnking like scenario 1, I can't really do any logic processing or task management in firebase simply those are out of scope in my case and just need the resources from db and storage to create a task to offload it to on premise/third party workers who has the right context to process the job. Actually if we want we can make a task worker (a particular docker container) specific. Any task can't be handled by any worker, a task is targeted for particular worker. So, that given. I want a better way to offload the tasks from firebase to the docker cluster so the fireside does't try to bloat managing queue and handing req/response for each worker while serving other requests from the app and spread the load pretty evenly across the entire system.

But to your point (scenario 2) if I use firebase queue library and as firebase auto scales , it shouldn't be problem. I am new to firebase also never used queue there..how does it performs under stress? Did you have any issue with increasing traffic? Please feel free to comment on any of these..I really need a second opinion on the direction I am heading.

[inlined]

In scenario 1, why do you need both Cloud Pub/Sub and GAE task queues? Also, why do you need both Cloud Function sand a Docker swarm? Is the latter because you've already invested in that infrastructure?

[kiahmed]

well, workers are 3rd party pre-packed binary processes that simply can't run on firebase which is also customer specific. Kubernetes/Docker cluster would be perfect model for that. Since we already have docker infra setup, so I thought that would be a no brainier to put'em there.

As far as both pub/sub and queue, we have primarily 2 major categories of data. One coming from existing user activities which pubsup can spit around whoever subscribe to that topic, other is new user on boarding which would require a container provisioning on demand and can grow at a fast rate. So I was thinking to put the later one in a queue and some process on premise would take care of that and notify when its done. I am little conflicted here, may be I can do it through the pubsub too with just another topic but then inside firebase I need to hold the requests in a queue.

[inlined]

If you don't want to carry data through the Firebase Database (which may not be the correct use case if the data is transient), Cloud Pub/Sub scales very well. You can also use Cloud Functions to respond to Pub/Sub messages or user creation events. GCF is designed for the auto-scaling case and might save you a good amount of money over kubernetes with a fixed VM pool if your workload is really that volatile.

[samtstern]

This is a very old (but good) thread so I am going to close the issue.

It's also worth noting that GCP launched a Task Queue product since this was posted: https://cloud.google.com/tasks/