Improve Job management via "Full duplex" Queue

[g41797]

FOR BRAINSTORMING

Target:

• provide more information to end-user about job status
• add possibility to cancel job

How to achieve:

JobStatus contains:

• id
• state (unknown, waiting, running, done, cancelled, failed....)
• process indicator (sequential number)
• process indicator [0.0 - 100.0]
• fail reason
• cancel initiator (user, ...)
• additional info (e.g. PID for cancel, timestamps, display info , ...)

JobStatus is Message Job Handler periodically (within processing loop) pushes JobStatus via Queue to Producer.

JobCancel is Message Monitoring app. pushes JobCancel to Queue, on the other side QueueLoop (implements LoopInterface) receives it ...

Function "run Job" moved from Adapter to Queue.

Flow is generic, does not depend on specific functionality of Adapter.

[g41797]

@xepozz, emoji is great example of current JobStatus - github adapter does not support notification for emoji, lack of information what happened and as result what should be improved. Could you please provide "additional information"?

[xepozz]

@g41797 I just read I didn't get the reason to have so flexible status checking as you wrote. I have some different goals what to do now before the package release and seems your idea doesn't suites the goals.

Also seems quite hard to implement progress/cancellation with additional jobs. Just imagine 100 RPS makes 100 * (amount of progress check messages) tasks. I'm not ready to discuss it now.

[samdark]

That's interesting since some queues do not provide any way to peek inside. But, as @xepozz noted, likely that's extra workload that will slow down queue processing.

[g41797]

First of all we need to understand whether

• end user wants this functionality (web page with updated status of running jobs + possibility to cancel...)
• it already supported by your famous friends Laravel and Symfony

My bet - answers are yes&yes

Do you have possibility to ask Yii users about this use case?

Regarding slow down processing - it's usual answer of busy developer I did it a lot in my career

[samdark]

My bet - answers are yes&yes

Of course, that's nice to have. But it could be implemented in multiple ways:

1. By introducing interfaces for it and each "driver" implementing it. In some queue engines that is supported by the engine itself so no extra workload will be needed. In some engines it's not supported so a "driver" will have to write extra data somewhere (see 2).
2. By always writing extra data i.e. keeping our own jobs registry along with statuses and IDs regardless of the "driver".

[g41797]

Agree But what is the flow from end-user point of the view? e.g.:

• Web page for submitting jobs
• Web page for monitoring
• Background workers invisible for user

Immediately raised question - "Who allowed to monitor jobs?" any user, admin etc

Or how many simultaneous monitoring apps/pages should be supported

There isn't a problem to define interfaces etc Main problem - flow definition and user reqs

What is Yii way to create "application definition "?

[samdark]

Web page for submitting jobs

No, definitely not. Jobs are submitted by code, not manually.

Web page for monitoring

Yes, could be part of https://github.com/yiisoft/yii-dev-panel

Background workers invisible for user

Yes.

"Who allowed to monitor jobs?" any user, admin etc

The one having access to dev panel. End user usually doesn't need to know about that but if we have an API for getting a job status, one might use it to inform user.

Or how many simultaneous monitoring apps/pages should be supported

Endless number.

[g41797]

Possible variants for storing of job statuses pushed by workers to queue:

1. user application receives JobStatuses via Queue , saves in user DB, panel periodically polling DB
2. Queue saves statuses in internal DB(jobs registry), panel periodically polling DB via queue api
3. Queue saves statuses in broker(it acts as jobs registry) - panel receives statuses via queue api

Using db (as jobs registry) or broker storage should be part of driver (transparent to queue)

db based jobs registry support will be standard part of yii framework itself it injected to driver in run-time

We will have following nested blocks:

• queue (common code) includes job processing removed from driver/adapter
  • drivers (broker specific code)
  • ?db job registry (common code)
    • ?db "drivers" (db specific code) similar to queue-adapter architecture

For end-user developers will be supplied standard docker-compose with yii, db and broker

Your turn

[samdark]

@viktorprogger, @s1lver thoughts?

[g41797]

adding insult to injury

In order to acts as "jobs registry" broker should support Event Sourcing (I learned it today) Usually it means that

• broker should support Streams
• PHP client library also should support Streams API

e.g.

• RabbitMQ supports Streams
• php-amqplib supports Streams, but not Streams API

as result - [100-1000???]RPS (clients of StreamAPI - 1000000RPS)

[g41797]

no thoughts no glads

issued #194 instead - for go i can start it today

[samdark]

https://github.com/yiisoft/yii2-queue/pull/510