[Feature] Implement pub/sub event interface for remotely managing the DAG and it's element's life cycle

[blaggacao]

Coming from the wish to spec job objects according to their metadata for fit resource consumption and gain more advanced operational insights (like monitoring), I want to offload the execution to a more powerful system (than python workers).

Therefore, implement a executor back-end listening on:

• Metadata, sufficient to reason about job object creation and resource provisioning
• Scheduling Events, to start a job object execution
• Mutating pre-execution job objects? (not sure if that's consistent with the channel implementation or if created job objects are immutable, which would make sense, indeed. Haven't checked.)

[blaggacao]

Possible candidates:

• https://airflow.apache.org/
• https://github.com/spotify/luigi

@guewen Is this something for you to consider on the longer run?

That would OCA/queue make a solid emerging standard job interface with batteries included but swapable. Currently, it's role as a job interface is diminished by having the implementation bundled tightly.

[blaggacao]

While mediator is thinkable, ideally, a well defined interface + (backend specific) wrapper would be preferable.

[lmignon]

I really like the current implementation! No dependency on external tools, transnational, scalable, ....

[blaggacao]

@lmignon Yes, of course, you are right with you're opinion. Actually, my intention is not to propose any external dependency, here.

Rather, what I try to propose, is a well defined interface for Odoo specific jobs that can be implemented by any back end, while still providing the current implementation as a default back end (batteries included, but swapable). This has two advantages:

• This module gets more reusable (well designed and generic interface definition)
• Some x-fertilization from other job queue implementation would spill back naturally and over time onto the development of this module (the interface)

Finally, it could, then, rightfully claim to be the canonical job implementation for Odoo. After all code re-usability is a proclaimed OCA goal.

At this stage, I just want to understand how fertile the ground would be for such idea. I think OCA/queue has huge potential, and I strongly believe inter-operability would further leverage that potential.

[guewen]

@blaggacao in the very first implementation, I wrote some abstractions to make the job execution swappable, in the case we would like to use another queue, etc. It was of course not perfect since the use case did not really exist. Throughout the evolution of the code, at some point we decided to remove the abstractions because they made the code more complex, nobody ever used them and by using an odoo/postgres-only queue and executor, we could implement some features that would maybe not work with an external backend (I don't have them in head but I remember having to make this choice ;)).

Not saying it cannot be done, but only YAGNI; if the need arrives, and we have no choice but to implement a different back-end, then I guess we'll first have some budget to work on it, second, a real use case and real implementation is required to do the correct choices in term of interfaces (I would even say we would need to test the implementation of 3 backends to be sure of the interface).

[blaggacao]

@theangryangel I'm braking a lance here for you're idea to use airflow/luigi. I guess it's interesting what @guewen reports about the choices he had to make

For sure, anything more odoo domain specific would have to have access to an odoo registry passed around, so if scaling / resource constraining the job queue individually is a requirement, then those individually deployed runner need to bundle an odoo registry (@guewen I guess that's the main choice you had to make).

But for tasks that live more in the space between Odoo and X system, airflow/luigi/sydnesis/etc. might be the better candidate. Would be interesting to see if any integration can be done so that both job domains can interact and inter-depend. I can imagine that airflow's DAG would have some interdependencies with Odoo domain specific jobs, after all.

[guewen]

One of the best properties of the current system is the fact that jobs are in the same transaction. The transaction is rollbacked, they are as well. If we wanted to create the jobs in a external queue, we would "modify the external world" that the transaction would not be able to rollback, then you have a whole new category of problems.

A common pattern to avoid this issue is to first create the jobs in a DB table, commit. Then a second transaction pushes them to the external job queue. So my opinion is that, if we needed to work on such bridge, the interface would be... the queue_job table.

• Odoo would create jobs there
• jobrunner should not be started
• an export from queue_job to the external queue_job should be implemented (either in direct after a NOTIFY or in small and frequent batches)
• then the heavy work is on the executor I guess

[theangryangel]

Yesterday I was doing some brief, very simple experiments with OCA/queue, airflow, etc. just loosely tying them together (effectively just making a delayed job signal to airflow to run a DAG on demand; not real integration as @blaggacao has suggested). I wanted to get a feel for what things might be like, what problems I might see that I hadn't considered yet, etc.

I think I'm starting to see the "category of problems" you've alluded @guewen and I'm not sure how I feel about the approach - mostly because of the time constraints I've got.

The thing that attracted me to airflow specifically was;

• the DAG style is very similar to an existing workflow the end users are familiar with
• the existing operators in airflow are extremely useful (a lot of what I'm looking at doing involves signaling to 3rd parties via FTP/SFTP rather than web APIs)
• by separating the presentation of said files from an Odoo module, deploying additional third parties with slightly different presentations is attractively easy and would not affect Odoo directly (I often cannot control the format of the files); however that may say more about the deployment strategy of Odoo we have and perhaps I'm looking at it the wrong way

@blaggacao At the end of the day I'm starting to think that OCA/connector, OCA/queue is probably more than good enough for now (the benefits of deep integration with Odoo, without a lot of work, etc.) may be more appropriate. I'll ping you on Skype later to discuss further?

[blaggacao]

@theangryangel could be true. The decoupling of lifecycles (and deployments) you mentioned, from what I've seen in patterns out there, is probably the strongest argument for offloading, indeed.

[blaggacao]

@guewen As a result of the discussion with @theangryangel we have considered a different option, mainly to achieve life cycle decoupling, which I want to share with you (not validated many assumptions, yet):

• Define a well-designed REST API (OCA/rest)
• Make that API the read "back end" to OCA/connector
• Implement CUD (of CRUD) back end as web hook triggers tightly coupled with a specific airflow implementation
• Implement airflow pipelines which recollect & aggregate / write data from/to the various (archaic) transport mechanisms and do transformations towards the Odoo REST API (without need for additional Odoo domain knowledge)

This way:

• The myriad of archaic systems can evolve independently without need for Odoo maintenance windows
• Odoo represents the authoritative API to the client system
• If further API life cycle decoupling is needed for one client, a lightweight airflow or syndesis on either side of the control realm (customer or us) can bridge this decoupled API evolution.

In this scenario, OCA/queue would be the component in charge as Odoo's internal job queue. I feel this is a solid approach, as we decouple systems according to their life cycle while still honoring your choice to make OCA/queue a domain specific job queue. In this light, it probably can stay that way and entirely focus on tight Odoo integration.

If any further interaction with an external queue shall be needed, like mutually conditioning the DAGs, maybe a pub/sub event interface would be just the missing bit which motivated the creation of this FR in the first place.

[blaggacao]

/cc @mlaitinen

[github-actions[bot]]

There hasn't been any activity on this issue in the past 6 months, so it has been marked as stale and it will be closed automatically if no further activity occurs in the next 30 days. If you want this issue to never become stale, please ask a PSC member to apply the "no stale" label.