search

mbuhot / ecto_job

Transactional job queue with Ecto, PostgreSQL and GenStage
https://hexdocs.pm/ecto_job/
MIT License
277 stars 36 forks source link

EctoJob UI #47

Open mbuhot opened 5 years ago

mbuhot commented 5 years ago

Following from the proof-of-concept in #46 this issue is to track the integration between ecto_job and a separate EctoJob UI from @AaronCowan.

joshuataylor commented 4 years ago

:wave:

So with pg_notify, we can be notified when new jobs are inserted -- awesome!

Is there a way to know when jobs are updated/deleted/completed/etc? It looks like pg_notify only runs for new records.

Demo of UI, using Phoenix Live View for updating when new items are added: Peek 2019-12-26 23-56

mbuhot commented 4 years ago

Awesome @joshuataylor ! You can probably prototype the UI by adding some triggers to your DB to do the pg_notify after update / delete on the queue table.

I’m wondering if the Notifications for UI should be a little different to ones used for workers. Worker notifications only include the queue name.

UI notifications payload could include the queue name, job ID, operation (insert/update/delete) and job status.

This would allow the UI to show visibility of all queues without having to configure them explicitly.

joshuataylor commented 4 years ago

Makes sense!

I'll make a new branch for ecto_job that'll play around with these concepts, then however it's decided we can just refactor the code that updates the pubsub with this.

The way I've done it is setup a Genserver with ecto_job_ui that listens with Postgrex, then sends Phoenix Pubsub so that every client connected (though there shouldn't be many for a tool like this) doesn't use pg_notify.

jeanparpaillon commented 4 years ago

I have just added mysql support. Mysql do not provide such pubsub feature, so jobs are polled regularly from DB. Would it be possible to use an abstraction of notifications instead of pg_notify ? As far as I understand, Producer module could deal with that.

mbuhot commented 4 years ago

How about embedding the UI within the Phoenix Live Dashboard? It looks like support for custom pages is going to be available soon: https://github.com/phoenixframework/phoenix_live_dashboard/issues/139

Metrics could be exposed via telemetry and shown on the existing LiveDashboard metrics page.