search

exacaster / lighter

REST API for Apache Spark on K8S or YARN
MIT License
91 stars 21 forks source link

feature request: Webhook to notify on status #622

Closed julienlau closed 2 weeks ago

julienlau commented 1 year ago

Hi,

It would be a nice feature if in addition to the REST endpoints there are new webhooks to notify on status change of :

It could be made using postgres trigger on state ?

julienlau commented 1 year ago

This may be implemented using postgres LISTEN / NOTIFY ?

In addition, I would suggest that rather than constantly polling postgres for new statements using SELECT, it may be appropriate to also use LISTEN / NOTIFY ?

NB: I never used this myself, but I was very surprised to learn it existed in Postgres ! https://www.postgresql.org/docs/current/sql-notify.html

julienlau commented 1 year ago

NB this feature is not implemented in H2 from what I know: https://github.com/h2database/h2database/issues/2761

Minutis commented 2 weeks ago

The feature itself sounds like a nice feature to add per se. But such notifications should probably not be implemented via storage backend. And instead the lighter should handle such events in some kind of fashion. We are open for contribution but this does not seem like a feature that would be implemented by us anytime soon.