schedule triggers based on time/cron

[trekze]

From Discord:

frigjord: Are there any plans to add automatic events to Hasura? E.x trigger event at a specific date/time.

hmexx: That's an interesting idea @frigjord . It does seem to fit really well with Hasura's philosophy of being the center/orchestrator of an asynchronous/serverless stack.

I think this feature would complement Event Triggers (including Console Triggers that Tiru is working on) really nicely in a way that enables the type of apps described in Hasura's 3factor-app design pattern.

I guess the current solution is to use a 3rd-party scheduling service like https://atrigger.com/ to fire a Hasura graphql call, which itself fires an Event Trigger.

[tmikaeld]

Can of course use Cron + a simple curl command, but a GUI would be much nicer.

[trekze]

If you have your own long-living backend then it’s not a huge value-add. But if hasura is your long-living orchestrator and everything else is serverless, then it’s a nice selling point.

[tmikaeld]

@hmexx No, I mean that you can use Cron + Curl even in the docker container and trigger Hasura triggers using it. That's a simple and effective solution. However, I'd rather see a proper native worker queue being added to Hasura, because it kind of makes sense and the queues could be added to the prostgres db itself so that it's persistent.

[dsandip]

@tmikaeld This seems like an interesting idea. We'll triage this in some time and also see if there's interest in the community to rank this higher in terms of priority.

[tmikaeld]

Additional use cases for timed triggers or message-queues:

• Batch processing
  • Automatically cut up really large inserts into smaller manageable and rate-limited requests.
• Transaction ordering
  • If for example 1000 people make orders at the same time this can be used to process in correct order.
• Manage downtime
  • Instead of having a large request being abrupted due to power failure or restarts, the queue will just continue where it left off.
• Monitoring
  • Message queues make it easier to log and monitor the progress of, you can see the failure state from a single view.

Just a few areas where this is useful for Hasura users.

[michaelhayman]

Would also be cool to add a 'delay' option to event triggers - for example, if a user creates a profile on my site, I want to hit a hook 2 weeks after they registered in order to send them an email or push notification.

As it stands I need to add some kind of worker which periodically either checks the database or a separate queue to process these kinds of jobs.

[Disturbing]

One of the use cases people do in event driven applications is delayed events.

In a more stateful approach, you would sleep a thread and/or use task based systems to run something after N time has passed. Problem is of course when the service crashes and/or goes down, that event that was queued is no longer there :).

My particular use case would be building a game where there are phases that are ran every N time and needs to be processed to go to the next phase. IE: Phase one = setting up the board in a game. Phase 2 is executing the AI of the board that was set up, then repeat.

It would be great to use hasura's events to trigger with an intial delay before firing and repeating.

Would also be cool to add a 'delay' option to event triggers - for example, if a user creates a profile on my site, I want to hit a hook 2 weeks after they registered in order to send them an email or push notification.

Long story short, seconding an event queue with delays. Of course we could build a 3rd party service to do this though ;)

[davie-robertson]

I have taken another approach and used Camunda's workflow engine to create time based events, these then call an 'external process' which in turn talks to Hasura - this is probably a massive overkill for 99% of use-cases but I thought it worth sharing as Camunda has a fully featured REST (and GraphQL) interface and can support complex business processes and cases.

[pnappa]

If you have your own long-living backend then it’s not a huge value-add. But if hasura is your long-living orchestrator and everything else is serverless, then it’s a nice selling point.

If everything else is serverless (i.e. AWS Lambdas), then using CloudWatch's "cron jobs" to invoke them based on a cron syntax is apposite (and free, too!).

[robnewton]

@tirumaraiselvan how will one-off scheduled jobs be handled if the time passes when the server is offline? Can these type of jobs be configured as “priority” to run if the time has passed as soon as the event engine is online again? If they are not configured as priority then they could be assumed safe to not execute if missed.

[tirumaraiselvan]

@robnewton An idea was to have something like a tolerance period i.e. a timerange say 30 mins or 6 hours. You can configure this per scheduled trigger. Now, if hasura was down and couldn't deliver the event on time, then when it comes back up, it will try to re-deliver the event if it falls within the tolerance period else it will drop the event (and join a "dead letter queue" -- yet to be implemented).

[robnewton]

Yes I think that would cover my use case perfectly. Will that launch along side this or be added to the roadmap for a future release?

[tirumaraiselvan]

It will be part of the initial release.

[weyert]

Initial release of what? Version 1.1?

[tirumaraiselvan]

@weyert Haha, initial release of Scheduled Trigger. This might be in v1.1 or v1.2.

[weyert]

Oh right, yeah, that makes sense :D

[flybayer]

It sounds like Scheduled Triggers is expected soon — will this include FIFO message queues?

I'm really needing FIFO queues in the app I'm currently building.

[tirumaraiselvan]

@flybayer Scheduled Triggers is based on time so the ordering is also based on time.

Event Triggers on the other hand is currently best-effort FIFO (but not guaranteed). We are thinking about improving it but it's a really hard problem.

[jflambert]

I'm trying this now, and it's really nice, but my first use case was going to be "purge hasura event logs"

TRUNCATE hdb_catalog.event_invocation_logs;
DELETE FROM hdb_catalog.event_log
WHERE delivered = TRUE OR error = TRUE;

I need this to run... once a day at least? otherwise these tables blow up with gigabytes of records (that's a different problem I guess) Now I'm seeing a "webhook" instead of a mutation/query dropdown. Hmm OK I could use myself as an endpoint (feels like that should have been part of the requirements here) but then...

Hasura can't track functions unless they're STABLE/IMMUTABLE. So I guess I can't really use your scheduled events for my own maintenance needs. Yes I know that's what pgagent is for. But as was mentioned earlier in this issue, Hasura is increasingly becoming our ecosystem and thanks to its built-in migrations/metadata/etc, it's nice to have a single source of truth.

Please consider adding raw SQL or "volatile" functions as triggers. In fact, why does the payload only accept JSON? Why not GraphQL?

Does this need its own separate issue? Or should I be looking at Kubernetes jobs to fulfill my needs...

[tirumaraiselvan]

@jflambert Scheduled Triggers will invoke a webhook because you may want to do many custom things.

The payload could perhaps be populated using a graphql query. Pls do raise a feature request for the same.

For your use-case of purging event logs, may I suggest using any db client or https://hasura.io/docs/1.0/graphql/manual/api-reference/schema-metadata-api/run-sql.html in your webhook?

[jflambert]

@tirumaraiselvan hey thanks! I didn't even know about "run SQL". Yes this works perfectly for now.

I also suggest adding a radio button under webhook:

• local hasura instance (http://localhost:8080/v1/query)
• custom endpoint

As well as a "run now" or "test" button. Three separate feature requests?