search

dagu-org / dagu

Developer-friendly, minimalism Cron alternative, but with much more capabilities. It aims to solve greater problems.
https://dagu.readthedocs.io
GNU General Public License v3.0
1.45k stars 138 forks source link

About database supported #539

Open HtcOrange opened 6 months ago

HtcOrange commented 6 months ago

After understanding the basic usage of dagu, I have a curious question. Why did Dagu choose disk as the storage? Is there any possible that we have plans to support database access? Usually, as we are familiar with, a physical node, regardless of its storage space, still carries the risk of being fully occupied. At the same time, some local files may also be unexpectedly deleted due to unforeseen failures. Therefore, when we start the dagu service using the command line, do we also face the aforementioned risks? And I noticed that when Dagu scheduler scheduled tasks, the scheduler uses a "watch mechanism" to detect the update status of various files in the current Dags directory. I feel that this is somewhat similar to ETCD. Of course, Redis and MySQL may also have some "watch mechanisms". So, does it provide a possibility to watch through a database? And, when using Dagu, we submitted a yaml file through the web UI, which describes the overall structure of the task, which actually has similarities with the declarative form of k8s. Therefore, if Dagu's k8s deployment is supported in the future, its access to ETCD or other distributed databases may also be more suitable.

yohamta commented 6 months ago

The reason Dagu did not use DBMS is that I was just too lazy to maintain database. I think its simplicity is a unique point of Dagu compared to other workflow engines such as Temporal. I like the idea of k8s and etcd support though. Maybe we can support both cases in the future.

zph commented 2 months ago

Chiming in for a perspective of a current user:

When considering dagu-like solutions, I was drawn to the design's simplicity and easy introspection. I also considered temporal at the same time but didn't want the complexity.

Dagu choosing to not have external dependency has advantages of:

  1. easy deployment
  2. low maintenance
  3. simple conceptual model of dags (minimal airflow w/o all the mess)

I haven't encountered many downsides yet, but I can predict the following:

  1. performance/memory issues with larger usage
  2. maintenance complexity (custom json storage)
  3. @HtcOrange 's callout of operational/hardware failures and wanting dagu to be a staleless service with disaggregated storage.

My own guess is that it will be awhile before those disadvantages outweigh the good benefits.

I also see a middle ground of having an alternate backend available by using sqlite b/c it's a file based sql vs a client/server model. It would solve disadvantages 1 and 2 for an extended period without jeopardizing the advantages. I think it would allow for reducing the risks of 3 because then it's syncing one sqlite file up to blob storage instead of a many json files. If making it more robust operationally is a project goal, it could allow for easy integration with one of the sqlite backup solutions like https://litestream.io/. Then layer on a job queue in sqlite (https://github.com/maragudk/goqite) or build a pseudo-watch system with sql triggers.

I don't think it's the right choice now but think if the disadvantages start to cause more issues, it would be worth investigating and designing 😸. As I've started using dagu, I've pondered the design choices and like the current simplicity.