Abstract the dagu's storage layer

[Arvintian]

Base on executor interface, dagu can scheduler any task. But dagu's storage rely on local file system. It is limit dagu deploy as HA mode, and scheduler massive DAG task. I think we can abstract the dagu's storage layer as interface. Like chartmuseum's storage, may be we can direct use chartmuseum's library. Let dagu run on cloud !

[yohamta]

Thanks for the great suggestion as always :)

It is limit dagu deploy as HA mode, and scheduler massive DAG task

Yeah, that's absolutely true. I quickly searched and found some other modules that might be useful.

• https://github.com/chartmuseum/storage
• https://github.com/Shopify/go-storage
• https://github.com/beyondstorage/go-storage

I was vaguely thinking of using shared mounted NFS (EFS, etc.) as the storage backend for reading and writing status files and logs, but as you say, it would be great if users also could use OSS (S3, GCS, etc) as the storage (with some caching mechanism to avoid unnecessary network cost).

Now I am thinking to try to create a live demo site using GCP free tier instance after I implemented a readonly feature to prevent some security attack.

[Arvintian]

Yes, use shared mounted NFS is ok to save dagu's data. I am try use dagu to build a devops paltform, mange object storage is easier than nfs. But it is ok, caching mechanism is important about performance. This feature we can do after and think more.

[yohamta]

Yeah, will do abstraction first, then we can think more about how we can utilize object storage :)

[yohamta]

This might be useful for storage abstraction. https://github.com/C2FO/vfs