search

argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
15k stars 3.2k forks source link

what about MongoDB support for workflow persistence store? #3715

Open jackywu opened 4 years ago

jackywu commented 4 years ago

Summary

Add MongoDB support for workflow persistence store

Use Cases

In my situation，there is a MongoDB cluster already，and we are strongly depend on MongoDB，so I think make MongoDB support for workflow persistence store is a good idea。Is it meaningful ？

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

sarabala1979 commented 4 years ago

Argo is using upper.io/v3 as a DB connection library. upper.io/v3 has mango DB support https://upper.io/db.v3/mongo. do you like to contribute mango DB implementation?

jackywu commented 4 years ago

@sarabala1979 , after review the upper.io/v3 and upper.io/v4, we know upper.io use labix/mgo underneath which was deprecated. And now the official mongo driver (mongo-go-driver) is released https://github.com/mongodb/mongo-go-driver, so, will we use the official driver ? if that，we should push a commit to upper first，do you have any suggestion？

jackywu commented 4 years ago

@dtaniwaki @jessesuen @alexmt , may I have your suggestion？

alexec commented 4 years ago

Have you tried using upper.io for Mongo? Does it work?

jackywu commented 4 years ago

@alexec , I'm trying, but I'm still worry about the labix/mgo module. because this module have no transaction supported.

alexec commented 4 years ago

no transaction supported.

That's Mongo for you. We only need transactions to commit the labels with the body of the request. We only do that because we need to index the labels for querying.

For Mongo I expect you would do some thing quite different.

One thought - if you can get the SQL to work - great. But if not, you could implement the WorkflowArchive interface using Mongo (e.g. create MongoWorkflowArchive. This would require some refactoring to achieve to reduce the amount of SQL that leaks through this abstraction.

jackywu commented 4 years ago

@alexec , thanks, I'll drill down

alexec commented 2 years ago

This could be fixed if we implement archive plugins.