search

treasure-data / digdag

Workload Automation System
https://www.digdag.io/
Apache License 2.0
1.3k stars 221 forks source link

[feature request] Lifecycle management plugin interface OR Global resource management plugin interface #1238

Open civitaspo opened 4 years ago

civitaspo commented 4 years ago

Hi all, I have a feature request: the lifecycle management plugin interface OR the global resource management plugin interface.

I wrote a plugin digdag-operator-pg_lock that can get locks by PostgreSQL(if you want to know detailed use case, see the plugin README). However I cannot control the connection pool effectively. This is because the current digdag plugin interface does not manage the lifecycle nor the global resource. I want to close the pool when the plugin cache is expired or to define the global connection pool.

Currently the plugin interface has configureBinder, but, even if I define a class as a singleton object in this method, the class does not become singleton. https://github.com/treasure-data/digdag/blob/aaefa00879e2a53bce21bd5df440fa6d0a02a461/digdag-spi/src/main/java/io/digdag/spi/Plugin.java#L9-L10

Because the plugin cache is expired in 10 minutes and re-create the plugin with a new child injector (and the injector is not referenced anywhere after creating the providerClass instance). https://github.com/treasure-data/digdag/blob/aaefa00879e2a53bce21bd5df440fa6d0a02a461/digdag-core/src/main/java/io/digdag/core/plugin/DynamicPluginLoader.java#L47 https://github.com/treasure-data/digdag/blob/aaefa00879e2a53bce21bd5df440fa6d0a02a461/digdag-core/src/main/java/io/digdag/core/plugin/DynamicPluginLoader.java#L67 https://github.com/treasure-data/digdag/blob/aaefa00879e2a53bce21bd5df440fa6d0a02a461/digdag-core/src/main/java/io/digdag/core/plugin/PluginSet.java#L26-L45

In addition, Guice does not manage the lifecycle. https://github.com/google/guice/issues/62 https://github.com/google/guice/issues/1069

So I'm in trouble because I can't implement a connection pool well. For the current digdag-operator-pg_lock implementation, the connection pool is closed with the finalize method, but it's not very good, I think.

That's why I want the feature. If you have any other good advice, please let me know...

civitaspo commented 4 years ago

How is the implementation like the below? https://github.com/treasure-data/digdag/compare/aaefa00879e2a53bce21bd5df440fa6d0a02a461...civitaspo:65ddc5035135a028445d767d63c3fb90192c26a6 .