logstash-plugins / logstash-codec-protobuf

Codec plugin for parsing Protobuf messages
Apache License 2.0
26 stars 16 forks source link

Fix issues when loading multiple concurrent pipelines #43

Closed jorgelbg closed 5 years ago

jorgelbg commented 5 years ago

⚠️ This PR is based on the changes included in #38.

While running multiple logstash pipelines we found an issue when multiple pipelines try to register (concurrently) the same dependency of class_name. This leads to unrecoverable errors that prevent the pipeline from starting with a RuntimeError similar to:

...
 :exception=>"RuntimeError",
 :message=>"trivago__DOT__logging__ProtoMyClass.header: \"header\" is already defined in \"trivago__DOT__logging__ProtoMyClass\".
...

or a different one where header dependency is not defined at all.

This happens because the DescriptorPool (generated_pool in the JRuby wrapper) it is not thread-safe (while loading protobuf classes) and it is shared between all pipelines in the same instance.

This PR adds a mutex around the loading protobuf in the register method, which means that each pipeline has exclusive access to the DescriptorPool when registering the main class and its dependencies.

The plugin is now marked as non-reloadable. If a pipeline receives a reload action the plugin will fail to register again since all definitions are already present in the DescriptorPool. Additionally, all protobuf classes/enums are registered globally and the JRuby wrapper checks the availability of the names.

jorgelbg commented 5 years ago

Looks like jdk8 is not supported out of the box in Xenial. The support goes from jdk9-jdk13:

Expected feature release number in range of 9 to 13, but got: 8
IngaFeick commented 5 years ago

Thanks @jorgelbg