Can't update pipeline rules in content pack

reshippie commented 4 years ago

excerpted from forum post

I created a content pack to distribute pipeline rules to the various clusters I have. I updated some of the rules, so I figured I could just update the revision number, upload that everywhere, and be all set.

I was able to upload the new content pack, but when I select “Install” from the Actions list on the new revision I get a 500 error. In the server logs I find the following error: Caused by: org.graylog2.contentpacks.exceptions.DivergingEntityConfigurationException: Different pipeline rule sources for pipeline rule with name "upload"

Expected Behavior

I expected that making a new revision to my content pack would allow me to update the pipeline rules in place. I assumed it would just overwrite the existing rules since they have the same names.

Current Behavior

When installing a content pack that contains a pipeline rule with the same name as an existing rule, there is a check to see if they contain the same "source." If they do not, an error is thrown.

2020-03-20T22:12:18.724Z DEBUG [PipelineRuleFacade] Expected source for rule "upload":
rule "upload"
when
  has_field("source") AND contains(to_string($message.source), "pod") AND
  ((has_field("file") AND contains(to_string($message.file), "upload")) OR
  (has_field("filebeat_source") AND contains(to_string($message.filebeat_source), "upload")))
then
  let result = grok("%{GROK_PATTERN}", to_string($message.message));
  set_fields(result);
end

Actual source:
rule "upload"
when
  has_field("source") AND contains(to_string($message.source), "pod") AND
  has_field("file") AND contains(to_string($message.file), "upload")
then
  let result = grok("%{GROK_PATTERN}", to_string($message.message));
  set_fields(result);
end
2020-03-20T22:12:18.724Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
org.graylog2.contentpacks.exceptions.ContentPackException: Failed to install content pack <e3fe4c5e-40c2-4dde-a4fb-0dac6a21f693/2>
        at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:158) ~[graylog.jar:?]
        at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:99) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.contentpacks.ContentPackResource.installContentPack(ContentPackResource.java:294) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_152]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_152]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_152]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_152]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: org.graylog2.contentpacks.exceptions.DivergingEntityConfigurationException: Different pipeline rule sources for pipeline rule with name "upload"
        at org.graylog2.contentpacks.facades.PipelineRuleFacade.compareRuleSources(PipelineRuleFacade.java:151) ~[graylog.jar:?]
        at org.graylog2.contentpacks.facades.PipelineRuleFacade.findExisting(PipelineRuleFacade.java:140) ~[graylog.jar:?]
        at org.graylog2.contentpacks.facades.PipelineRuleFacade.findExisting(PipelineRuleFacade.java:126) ~[graylog.jar:?]
        at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:131) ~[graylog.jar:?]
        ... 29 more

Possible Solution

Provide a "Yes, I know what I'm doing" checkbox to allow overwrites of existing pipeline rules.
I noticed that the Grok Pattern and Lookup Table facades also contain similar comparison code, so I would assume such an option would be useful with those as well.

Steps to Reproduce (for bugs)

Create a content pack with a pipeline rule.
Install content pack on a Graylog node.
Change the pipeline rule's source code, but not its name.
Increase the revision number of the content pack.
Try to install the new version of the content pack on the same Graylog node.

Context

I'm trying to use content packs to distribute pipeline rules to multiple clusters. My workaround is to manually update each pipeline rule in each cluster, but that does not scale well and is time consuming.

Your Environment

Graylog Version: Graylog 3.1.4+1149fe1
Elasticsearch Version: ES: 6.8.3
MongoDB Version: 4.0.16
Operating System: macOS Mojave: 10.14.6
Browser version: Google Chrome: 78.0.3904.70

asachs01 commented 3 years ago

Encountered this again in HS-584826471.

kroepke commented 3 years ago

This functionality is currently not implemented, so we need to revisit it for an upcoming release.

NEO-AMiGA commented 2 years ago

I'm also running into this in 4.2.4 and 4.2.5. Would be nice with a solution since it feels like this breaks the whole idea of content packs? 🤔

@reshippie What workaround route have you gone for? Something that isn't a pita? 🥵

reshippie commented 2 years ago

I don't know if this qualifies as a "workaround," but I'm no longer using Graylog.

I still think this is a bug, but it no longer affects me.

Graylog2 / graylog2-server