Error in Import content pack

[greenx]

Hi, ALL! I have two similar graylog systems (test->prod). I created on test system: a few syslog udp inputs and grok patter extractor on each input. I wanted copy this setting to production system. In "Create a content pack" I filling field "Name", "Description", "Category", selecting needed "inpurs" and "grok pattern" (I don't have output, stream, a dashboard) and download content pack. Next, on product system in "Content packs" - I pressed "import Content pack", select file and upload him. Now, I select my content pack and press "apply content" and receive error... something like "error see logs"

Expected Behavior

I expected that import settings (inputs,extractors, grok patterns)

Current Behavior

in logs I saw the following:

2016-08-11T12:02:06.758+03:00 ERROR [BundleImporter] Error while creating dashboards. Starting rollback.
org.graylog2.plugin.database.ValidationException: Missing parameters source_field or target_field.
        at org.graylog2.bundles.BundleImporter.addExtractor(BundleImporter.java:313) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleImporter.addExtractors(BundleImporter.java:302) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleImporter.createMessageInput(BundleImporter.java:292) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleImporter.createInputs(BundleImporter.java:253) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleImporter.runImport(BundleImporter.java:133) [graylog.jar:?]
...skip...
2016-08-11T12:02:06.759+03:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: ValidationException{message=Missing parameters source_field or target_field., errors={_=[ValidationFailed{passed=false, error=Missing parameters source_field or target_field.}]}}
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleImporter.runImport(BundleImporter.java:142) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleService.applyConfigurationBundle(BundleService.java:112) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleService.applyConfigurationBundle(BundleService.java:105) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.bundles.BundleResource.applyBundle(BundleResource.java:178) ~[graylog.jar:?]
..skip..

I understand that there are two errors: 1.) Wrong text in error description. I don't use dashboards (absolutely). In this place - https://github.com/Graylog2/graylog2-server/blob/9dcad95d1469384fcb57173676dbf7d5a27e2572/graylog2-server/src/main/java/org/graylog2/bundles/BundleImporter.java#L138

2.) And main error - "Missing parameters source_field or target_field." 2.1.) In my json - target_field is empty:

    "inputs": [
        {
            "configuration": {
                "allow_override_date": true,
                "bind_address": "0.0.0.0",
                "expand_structured_data": false,
                "force_rdns": true,
                "override_source": null,
                "port": 51403,
                "recv_buffer_size": 262144,
                "store_full_message": false
            },
            "extractors": [
                {
                    "condition_type": "NONE",
                    "condition_value": "",
                    "configuration": {
                        "grok_pattern": "%{_swift_container}"
                    },
                    "converters": [],
                    "cursor_strategy": "COPY",
                    "order": 0,
                    "source_field": "message",
                    "target_field": "",
                    "title": "container message parse",
                    "type": "GROK"
                }
            ],
...skip...

2.2.) Next, I found it string

// grok extractor, for example, has no target field, so set it to source to satisf

That is, the field can be left blank? Here - https://github.com/Graylog2/graylog2-web-interface/blob/eb084699edf5686bf124b4086d81ecfdc2e3fb2c/app/controllers/ExtractorsController.java#L137

2.3.) In this case, adding grok patter always is TRUE in this code:

        if (extractorDescription.getSourceField().isEmpty() || extractorDescription.getTargetField().isEmpty()) {
            throw new ValidationException("Missing parameters source_field or target_field.");
}

Here - https://github.com/Graylog2/graylog2-server/blob/9dcad95d1469384fcb57173676dbf7d5a27e2572/graylog2-server/src/main/java/org/graylog2/bundles/BundleImporter.java#L312

Possible Solution

I do not know much about the graylog - maybe it will be enough to change a logical expression.

Steps to Reproduce (for bugs)

1. Create content pack with grok pattern
2. Import this content pack
3. Apple it.

   Environment

• Graylog Version: 2.0.3
• Elasticsearch Version: 2.3.5
• MongoDB Version: 3.2.8
• Operating System: CentOS 7.2.1511
• Browser version: FireFox 47.0.1 for linux

[joschi]

@greenx Please attach the complete content pack which triggers the error.

[greenx]

attached in bottom

[greenx]

By the way, a rollback in case of failure does not work. Each pattern must be deleted as much time, how many times tried to apply content pack!

[edmundoa]

Hi @greenx,

Just to double check: does your _swift_expirer grok pattern has a new line in it? At least the content pack you posted in the issue has one.

[joschi]

@greenx As a quick side note, you should upload the JSON file of the content pack and not paste it into the comment field as text.

[greenx]

content_pack.zip

[edmundoa]

Thank you @greenx!

The new line was only in the copied text, I guess it was a typo while copy/pasting the long text.

I have been able to reproduce the issue you described, will create a patch for it soon.