Extractor JSON Export provides wrong format

[muellerk22]

Expected Behavior

Once an Extractor is configured via the GUI and the Export extractor actions is selected the GUI presents the extractor array. A user should be able to cut and paste the extractor and place it into a contentpack. once the contentpack is loaded upon restart of graylog it should be loaded and available in the application.

Current Behavior

The extractor is ignored with this wording:

org.graylog2.periodical.ContentPackLoaderPeriodical- Couldn't parse content pack....

for

UnrecognizedPropertyException: Unrecognized field "extractor_type"

here is the JSON the GUI provides me:

{
  "extractors": [
    {
      "title": "asdfasdf",
      "extractor_type": "json",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "",
      "extractor_config": {
        "list_separator": ", ",
        "kv_separator": "=",
        "key_prefix": "",
        "key_separator": "_",
        "replace_key_whitespace": false,
        "key_whitespace_replacement": "_"
      },
      "condition_type": "none",
      "condition_value": ""
    }
  ],
  "version": "2.2.0-SNAPSHOT"
}

Possible Solution

The code that generates the output in the GUI should output a format the application understand when read in at startup. There should be a unit test created that proves all types of extactors exported can be read at start up.

Steps to Reproduce (for bugs)

1. run the graylog2/server docker image. v2.2.3+7adc951
2. configure a raw UDP input
3. configure a JSON extractor.
4. Select the extract action to export.
5. Cut and paste the suggested extractor array and input it into a contentpact.
6. start your graylog server.
7. inspect the log to see the exception

Context

I would like custom input/extractor configuration to be autoloaded when using your base docker image.

Your Environment

I'm using docker and the image for dockerhub https://hub.docker.com/r/graylog2/server/

Here is the contentpack I'm using that is failing.

{
    "category": "Inputs",
    "dashboards": [],
    "description": "a global UDP JSON input on port 10200 for risk stats",
    "grok_patterns": [],
    "inputs": [
        {
            "configuration": {
                "bind_address": "0.0.0.0",
                "decompress_size_limit": 8388608,
                "override_source": null,
                "port": 10200,
                "recv_buffer_size": 262144
            },
            "extractors": [
                {
                    "title": "Risk Library Stats Extractor",
                    "extractor_type": "json",
                    "converters": [],
                    "order": 0,
                    "cursor_strategy": "copy",
                    "source_field": "message",
                    "target_field": "",
                    "extractor_config": {
                        "list_separator": ", ",
                        "kv_separator": "=",
                        "key_prefix": "",
                        "key_separator": "_",
                        "key_whitespace_replacement": "_",
                        "replace_key_whitespace": false
                    },
                    "condition_type": "NONE",
                    "condition_value": "",
                }
            ],
            "global": true,
            "static_fields": {},
            "title": "Risk Stats UDP 10200",
            "type": "org.graylog2.inputs.raw.udp.RawUDPInput"
        }
    ],
    "name": "Risk Stats UDP 10200",
    "outputs": [],
    "streams": []
}

[muellerk22]

Also, not that i change extractor_type to type the value is in the wrong format too. GUI output was "json" and the appalication barks on that.

org.graylog2.periodical.ContentPackLoaderPeriodical - Couldn't parse content pack in file /usr/share/vagrant/graylog-configs/contentpacks/risk-stats-graylog.json. Skipping
com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not deserialize value of type org.graylog2.plugin.inputs.Extractor$Type from String "json": value not one of declared Enum instance names: [COPY_INPUT, SUBSTRING, SPLIT_AND_INDEX, REGEX, JSON, REGEX_REPLACE, GROK]
 at [Source: [B@26516b4a; line: 18, column: 29] (through reference chain: org.graylog2.bundles.ConfigurationBundle["inputs"]->java.util.HashSet[0]->org.graylog2.bundles.Input["extractors"]->java.util.ArrayList[0]->org.graylog2.bundles.Extractor["type"])

[muellerk22]

same for the value on condition_type needs to be Uppercase too. "condition_type": "NONE",

[joschi]

A user should be able to cut and paste the extractor and place it into a contentpack.

No, this is neither possible, nor is it a goal.

The "extractor export" is basically a legacy (mis-) feature which was superseded by the content packs functionality.

[muellerk22]

Then why have the export functionality enabled? Remove it. If not, support it.

Look here is your documentation from here http://docs.graylog.org/en/2.2/pages/extractors.html

You can still import extractors from JSON if you want to. Just copy the JSON extractor export into the import dialog of a message input of the fitting type

I'm just following your documentation by finding the export button to learn the format to place into a content pack. The documentation is lacking with regard to what the format is and the options available. the export is a nice tool to see that format.

I'm just saying that is a feature is present in a GUI and documented then support the functionality.

How does one learn the format for content packs without taking a random walk in the market place?

[joschi]

Then why have the export functionality enabled? Remove it. If not, support it.

The functionality is supported as described, namely on the System → Inputs → Manage Extractors page through Actions → Import extractors and Export extractors.

How does one learn the format for content packs without taking a random walk in the market place?

You can either create a content pack from your own Graylog instance (via System → Content Packs) or peek at the source code at https://github.com/Graylog2/graylog2-server/tree/2.2.3/graylog2-server/src/main/java/org/graylog2/bundles.

[muellerk22]

OK, The GUI export is for the GUI import versus GUI export place inside a contentpack for autoload at start up. That is unfortunate.

The System → Content Packs is not a good way to learn to learn the format to place in a contectpack and get it read in at auto. I will say, the error logging when Graylog starts is good at explaining what values are possible and I was able to use this and 8 restarts of a VM starting your docker image to get one that works. on to the other 3 I need to create.

thanks for working with me to get to i the bottom of this.