Content packs entities not found after installation

[arugifa]

Problem Description

I'm currently trying to install a custom content pack on Graylog 3.1.3:

{
  "v": 1,
  "id": "5cb92464-6315-4abb-bea6-45b77ff98639",
  "rev": 1,
  "name": "Test Content Pack",
  "summary": "Test Content Pack",
  "description": "",
  "vendor": "graylog",
  "url": "",
  "parameters": [],
  "entities": [
    {
      "v": "1",
      "type": {
        "name": "input",
        "version": "1"
      },
      "id": "276ae5d3-6fbc-4790-a4ec-b4aa8cfe505a",
      "data": {
        "title": {
          "@type": "string",
          "@value": "GELF UDP"
        },
        "configuration": {
          "recv_buffer_size": {
            "@type": "integer",
            "@value": 262144
          },
          "port": {
            "@type": "integer",
            "@value": 12201
          },
          "number_worker_threads": {
            "@type": "integer",
            "@value": 2
          },
          "bind_address": {
            "@type": "string",
            "@value": "0.0.0.0"
          },
          "decompress_size_limit": {
            "@type": "integer",
            "@value": 8388608
          }
        },
        "static_fields": {},
        "type": {
          "@type": "string",
          "@value": "org.graylog2.inputs.gelf.udp.GELFUDPInput"
        },
        "global": {
          "@type": "boolean",
          "@value": true
        },
        "extractors": []
      },
      "constraints": [
        {
          "type": "server-version",
          "version": ">=3.1.3+cda805f"
        }
      ]
    }
  ]
}

Entities packaged in the content pack are correctly created, and I can see them on the dashboard. However, when retrieving installation details of the content pack via the web API, Graylog cannot find these entities on the system (found_on_system key):

curl http://127.0.0.1:9000/api/system/content_packs/5cb92464-6315-4abb-bea6-45b77ff98639/installations

{
  "total": 1,
  "installations": [
    {
      "_id": "5dfcfeefcff47e00133d3cc6",
      "content_pack_id": "5cb92464-6315-4abb-bea6-45b77ff98639",
      "content_pack_revision": 1,
      "parameters": {},
      "entities": [
        {
          "id": "5dfcfeefcff47e00133d3cc4",
          "type": {
            "name": "input",
            "version": "1"
          },
          "content_pack_entity_id": "276ae5d3-6fbc-4790-a4ec-b4aa8cfe505a",
          "title": "GELF UDP",
          "found_on_system": false
        }
      ],
      "comment": "",
      "created_at": "2019-12-20T17:03:43.740Z",
      "created_by": "admin"
    }
  ]
}

The same way, when retrieving details of created entities, these ones are not linked with the content pack they belong to (content_pack's value set to null):

curl http://127.0.0.1:9000/api/system/inputs

{
  "inputs": [
    {
      "title": "GELF UDP",
      "global": true,
      "name": "GELF UDP",
      "content_pack": null,
      "created_at": "2019-12-20T17:03:43.729Z",
      "type": "org.graylog2.inputs.gelf.udp.GELFUDPInput",
      "creator_user_id": "admin",
      "attributes": {
        "recv_buffer_size": 262144,
        "port": 12201,
        "bind_address": "0.0.0.0",
        "decompress_size_limit": 8388608,
        "number_worker_threads": 2
      },
      "static_fields": {},
      "node": null,
      "id": "5dfcfeefcff47e00133d3cc4"
    }
  ],
  "total": 1
}

Expected Behavior

I expect Graylog to correctly make the association between content packs and the entities created during their installation.

Context

I'm trying to import content packs during deployment of Graylog on Kubernetes with a Helm chart.

My Environment

• Graylog Version: 3.1.3+cda805f
• Elasticsearch Version: 6.8.2
• MongoDB Version: 3.6.16
• Operating System: macOS + Docker
• Browser version: Firefox 71.0

[bernd]

@arugifa All entities in the installations returned by the /api/system/content_packs/<contne-pack-id>/installations API call have been created. That the found_on_system field is false doesn't mean that the entity hasn't been created, it means that the entity didn't exist when the content pack was installed. If you install a content pack and a grok pattern already existed before the installation, found_on_system will be true. This is used to make sure we don't remove entities on content pack installation that existed before the content pack installation.

Does that help?

[arugifa]

Thanks for the explanation @bernd

However, I noticed that when installing entities with a Content Pack, I can then delete them manually. I would have thought that Graylog would forbid such action, and that entities created with a Content Pack could only be removed afterwards by uninstalling the Content Pack.

I'm trying to provision Graylog with Content Packs, and this behavior makes things harder to reason about. But maybe Content Packs are not appropriate for this task 🤔

[bernd]

@arugifa Yes, the system is currently not preventing entities installed by content packs to be removed. Implementing this will require massive changes to all subsystems, unfortunately. We have been thinking about an updated config system that would help you with your use case. But we don't know if or when we will implement this. Sorry.

I am closing this issue now. Feel free to re-open if you have more questions. Thank you!