Error 500 on inputs screen

[dschutterop]

Hi folks,

I disabled the rsyslogd service on the VM prior to enabling the UDP syslog input, because it failed on startup (figured rsyslogd hogs port 514).

After this, can't verify cause-> effect mind you, the inputs page won't load, comes up with an error 500 (Fetching Inputs failed with status: Error: cannot GET http://10.38.12.102:12900/system/inputs (500))

The link target says: {"type":"ApiError","message":"com.mongodb.BasicDBObject cannot be cast to com.mongodb.BasicDBList"}

server_log.txt

No log entries in the nginx log or Mongo log

Just slap me if I made a mistake...

[joschi]

@dschutterop Which version of Graylog are you using?

[dschutterop]

Graylog 2.0.0-alpha.1 (2db6b2e) on graylog-beta (Oracle Corporation 1.8.0_66 on Linux 3.19.0-25-generic)

(Sorry)

[joschi]

{"type":"ApiError","message":"com.mongodb.BasicDBObject cannot be cast to com.mongodb.BasicDBList"}

That looks like a bug. Could you please check the logs of the Graylog server node for a more detailed error message?

EDIT: Never mind, I've just realized you already attached the server logs in the first post of this bug report. Thanks! :heart:

[joschi]

Relevant stack trace:

2016-02-10_15:50:48.12086 java.lang.ClassCastException: com.mongodb.BasicDBObject cannot be cast to com.mongodb.BasicDBList
2016-02-10_15:50:48.12122       at org.graylog2.inputs.InputImpl.getStaticFields(InputImpl.java:120) ~[graylog.jar:?]
2016-02-10_15:50:48.12187       at org.graylog2.rest.resources.system.inputs.InputsResource.getInputSummary(InputsResource.java:208) ~[graylog.jar:?]
2016-02-10_15:50:48.12444       at org.graylog2.rest.resources.system.inputs.InputsResource.lambda$list$0(InputsResource.java:108) ~[graylog.jar:?]
2016-02-10_15:50:48.12724       at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_66]
2016-02-10_15:50:48.12728       at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_66]
2016-02-10_15:50:48.12980       at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_66]
2016-02-10_15:50:48.13368       at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_66]
2016-02-10_15:50:48.13588       at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_66]
2016-02-10_15:50:48.13639       at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_66]
2016-02-10_15:50:48.14011       at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_66]
2016-02-10_15:50:48.14054       at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_66]
2016-02-10_15:50:48.14787       at org.graylog2.rest.resources.system.inputs.InputsResource.list(InputsResource.java:109) ~[graylog.jar:?]

[joschi]

@dschutterop Please attach a dump of the inputs collection from MongoDB on this ticket (or send it to jochen@graylog.com if it contains sensitive data like passwords).

You can get a dump of this collection by running the following mongoexport command in the virtual appliance:

mongoexport --db graylog --collection inputs --out inputs.json

[dschutterop]

Hi Jochen,

Will do, can't reach this environment from home so it 'll be first thing tomorrow morning.

Greets,

Daniel

Sent from my iPhone

On 10 Feb 2016, at 18:03, Jochen Schalanda notifications@github.com wrote:

@dschutterop Please attach a dump of the inputs collection from MongoDB on this ticket (or send it to jochen@graylog.com if it contains sensitive data like passwords).

You can get a dump of this collection by running the following mongoexport command in the virtual appliance:

mongoexport --db graylog2 --collection inputs --out inputs.json — Reply to this email directly or view it on GitHub.

[dschutterop]

Hmm..

2016-02-11T07:38:42.661+0000 connected to: localhost 2016-02-11T07:38:42.661+0000 exported 0 records

(And an empty inputs.json of course)

[dschutterop]

And a screendump for clarification, just in case.

[joschi]

2016-02-11T07:38:42.661+0000 connected to: localhost 2016-02-11T07:38:42.661+0000 exported 0 records

Please make sure that the MongoDB database name (graylog in my example) is correct. The old default was graylog2, so maybe the database name in the mongoexport command was simply incorrect. You can also retrieve the same information using the MongoDB shell and the following command:

db.inputs.find().pretty()

[dschutterop]

Back again, didn't realize the database name evolution.. I've got two json objects I've prettified 'em for you for the sake of readability:

{
  "_id": {
    "$oid": "56b31366e4b0724a1259bdfe"
  },
  "title": "appliance-gelf-udp",
  "type": "org.graylog2.inputs.gelf.udp.GELFUDPInput",
  "creator_user_id": "admin",
  "configuration": {
    "override_source": "",
    "allow_override_date": true,
    "recv_buffer_size": 1048576,
    "port": 12201,
    "bind_address": "0.0.0.0"
  },
  "created_at": {
    "$date": "2016-02-04T09:01:26.558Z"
  },
  "content_pack": "56b31366e4b0724a1259bdb1",
  "node_id": "247d6c04-6991-495c-8afe-ea0158bce9e5",
  "static_fields": [
    {
      "value": "true",
      "key": "from_gelf"
    }
  ],
  "extractors": [
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "Remote Address",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "remote_addr",
      "extractor_config": {
        "regex_value": "nginx:\\s+(\\S+)"
      },
      "condition_value": "^nginx:",
      "converters": [

      ],
      "id": "29a3f4d5-206c-409b-8b39-4692e14037f7",
      "order": {
        "$numberLong": "0"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "Remote User",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "remote_user",
      "extractor_config": {
        "regex_value": "nginx: \\S+ - (\\S+)"
      },
      "condition_value": "^nginx:",
      "converters": [

      ],
      "id": "18b4642b-9181-498e-82c9-e7abfb022b05",
      "order": {
        "$numberLong": "1"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "Request Timestamp",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "timestamp",
      "extractor_config": {
        "regex_value": "nginx:.+?\\[(.+?)\\]"
      },
      "condition_value": "^nginx:",
      "converters": [
        {
          "type": "date",
          "config": {
            "date_format": "dd\/MMM\/YYYY:HH:mm:ss Z"
          }
        }
      ],
      "id": "98d6537b-91eb-4f29-a791-b727dfaa8754",
      "order": {
        "$numberLong": "2"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "Request Verb",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "request_verb",
      "extractor_config": {
        "regex_value": "nginx:.+\\[.+\\] \"(\\S+)"
      },
      "condition_value": "^nginx:",
      "converters": [

      ],
      "id": "fe2a7d6a-3a51-4957-bcd0-eb8d5fb503fd",
      "order": {
        "$numberLong": "3"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "Request Path",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "request_path",
      "extractor_config": {
        "regex_value": "nginx:.+?\"\\S+ (\\S+).+\""
      },
      "condition_value": "^nginx:",
      "converters": [
        {
          "type": "numeric",
          "config": {

          }
        }
      ],
      "id": "0693f02a-b469-4e95-afa5-a70edf3aa789",
      "order": {
        "$numberLong": "4"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "HTTP Version",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "http_version",
      "extractor_config": {
        "regex_value": "nginx:.+HTTP\/(\\S+)\""
      },
      "condition_value": "^nginx:",
      "converters": [

      ],
      "id": "050f24f0-2c55-4809-ab58-dc3ccffc6777",
      "order": {
        "$numberLong": "5"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "Response Status",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "response_status",
      "extractor_config": {
        "regex_value": "nginx:.+?HTTP\/\\S+\" (\\d+)"
      },
      "condition_value": "^nginx:",
      "converters": [
        {
          "type": "numeric",
          "config": {

          }
        }
      ],
      "id": "14762d59-f0be-4365-a2ec-21ec17cda90e",
      "order": {
        "$numberLong": "6"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "Response Bytes",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "response_bytes",
      "extractor_config": {
        "regex_value": "nginx:.+?HTTP\/\\S+\" \\d+ (\\d+)"
      },
      "condition_value": "^nginx:",
      "converters": [
        {
          "type": "numeric",
          "config": {

          }
        }
      ],
      "id": "9847ee78-4bc6-4ff0-9bd0-fbd9c03c5918",
      "order": {
        "$numberLong": "7"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "HTTP Referer",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "http_referer",
      "extractor_config": {
        "regex_value": "nginx:.+?HTTP\/\\S+\" \\d+ \\d+ \"(.+?)\""
      },
      "condition_value": "^nginx:",
      "converters": [

      ],
      "id": "b2164cf7-99cd-40d5-b40f-1ea2cd0c36bd",
      "order": {
        "$numberLong": "9"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "HTTP User Agent",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "http_user_agent",
      "extractor_config": {
        "regex_value": "nginx:.+?HTTP\/\\S+\" \\d+ \\d+ \".+?\" \"(.+?)\""
      },
      "condition_value": "^nginx:",
      "converters": [

      ],
      "id": "b386de05-fc62-4b45-9968-bf0d5b6bb870",
      "order": {
        "$numberLong": "8"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "Connection ID",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "connection_id",
      "extractor_config": {
        "regex_value": "connection=(.+?)\\|"
      },
      "condition_value": ".+connection=.+",
      "converters": [
        {
          "type": "numeric",
          "config": {

          }
        }
      ],
      "id": "a722016c-fa31-4531-a908-98b1c0c77d85",
      "order": {
        "$numberLong": "10"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "Connection requests",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "connection_requests",
      "extractor_config": {
        "regex_value": "connection_requests=(.+?)\\|"
      },
      "condition_value": ".+connection_requests=.+",
      "converters": [
        {
          "type": "numeric",
          "config": {

          }
        }
      ],
      "id": "16dd12fa-bbc8-480c-9a72-19f441036419",
      "order": {
        "$numberLong": "11"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "Response time",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "millis",
      "extractor_config": {
        "regex_value": "millis=(.+?)>"
      },
      "condition_value": ".+millis=.+",
      "converters": [
        {
          "type": "numeric",
          "config": {

          }
        }
      ],
      "id": "5cb46da6-2435-4518-a338-beba33408610",
      "order": {
        "$numberLong": "12"
      }
    },
    {
      "creator_user_id": "admin",
      "source_field": "message",
      "condition_type": "regex",
      "title": "Message",
      "type": "regex",
      "cursor_strategy": "copy",
      "target_field": "message",
      "extractor_config": {
        "regex_value": "nginx:.+?\\\"(\\S+.+HTTP\\\/\\S+)\\\" \\d+"
      },
      "condition_value": "^nginx:",
      "converters": [

      ],
      "id": "967d9e83-eca3-4c1c-a673-37f0dc6d57bb",
      "order": {
        "$numberLong": "13"
      }
    }
  ]
}
{
  "_id": {
    "$oid": "56b31366e4b0724a1259be02"
  },
  "creator_user_id": "admin",
  "configuration": {
    "expand_structured_data": false,
    "recv_buffer_size": 262144,
    "port": 514,
    "override_source": null,
    "force_rdns": false,
    "allow_override_date": true,
    "bind_address": "0.0.0.0",
    "store_full_message": false
  },
  "name": "Syslog UDP",
  "created_at": {
    "$date": "2016-02-10T15:25:53.135Z"
  },
  "global": false,
  "type": "org.graylog2.inputs.syslog.udp.SyslogUDPInput",
  "title": "appliance-syslog-udp",
  "content_pack": null,
  "static_fields": {
    "from_syslog": "true"
  },
  "node_id": "247d6c04-6991-495c-8afe-ea0158bce9e5"

[joschi]

{
  "_id": {
    "$oid": "56b31366e4b0724a1259be02"
  },
  "name": "Syslog UDP",
  "type": "org.graylog2.inputs.syslog.udp.SyslogUDPInput",
  "title": "appliance-syslog-udp",
  "static_fields": {
    "from_syslog": "true"
  },
  "node_id": "247d6c04-6991-495c-8afe-ea0158bce9e5"
}

The static_fields object is invalid here and is causing the exception you've posted originally. The correct form would be as follows (compare to the appliance-gelf-udp input description):

  "static_fields": [
    {
      "key": "from_syslog",
      "value": "true"
    }
  ],

Did you edit this directly in MongoDB or was this a fresh installation of the Graylog 2.0.0-alpha.1 virtual appliance? If it was the latter, could you please create a new virtual machine from the OVA and check if the error still occurs?

[dschutterop]

I wouldn't dare editing the mongoDB directly and open an issue afterwards :p It was a clean deploy from the OVA.

I'll redeploy and see what happens.

[mariussturm]

this is maybe an error in the appliance content-pack: https://github.com/Graylog2/omnibus-graylog2/blob/1.3/files/data/contentpacks/content_pack_appliance.json#L18

Going to fix it for the next alpha!

[dschutterop]

Cool. I'll correct it manually and be on my merry way :)

Thanks guys.

[joschi]

@mariussturm

this is maybe an error in the appliance content-pack: https://github.com/Graylog2/omnibus-graylog2/blob/1.3/files/data/contentpacks/content_pack_appliance.json#L18

The format for static_fields in a content pack is different from what's stored in MongoDB. Don't let that fool you just because both looks like JSON. :wink:

[joschi]

@dschutterop I've just downloaded the OVA from https://packages.graylog2.org/releases/graylog-omnibus/ova/graylog-beta.ova and created a fresh virtual machine (using VirtualBox 5.0.14). The inputs were created correctly, including the static_fields object. Could you please re-download the OVA and try again?

[dschutterop]

I've just deployed the OVA I downloaded this week, let me try to break it first.. I'll re-create the steps with which I broke it last time before I download a fresh copy :)

[dschutterop]

That's interesting, can't seem to break it after re-deploying.

• Checked the inputs page -> loads without a problem;
• both inputs are running (Unlike last time when the syslog UDP had failed)

When I saw the failed input (514/UDP) last time, I put 2 and 2 together and killed the rsyslogd (possibly hogging 514) and restarted the input (probably a few times). After that, I kept getting the 500 error.

Must have been an misfire somewhere, works like a charm now...

[joschi]

@dschutterop Thanks for the update!

I'll close this issue for now since I can't reproduce the problem and the code for importing content packs and for handling inputs looks sane in the relevant places.

Feel free to add a comment if the problem occurs again and we'll reopen the issue.

[stephenrenshaw]

I have the exact same problem in alpha-5 fresh install, i've tried the same steps above just redeploying the .ova now to see if that clears it.

Reinstall makes no difference still cant access the inputs page after the first time, same 500 error couldnt not retrieve inputs.

==> /var/log/graylog/server/current <== 2016-03-08_11:48:09.13046 2016-03-08 11:48:09,129 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource 2016-03-08_11:48:09.13151 java.lang.ClassCastException: com.mongodb.BasicDBObject cannot be cast to com.mongodb.BasicDBList 2016-03-08_11:48:09.13365 at org.graylog2.inputs.InputImpl.getStaticFields(InputImpl.java:120) ~[graylog.jar:?] 2016-03-08_11:48:09.13552 at org.graylog2.rest.resources.system.inputs.InputsResource.getInputSummary(InputsResource.java:208) ~[graylog.jar:?] 2016-03-08_11:48:09.13987 at org.graylog2.rest.resources.system.inputs.InputsResource.lambda$list$0(InputsResource.java:108) ~[graylog.jar:?] 2016-03-08_11:48:09.14052 at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_74] 2016-03-08_11:48:09.14163 at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_74] 2016-03-08_11:48:09.14398 at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_74] 2016-03-08_11:48:09.14542 at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_74] 2016-03-08_11:48:09.14649 at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_74] 2016-03-08_11:48:09.14798 at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_74] 2016-03-08_11:48:09.14841 at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_74] 2016-03-08_11:48:09.14911 at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_74] 2016-03-08_11:48:09.15090 at org.graylog2.rest.resources.system.inputs.InputsResource.list(InputsResource.java:109) ~[graylog.jar:?] 2016-03-08_11:48:09.15303 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_74] 2016-03-08_11:48:09.15360 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_74] 2016-03-08_11:48:09.15435 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_74] 2016-03-08_11:48:09.15881 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_74] 2016-03-08_11:48:09.15939 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?] 2016-03-08_11:48:09.16037 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?] 2016-03-08_11:48:09.16105 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?] 2016-03-08_11:48:09.16251 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?] 2016-03-08_11:48:09.16312 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?] 2016-03-08_11:48:09.16452 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?] 2016-03-08_11:48:09.16597 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?] 2016-03-08_11:48:09.16740 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?] 2016-03-08_11:48:09.16809 at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?] 2016-03-08_11:48:09.18566 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?] 2016-03-08_11:48:09.18769 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?] 2016-03-08_11:48:09.19410 at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?] 2016-03-08_11:48:09.19491 at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?] 2016-03-08_11:48:09.19881 at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?] 2016-03-08_11:48:09.19964 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?] 2016-03-08_11:48:09.20216 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?] 2016-03-08_11:48:09.20284 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?] 2016-03-08_11:48:09.20495 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?] 2016-03-08_11:48:09.20562 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?] 2016-03-08_11:48:09.20658 at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]

[stephenrenshaw]

found a fix delete the mongodb database graylog and it goes away

[dschutterop]

That's a little drastic, dontcha think?

Sent from my iPhone

On 08 Mar 2016, at 16:07, stephenrenshaw notifications@github.com wrote:

found a fix delete the mongodb database graylog and it goes away

— Reply to this email directly or view it on GitHub.

[stephenrenshaw]

nope, something must have broken in putting together the alpha-5 .ova graylog recreated the db and everything now works as expected.