API browser (Swagger interface) does not work with AWS appliance

[drinkyouroj]

I've implemented Graylog using an AWS AMI (ami-d5cc48b5), following the directions at http://docs.graylog.org/en/2.2/pages/installation/aws.html. Everything is working fine, except for the Swagger API browser -- which does not appear to do anything when you use it.

Expected Behavior

When accessing the Swagger API browser, clicking on links should do something ("Show/Hide" methods, "List Operations," etc.)

Current Behavior

When I load the page (https://graylog.example.com/api/api-browser) there are a number of errors in the console. I've attached the output from Chrome 56.0.2924.87 (64-bit) on page load: graylog.allstardirectories.com-1487096853399.txt

Clicking any of the links (besides the "Raw" links) on the page does not appear to do anything (besides change the text color to black); the console does not produce any additional output. The "Raw" links always produce an Internal Server Error (if a parameter is required -- e.g., cluster/node/metrics), or a response (if no parameter is required - e.g., cluster/metrics). Neither type of "Raw" link produces any output in the console.

Possible Solution

Steps to Reproduce (for bugs)

1. Install Graylog following the documentation here: http://docs.graylog.org/en/2.2/pages/installation/aws.html
2. Configure system using graylog-ctl script and graylog-settings.json.
3. Open the REST API browser page.
4. Click around; nothing happens.

Context

We are trying to use the API browser to assist with building external scripts that will access Graylog data via the API.

Your Environment

root@aws-gravux01:~# cat /etc/graylog/graylog-settings.json { "timezone": "America/Los_Angeles", "smtp_server": "email-smtp.us-west-2.amazonaws.com", "smtp_port": 587, "smtp_user": "XXXXXXXXXXXXXXX", "smtp_password": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", "smtp_from_email": "graylog@example.com", "smtp_web_url": "https://graylog.example.com", "smtp_no_tls": false, "smtp_no_ssl": true, "master_node": "127.0.0.1", "local_connect": false, "current_address": "172.30.0.189", "last_address": "172.30.0.189", "enforce_ssl": true, "journal_size": 1, "node_id": "296c16ef-c67a-4e05-b1e0-763897135704", "internal_logging": true, "web_listen_uri": "http://127.0.0.1:9000/", "web_endpoint_uri": false, "rest_listen_uri": "http://127.0.0.1:9000/api", "rest_transport_uri": "https://graylog.example.com/api", "external_rest_uri": "https://graylog.example.com/api", "custom_attributes": { "graylog-server": { "memory": "1800m" }, "elasticsearch": { "memory": "4400m" } } }

This is all running on an EC2 T2.Large instance, with an Elastic IP mapped.

• Graylog Version: v2.2.0+d9681cb
• Elasticsearch Version: 2.4.2
• MongoDB Version: 2.4.9
• Operating System: Ubuntu 14.04.5 LTS
• Browser version: tried current versions of Chrome, Firefox, and Microsoft Edge - all behave the same way.

[drinkyouroj]

I've updated Graylog to v2.2.1+4829190 (following the directions here) and still see the same behavior. Attached is console output from Chrome 56.0.2924.87 (64-bit). graylog.allstardirectories.com-1487788174158.txt

[joschi]

@drinkyouroj Please post the result of the following command:

curl -i -X GET -H 'Accept: application/json' 'https://graylog.allstardirectories.com/api/api-docs/system/metrics/%7BmetricName%7D/'

[joschi]

@drinkyouroj FWIW, we're also running Graylog on AWS and this is the output:

$ curl -i -X GET -H 'Accept: application/json' 'https://graylog.example.com:9000/api/api-docs/system/metrics/%7BmetricName%7D/'
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET
Access-Control-Allow-Headers: Content-Type, api_key, Authorization
X-Graylog-Node-ID: 3c1749a2-abcd-4e23-0987-f0b216e50e36
X-Runtime-Microseconds: 1923
Content-Type: application/json
Date: Wed, 22 Feb 2017 19:58:13 GMT
Content-Length: 175

{"models":{},"apiVersion":"2.2.0+d9681cb","swaggerVersion":"1.2","apis":[],"basePath":"https://graylog.example.com:9000/api","resourcePath":"/system/metrics/{metricName}"}

Are you using a reverse proxy between Graylog and the Internet?

[drinkyouroj]

@joschi , I ran the command from the same server that is running Graylog itself (the AMI appliance). Output is below. Thanks so much for your help!

justin@aws-gravux01:~$ curl -i -X GET -H 'Accept: application/json' 'https://graylog.allstardirectories.com/api/api-docs/system/metrics/%7BmetricName%7D/'
HTTP/1.1 500 Internal Server Error
Server: nginx/1.9.14
Date: Wed, 22 Feb 2017 20:39:43 GMT
Content-Type: text/html;charset=ISO-8859-1
Content-Length: 1760
Connection: keep-alive

<!DOCTYPE html>
<html>
<head>
<meta name="robots" content="noindex, nofollow">
<meta charset="UTF-8">
<title>Graylog - Internal Server Error</title>
</head>
<body>
<h1>Internal Server Error</h1>

<h2>Reason</h2>
<p>java.net.URISyntaxException: Illegal character in path at index 69: http://graylog.allstardirectories.com:80/api/api-docs/system/metrics/{metricName}/</p>
<h2>Stacktrace</h2>
<p><pre>
java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal character in path at index 69: http://graylog.allstardirectories.com:80/api/api-docs/system/metrics/{metricName}/
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.getRequestUri(GrizzlyHttpContainer.java:491)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:366)
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.URISyntaxException: Illegal character in path at index 69: http://graylog.allstardirectories.com:80/api/api-docs/system/metrics/{metricName}/
        at java.net.URI$Parser.fail(URI.java:2848)
        at java.net.URI$Parser.checkChars(URI.java:3021)
        at java.net.URI$Parser.parseHierarchical(URI.java:3105)
        at java.net.URI$Parser.parse(URI.java:3053)
        at java.net.URI.&lt;init&gt;(URI.java:588)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.getRequestUri(GrizzlyHttpContainer.java:489)
        ... 6 more

</pre></p>

</body>
</html>
justin@aws-gravux01:~$

[joschi]

@drinkyouroj Please also post the output of the following two commands:

dpkg -l

curl -i -X GET -H 'Accept: application/json' 'https://graylog.allstardirectories.com/api?pretty=true'

[drinkyouroj]

@joschi The output you requested is below. Thanks again!

dpkg-l.txt

justin@aws-gravux01:~$ curl -i -X GET -H 'Accept: application/json' 'https://graylog.allstardirectories.com/api?pretty=true'
HTTP/1.1 301 Moved Permanently
Server: nginx/1.9.14

Date: Wed, 22 Feb 2017 23:41:30 GMT
Content-Type: text/html
Content-Length: 185
Location: https://graylog.allstardirectories.com/api/?pretty=true
Connection: keep-alive

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.9.14</center>
</body>
</html>
justin@aws-gravux01:~$

[joschi]

@drinkyouroj Oh, I didn't expect the redirect.

Please post the output of:

curl -i -X GET -H 'Accept: application/json' 'https://graylog.allstardirectories.com/api/?pretty=true'

[drinkyouroj]

@joschi Here you go!

justin@aws-gravux01:~$ curl -i -X GET -H 'Accept: application/json' 'https://graylog.allstardirectories.com/api/?pretty=true'
HTTP/1.1 200 OK
Server: nginx/1.9.14
Date: Thu, 23 Feb 2017 17:38:26 GMT
Content-Type: application/json
Content-Length: 253
Connection: keep-alive
X-Graylog-Node-ID: 296c16ef-c67a-4e05-b1e0-763897135704
X-Runtime-Microseconds: 393

{
  "cluster_id" : "0a8e690a-c452-42d9-9350-2bbc135d706f",
  "node_id" : "296c16ef-c67a-4e05-b1e0-763897135704",
  "version" : "2.2.1+4829190",
  "tagline" : "Manage your logs in the dark and have lasers going and make it look like you're from space!"
}justin@aws-gravux01:~$

[drinkyouroj]

@joschi , I just realized I missed one of your earlier comments. Yes, there is an nginx reverse proxy between Graylog and the Internet -- but it's been put in place via the graylog-ctl script, following the directions here. There is not a proxy in our AWS VPC; all instances have direct access to the Internet.

[gilad-feelter]

have the same issue

[dlxeon]

I have same issue too. Using Graylog installed from omnibus package, updated up to latest 2.3.1. I try to access that from the Internet using HTTPS. API browser documentation is not expanding and "Raw" links produce errors with curly braces like above. Tested on latest Edge / FF / Chrome with no difference at all.

[danielricart]

same issue here with 2.3.0 from AWS appliance using the included nginx as reverse proxy on port 80 + ELB in front to balance requests among different graylog server nodes.

[radykal-com]

the problem is with nginx config, if using the port 9000 instead of the exposed by nginx (80) api-browser works.

Looking at the browser errors all the requests with special characters like curly braces are decoded by nginx and so making requests with invalid charactes to graylog which results in errors like: java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal character in path at index 53: http://127.0.0.1/api/api-docs/system/metrics/{metricName}/history at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.getRequestUri(GrizzlyHttpContainer.java:491) at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:366) at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.URISyntaxException: Illegal character in path at index 53: http://192.168.107.22:80/api/api-docs/system/metrics/{metricName}/history at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.checkChars(URI.java:3021) at java.net.URI$Parser.parseHierarchical(URI.java:3105) at java.net.URI$Parser.parse(URI.java:3053) at java.net.URI.<init>(URI.java:588) at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.getRequestUri(GrizzlyHttpContainer.java:489) ... 6 more

browser original request is: http://127.0.0.1/api/api-docs/system/metrics/%7BmetricName%7D/history but nginx proxied request to graylog is: http://127.0.0.1/api/api-docs/system/metrics/{metricName}/history So all encoded characters are decoded by nginx and so generating invalid requests

Check: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass I think the way we want it to work is as defined here:

If proxy_pass is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI: location /some/path/ { proxy_pass http://127.0.0.1; }

The issue is easily solved by fixing the nginx.conf as follow: Edit the /opt/graylog/conf/nginx/nginx.conf

Locate the proxy_pass directive under the "Location /" block and delete the last slash from the url, in my example the original line is: proxy_pass http://localhost:9000/ replace it with: proxy_pass http://localhost:9000 (just drop the trailing slash) Save the file and run graylog-ctl restart nginx to apply the changes

CARE, running some graylog-ctl commands may change this file to the original version again (any command that run the chef recipes again)

So, I think this issue should be moved to https://github.com/Graylog2/omnibus-graylog2 project as the error is in the nginx.conf template file: https://github.com/Graylog2/omnibus-graylog2/blob/2.3/files/graylog-cookbooks/graylog/templates/default/nginx.conf.erb (Lines 20, 56 & 78)

[mariussturm]

This is fixed in the omnibus packages 2.2.3-4 and 2.3.1-4, please update if you encounter this issue. The fix will be part of the pre-build appliances with the upcoming 2.4 release.

[drinkyouroj]

I still have the same issue after applying the change in Graylog 2.2.3+7adc951 (from the omnibus installer).

justin@aws-gravux01:~$ sudo vim /opt/graylog/conf/nginx/nginx.conf
justin@aws-gravux01:~$ sudo graylog-ctl restart nginx
ok: run: nginx: (pid 20998) 0s
justin@aws-gravux01:~$ grep proxy_pass /opt/graylog/conf/nginx/nginx.conf
        proxy_pass http://localhost:9000;
        proxy_pass_request_headers on;
        proxy_pass http://localhost:9000/api/;

[radykal-com]

Looks like last line has the trailing slash and it shouldn't. Can you try to remove the trailing slash from proxy_pass http://localhost:9000/api/ then run graylog-ctl restart nginx again?

Did you edit it following my steps or did you update the omnibus package with latest build of 2.2.3 (2.2.3-4)?

[drinkyouroj]

I edited following your steps. However, when I remove the trailing slash from the proxy_pass http://localhost:9000/api/; line and restart nginx again, I receive the following error and cannot login: https://imgur.com/a/nkJ0f

[radykal-com]

Mmmh, I'll have to test it in a https setup, it's working fine in multiple installations (all http)

[radykal-com]

@drinkyouroj Hello, I have tested a new single instance with the same AMI as you, with ssl and reproduced the first issue and also the session problem when doing the last change i told you to do.

What worked for me in this setup is to remove the whole block of the api location:

location /api/ {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_pass http://localhost:9000/api/;
      }

(along removing the trailing slash of the parent location proxy_pass directive) I'm not sure why is this block suposed to be required only for ssl, as it is not present in regular http setup, and all directives of parent "location /" should work with the /api too. This block was introduced in https://github.com/Graylog2/omnibus-graylog2/commit/6abfb20665ec8442bbf30e234170dcf04e1a8393#diff-a3a989c1e843f092cf05d7d0e1650c7e by @mariussturm but I'm not sure of the special configuration of this location, maybe is related with the migration to graylog2 rest interface.

@mariussturm Can you check if this block is still required for some specific feature I can not figure out? If it's not required anymore, deleting it would fix this issue for https setups

[mariussturm]

@radykal-com yes, I think this block is obsolete. I removed it in 2.4.0-beta.1-2.

[rrittenhouse]

@mariussturm When will these fixes be pushed into the current latest 2.3.2 version?

[drinkyouroj]

@radykal-com I tested by removing the entire location /api/ block and that did, indeed, solve my problem (Graylog 2.2.3+7adc951 on graylog (Oracle Corporation 1.8.0_101 on Linux 3.13.0-119-generic))! Thank you very much for your help, everyone!

[radykal-com]

Nice to hear it. Would be nice to have it ported to 2.3.x branch

[mariussturm]

@rrittenhouse I just backmerged this into omnibus 2.3.2-3.

[rrittenhouse]

@mariussturm Awesome. This will help us immensely when upgrading our other boxes. Thank you!

[radykal-com]

Yeah, thanks you @mariussturm