GraphQL: Unable to interface with endpoint

[ljfds]

I'm following the documentation (https://www.ghostwriter.wiki/features/graphql-api) and trying to interface with the GraphQL endpoint

Using the example request:

import json
import requests

headers = {"Content-Type": "application/json", }

def prepare_query(query, operation):
  return json.dumps({
    "query": query,
    "operationName": operation
  })

def post_query(headers, data):
  return requests.post(
    "https://127.0.0.1/v1/graphql",
    headers=headers,
    data=data
  )

# Stacked query with `Login` and `Whoami` operations
query = """
  mutation Login {
    login(password:"<redacted>", username:"<redacted>") {
      token expires
    }
  }

  query Whoami {
    whoami {
      username role expires
    }
  }
  """

# Send query and set `Login` as the `operationName`
response = post_query(headers, prepare_query(query, "Login"))
# Get the JWT from the response and add it to the headers
token = response.json()["data"]["login"]["token"]
headers["Authorization"] = f"Bearer {token}"
# Send the query again but execute the `Whoami` operation this time
response = post_query(headers, prepare_query(query, "Whoami"))
# Print our JWT's whoami informaiton
print(response.json())

When using the example request, I receive the following response as part of a 200: {'errors': [{'extensions': {'path': '$', 'code': 'unexpected'}, 'message': 'Invalid response from authorization hook'}]}

I created an API key and used that directly as the Bearer token, but receive the same Invalid response from authorization hook error as above.

I enabled Hasura, and similarly receive an error when attempting to perform the login query (Hasura POST's to https://127.0.0.1/v1/graphql):

However, I can perform 'other' types of queries (when authenticated with the x-hasura-admin-secret) without an issue:

Can someone please assist with the issue and let me know how I can interface with the API via python without relying on Hasura?

Does the documentation need to be updated

[chrismaddalena]

Hey @ljfds,

What do you see if you run ./ghostwriter-cli logs all?

In the Hasura Console, you can check and then uncheck the box next to x-hasura-admin-secret to trigger a webhook request and then check the logs.

That message (Invalid response from authorization hook) means Hasura couldn't properly communicate with the web server. Hasura expects a 200 OK or a 401 Unauthorized. If it gets back anything else it won't recognize it and you'll see this message.

If you look to the left side of the Hasura Console in your screenshot, the Explorer pane is empty. Without an Authorization header, that pane should show only the login mutation. You don't even see that because Hasura couldn't talk to the authentication webhook.

What you want to see in the logs is something like this:

ghostwriter-django-1 | INFO: 172.20.0.6:37994 - "GET /api/webhook HTTP/1.1" 200 OK

Look for a line like that and check the response code. Since yours is likely an error of some sort, there should be some lines after it with some information. The additional information might come from Django or Hasura.

You can also test the webhook by visiting https://127.0.0.1:443/api/webhook. You should get back a 200 OK with this JSON:

{"X-Hasura-Role": "public", "X-Hasura-User-Id": "-1", "X-Hasura-User-Name": "anonymous"}

If there's an error, it may be easier to read there, but that will depend on if this is a production environment (default) or a development environment. The latter outputs more debugging info to your browser.

[chrismaddalena]

One other thing to look at:

https://www.ghostwriter.wiki/getting-started/quickstart#customizing-the-domain-name-or-ip-address

Hasura needs to talk to https://{NGINX_HOST}:{NGINX_PORT}. The default value for {NGINX_HOST} is nginx. That works as long as nginx appears in the list of allowed hostnames.

I did have a situation where nginx did not work because of the TLS certificate. In that case, setting NGINX_HOST to the domain name used for DNS and the certificate resolved the issue–i.e., there is DNS and a cert for ghostwriter.foo.bar, so NGINX_HOST changes to ghostwriter.foo.bar.

It's been a while, but in that situation, there were no logs because a connection was never properly established. That made it difficult to troubleshoot. If you aren't seeing anything in the logs, that may be the solution for you.

[ljfds]

Thanks @chrismaddalena

I looked into this based on your suggestions, but not sure where to go next.

When having x-hasura-admin-secret set, everything works fine in the hasura console (as per the previous comment/screenshots).

I can also access the web hook endpoint in my browser without issue:

When unsetting x-hasura-admin-secret in the hasura console (or making a direct connection with python), I receive the same {'errors': [{'extensions': {'path': '$', 'code': 'unexpected'}, 'message': 'Invalid response from authorization hook'}]} error message, and can't access the login mutation.

In hasura, there is also a pop-up when unsetting the x-hasura-admin-secret variable:

Looking at the logs (as per your suggestion), I can see the problem:

{"type":"webhook-log","timestamp":"2022-08-02T00:39:28.603+0000","level":"error","detail":{"response":"<html>\r\n<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>The plain HTTP request was sent to HTTPS port</center>\r\n<hr><center>nginx/1.21.1</center>\r\n</body>\r\n</html>\r\n","url":"http://nginx:443/api/webhook","method":"GET","http_error":null,"message":null,"status_code":400}}
{"type":"http-log","timestamp":"2022-08-02T00:39:28.603+0000","level":"error","detail":{"operation":{"error":{"path":"$","error":"Invalid response from authorization hook","code":"unexpected"},"request_id":"f47e04ce-98b4-4022-9e95-bc2a70b78783","response_size":83,"request_mode":"error","raw_query":"{\"query\":\"\\n    query IntrospectionQuery {\\n      __schema {\\n        queryType { name }\\n        mutationType { name }\\n        subscriptionType { name }\\n        types {\\n          ...FullType\\n        }\\n        directives {\\n          name\\n          description\\n          locations\\n          args {\\n            ...InputValue\\n          }\\n        }\\n      }\\n    }\\n\\n    fragment FullType on __Type {\\n      kind\\n      name\\n      description\\n      fields(includeDeprecated: true) {\\n        name\\n        description\\n        args {\\n          ...InputValue\\n        }\\n        type {\\n          ...TypeRef\\n        }\\n        isDeprecated\\n        deprecationReason\\n      }\\n      inputFields {\\n        ...InputValue\\n      }\\n      interfaces {\\n        ...TypeRef\\n      }\\n      enumValues(includeDeprecated: true) {\\n        name\\n        description\\n        isDeprecated\\n        deprecationReason\\n      }\\n      possibleTypes {\\n        ...TypeRef\\n      }\\n    }\\n\\n    fragment InputValue on __InputValue {\\n      name\\n      description\\n      type { ...TypeRef }\\n      defaultValue\\n    }\\n\\n    fragment TypeRef on __Type {\\n      kind\\n      name\\n      ofType {\\n        kind\\n        name\\n        ofType {\\n          kind\\n          name\\n          ofType {\\n            kind\\n            name\\n            ofType {\\n              kind\\n              name\\n              ofType {\\n                kind\\n                name\\n                ofType {\\n                  kind\\n                  name\\n                  ofType {\\n                    kind\\n                    name\\n                  }\\n                }\\n              }\\n            }\\n          }\\n        }\\n      }\\n    }\\n  \"}"},"request_id":"f47e04ce-98b4-4022-9e95-bc2a70b78783","http_info":{"status":200,"http_version":"HTTP/1.1","url":"/v1/graphql","ip":"172.20.0.1","method":"POST","content_encoding":null}}}

The plain HTTP request was sent to HTTPS port is the error, due to the request being sent to the HTTP endpoint: http://nginx:443/api/webhook

I tried to naively fix this by updating production.yaml from (setting/changing between http and https for both the ACTIONS_URL_BASE and HASURA_GRAPHQL_AUTH_HOOK variables):

  - ACTIONS_URL_BASE=http://${NGINX_HOST}:${NGINX_PORT}/api
  - HASURA_GRAPHQL_AUTH_HOOK=http://${NGINX_HOST}:${NGINX_PORT}/api/webhook

To:

  - ACTIONS_URL_BASE=https://${NGINX_HOST}:${NGINX_PORT}/api
  - HASURA_GRAPHQL_AUTH_HOOK=https://${NGINX_HOST}:${NGINX_PORT}/api/webhook

However, after bringing everything down and then up - this change makes it worse, not better as I can't use the hasura console, or login via python:

{"type":"http-log","timestamp":"2022-08-02T03:43:26.661+0000","level":"error","detail":{"operation":{"error":{"path":"$","error":"webhook authentication request failed","code":"unexpected"},"request_id":"c076f52a-824d-4809-9e64-267b2de08ad9","response_size":80,"request_mode":"error","raw_query":"{\"query\": \"\\n  mutation Login {\\n    login(password:\\\"<redacted>\\\", username:\\\"<redacted>\\\") {\\n      token expires\\n    }\\n  }\\n\\n  query Whoami {\\n    whoami {\\n      username role expires\\n    }\\n  }\\n  \", \"operationName\": \"Login\"}"},"request_id":"c076f52a-824d-4809-9e64-267b2de08ad9","http_info":{"status":200,"http_version":"HTTP/1.1","url":"/v1/graphql","ip":"172.21.0.1","method":"POST","content_encoding":null}}}

Ghostwriter is just running locally to test, I haven't configured a 'proper' TLS certificate, or made any changes from the defaults.

I also tried to verify the hostnames weren't causing an issue, by adding a * wildcard for the allowhost allowed hosts option.

Do you have any ideas how I can get this to work/what's going wrong?

[chrismaddalena]

We know the webhook is working because you're able to connect to https://127.0.0.1/api/webhook and get back the correct response. That narrows the issue down to a connection issue between the two containers. I think I have identified the root issue, a potential mix-up with default settings.

Hasura needs to be told to accept the self-signed certificate by adding the Nginx host to Hasura's list allow list for insecure TLS. Ghostwriter already has the Docker host.docker.internal address added to that, so that should work as your Nginx hostname.

Please try this:

./ghostwriter-cli allowhost nginx
./ghostwriter-cli config set NGINX_HOST host.docker.internal
./ghostwriter-cli config set HASURA_GRAPHQL_INSECURE_SKIP_TLS_VERIFY true
./ghostwriter-cli containers down
./ghostwriter-cli containers up

The first command will make sure nginx is an allowed hostname. It should be in there by default.

The next command will set the NGINX_HOST value to the internal address Docker containers can use to talk to each other and make it so Hasura will try connecting to https://host.docker.internal/api/webhook for authentication.

The third command will allow Hasura to skip TLS verification. The final commands will restart the containers for the changes.

[ljfds]

Thanks, I followed the instructions (with some changes for the linux cli / allowhost setting):

root@ghostwriter:~/Ghostwriter# ./ghostwriter-cli-linux config allowhost nginx
[+] Configuration successfully updated. Bring containers down and up for changes to take effect.
root@ghostwriter:~/Ghostwriter# ./ghostwriter-cli-linux config set NGINX_HOST host.docker.internal
[+] Configuration successfully updated. Bring containers down and up for changes to take effect.
root@ghostwriter:~/Ghostwriter# ./ghostwriter-cli-linux config set HASURA_GRAPHQL_INSECURE_SKIP_TLS_VERIFY true
[+] Configuration successfully updated. Bring containers down and up for changes to take effect.
root@ghostwriter:~/Ghostwriter# ./ghostwriter-cli-linux containers down
[+] Bringing down the production environment
[+] Running `docker-compose` to bring down the containers with production.yml...
Stopping ghostwriter_nginx_1          ... 
Stopping ghostwriter_queue_1          ... 
Stopping ghostwriter_graphql_engine_1 ... 
Stopping ghostwriter_django_1         ... 
Stopping ghostwriter_redis_1          ... 
Stopping ghostwriter_postgres_1       ... 
Stopping ghostwriter_nginx_1          ... done
Stopping ghostwriter_graphql_engine_1 ... done
Stopping ghostwriter_queue_1          ... done
Stopping ghostwriter_django_1         ... done
Stopping ghostwriter_postgres_1       ... done
Stopping ghostwriter_redis_1          ... done
Removing ghostwriter_nginx_1          ... 
Removing ghostwriter_queue_1          ... 
Removing ghostwriter_graphql_engine_1 ... 
Removing ghostwriter_django_1         ... 
Removing ghostwriter_redis_1          ... 
Removing ghostwriter_postgres_1       ... 
Removing ghostwriter_nginx_1          ... done
Removing ghostwriter_redis_1          ... done
Removing ghostwriter_postgres_1       ... done
Removing ghostwriter_queue_1          ... done
Removing ghostwriter_graphql_engine_1 ... done
Removing ghostwriter_django_1         ... done
Removing network ghostwriter_default
root@ghostwriter:~/Ghostwriter# ./ghostwriter-cli-linux containers up
[+] Bringing up the production environment
[+] Running `docker-compose` to bring up the containers with production.yml...
Creating network "ghostwriter_default" with the default driver
Creating ghostwriter_redis_1 ... 
Creating ghostwriter_postgres_1 ... 
Creating ghostwriter_redis_1    ... done
Creating ghostwriter_postgres_1 ... done
Creating ghostwriter_django_1   ... 
Creating ghostwriter_django_1   ... done
Creating ghostwriter_graphql_engine_1 ... 
Creating ghostwriter_nginx_1          ... 
Creating ghostwriter_queue_1          ... 
Creating ghostwriter_queue_1          ... done
Creating ghostwriter_graphql_engine_1 ... done
Creating ghostwriter_nginx_1          ... done
root@ghostwriter:~/Ghostwriter# 

However the problem persists (but changed to host.docker.internal):

{"type":"webhook-log","timestamp":"2022-08-03T23:26:53.439+0000","level":"error","detail":{"response":null,"url":"http://host.docker.internal:443/api/webhook","method":"GET","http_error":{"type":"http_exception","message":"Connection failure","request":{"secure":false,"path":"/api/webhook","responseTimeout":"ResponseTimeoutDefault","queryString":"","method":"GET","requestHeaders":{"x-request-id":"0774e5d7-0295-4618-a3b9-d4425518b6d1","Content-Type":"application/json","cookie":"<REDACTED>","X-Forwarded-Proto":"https","sec-fetch-dest":"empty","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","User-Agent":"hasura-graphql-engine/v2.7.0","X-Real-IP":"172.23.0.1","X-Forwarded-Port":"443","X-Forwarded-For":"172.23.0.1"},"host":"host.docker.internal","port":443}},"message":null,"status_code":null}}
{"type":"http-log","timestamp":"2022-08-03T23:26:53.439+0000","level":"error","detail":{"operation":{"error":{"path":"$","error":"webhook authentication request failed","code":"unexpected"},"request_id":"0774e5d7-0295-4618-a3b9-d4425518b6d1","response_size":80,"request_mode":"error","raw_query":"{\"query\":\"\\n    query IntrospectionQuery {\\n      __schema {\\n        queryType { name }\\n        mutationType { name }\\n        subscriptionType { name }\\n        types {\\n          ...FullType\\n        }\\n        directives {\\n          name\\n          description\\n          locations\\n          args {\\n            ...InputValue\\n          }\\n        }\\n      }\\n    }\\n\\n    fragment FullType on __Type {\\n      kind\\n      name\\n      description\\n      fields(includeDeprecated: true) {\\n        name\\n        description\\n        args {\\n          ...InputValue\\n        }\\n        type {\\n          ...TypeRef\\n        }\\n        isDeprecated\\n        deprecationReason\\n      }\\n      inputFields {\\n        ...InputValue\\n      }\\n      interfaces {\\n        ...TypeRef\\n      }\\n      enumValues(includeDeprecated: true) {\\n        name\\n        description\\n        isDeprecated\\n        deprecationReason\\n      }\\n      possibleTypes {\\n        ...TypeRef\\n      }\\n    }\\n\\n    fragment InputValue on __InputValue {\\n      name\\n      description\\n      type { ...TypeRef }\\n      defaultValue\\n    }\\n\\n    fragment TypeRef on __Type {\\n      kind\\n      name\\n      ofType {\\n        kind\\n        name\\n        ofType {\\n          kind\\n          name\\n          ofType {\\n            kind\\n            name\\n            ofType {\\n              kind\\n              name\\n              ofType {\\n                kind\\n                name\\n                ofType {\\n                  kind\\n                  name\\n                  ofType {\\n                    kind\\n                    name\\n                  }\\n                }\\n              }\\n            }\\n          }\\n        }\\n      }\\n    }\\n  \"}"},"request_id":"0774e5d7-0295-4618-a3b9-d4425518b6d1","http_info":{"status":200,"http_version":"HTTP/1.1","url":"/v1/graphql","ip":"172.23.0.1","method":"POST","content_encoding":null}}}

The webhook endpoint still works, and Hasura works when the x-hasura-admin-secret is set.

[ljfds]

The error seems more to do with the request than the host endpoint:

oot@ghostwriter:~/Ghostwriter# docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED         STATUS         PORTS                                                                                  NAMES
d838947207b4   ghostwriter_production_graphql    "docker-entrypoint.s…"   7 minutes ago   Up 7 minutes   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:9691->9691/tcp, :::9691->9691/tcp   ghostwriter_graphql_engine_1
003819e36e13   ghostwriter_production_nginx      "/docker-entrypoint.…"   7 minutes ago   Up 7 minutes   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                               ghostwriter_nginx_1
ccba428abc96   ghostwriter_production_queue      "/entrypoint /start-…"   7 minutes ago   Up 7 minutes                                                                                          ghostwriter_queue_1
53ec6728aa26   ghostwriter_production_django     "/entrypoint /start"     7 minutes ago   Up 7 minutes                                                                                          ghostwriter_django_1
a6e5b48bc750   ghostwriter_production_postgres   "docker-entrypoint.s…"   7 minutes ago   Up 7 minutes   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp                                              ghostwriter_postgres_1
84db84725e08   ghostwriter_production_redis      "docker-entrypoint.s…"   7 minutes ago   Up 7 minutes   6379/tcp                                                                               ghostwriter_redis_1
root@ghostwriter:~/Ghostwriter# docker exec -it d838947207b4 sh
# curl -k https://nginx/api/webhook
{"X-Hasura-Role": "public", "X-Hasura-User-Id": "-1", "X-Hasura-User-Name": "anonymous"}# 
# curl http://nginx:443/api/webhook
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.21.1</center>
</body>
</html>

When x-hasura-admin-secret is not set, the request appears to go port 80 as HTTPs, while when it is set the request goes to port 443 as HTTPs.

Is there some config option to set or fix that?

I reset the host to nginx, and changed the HASURA_GRAPHQL_AUTH_HOOK endpoint: - HASURA_GRAPHQL_AUTH_HOOK=https://${NGINX_HOST}:${NGINX_PORT}/api/webhook

But this also doesn't work:

{"type":"webhook-log","timestamp":"2022-08-03T23:41:42.821+0000","level":"error","detail":{"response":null,"url":"https://nginx:443/api/webhook","method":"GET","http_error":{"type":"http_exception","message":"Internal Exception","request":{"secure":true,"path":"/api/webhook","respo
nseTimeout":"ResponseTimeoutDefault","queryString":"","method":"GET","requestHeaders":{"x-request-id":"01ab2ddc-698e-43e5-b69c-bd13aab85cf6","Content-Type":"application/json","cookie":"<REDACTED>","X-Forwarded-Proto":"https","sec-fetch-dest":"empty","sec-fetch-site":"same-origin","
sec-fetch-mode":"cors","User-Agent":"hasura-graphql-engine/v2.7.0","X-Real-IP":"172.24.0.1","X-Forwarded-Port":"443","X-Forwarded-For":"172.24.0.1"},"host":"nginx","port":443}},"message":null,"status_code":null}}
{"type":"http-log","timestamp":"2022-08-03T23:41:42.821+0000","level":"error","detail":{"operation":{"error":{"path":"$","error":"webhook authentication request failed","code":"unexpected"},"request_id":"01ab2ddc-698e-43e5-b69c-bd13aab85cf6","response_size":80,"request_mode":"error
","raw_query":"{\"query\":\"\\n    query IntrospectionQuery {\\n      __schema {\\n        queryType { name }\\n        mutationType { name }\\n        subscriptionType { name }\\n        types {\\n          ...FullType\\n        }\\n        directives {\\n          name\\n        
  description\\n          locations\\n          args {\\n            ...InputValue\\n          }\\n        }\\n      }\\n    }\\n\\n    fragment FullType on __Type {\\n      kind\\n      name\\n      description\\n      fields(includeDeprecated: true) {\\n        name\\n        des
cription\\n        args {\\n          ...InputValue\\n        }\\n        type {\\n          ...TypeRef\\n        }\\n        isDeprecated\\n        deprecationReason\\n      }\\n      inputFields {\\n        ...InputValue\\n      }\\n      interfaces {\\n        ...TypeRef\\n     
 }\\n      enumValues(includeDeprecated: true) {\\n        name\\n        description\\n        isDeprecated\\n        deprecationReason\\n      }\\n      possibleTypes {\\n        ...TypeRef\\n      }\\n    }\\n\\n    fragment InputValue on __InputValue {\\n      name\\n      desc
ription\\n      type { ...TypeRef }\\n      defaultValue\\n    }\\n\\n    fragment TypeRef on __Type {\\n      kind\\n      name\\n      ofType {\\n        kind\\n        name\\n        ofType {\\n          kind\\n          name\\n          ofType {\\n            kind\\n           
 name\\n            ofType {\\n              kind\\n              name\\n              ofType {\\n                kind\\n                name\\n                ofType {\\n                  kind\\n                  name\\n                  ofType {\\n                    kind\\n     
               name\\n                  }\\n                }\\n              }\\n            }\\n          }\\n        }\\n      }\\n    }\\n  \"}"},"request_id":"01ab2ddc-698e-43e5-b69c-bd13aab85cf6","http_info":{"status":200,"http_version":"HTTP/1.1","url":"/v1/graphql","ip":"172.24.0.1","method":"POST","content_encoding":null}}}

Just to note, this is a default install on Linux based on the wiki instructions, I didn't change anything (except what has been discussed above).

[chrismaddalena]

Thanks for sending back your results. Try running this: ./ghostwriter-cli-macos config get DJANGO_ALLOWED_HOSTS

You should see something like this:

$ ./ghostwriter-cli-macos config get DJANGO_ALLOWED_HOSTS
[+] Getting configuration values:

 Setting                                       Value
 –––––––                                     –––––––
 DJANGO_ALLOWED_HOSTS   localhost 127.0.0.1 django nginx host.docker.internal ghostwriter.local

The above output is the default. If nginx is not in the returned string, that would explain why your attempt to curl with that service name returned a 400 error. Please confirm host.docker.internal is in the string.

Please also check if host.docker.internal is in your "Insecure TLS Allow List" in the Hasura Console. It should be there by default, but authentication webhook requests will fail if it's not there. Click the gear icon in the upper-right corner and then select "Insecure TLS Allow List" from the left-hand menu.

The host.docker.internal address should have worked because it is in both of those lists by default. The DJANGO_ALLOWED_HOSTS entry recognizes it for Host header validation and the Insecure TLS Allow List tells Hasura to trust the hostname when the hostname does not match the certificate or there is a self-signed certificate in use.

[ljfds]

root@ghostwriter:~/Ghostwriter# ./ghostwriter-cli-linux config get DJANGO_ALLOWED_HOSTS
[+] Getting configuration values:

 Setting        Value
 –––––––        –––––––
 DJANGO_ALLOWED_HOSTS   localhost 127.0.0.1 172.20.0.5 django host.docker.internal ghostwriter.local * nginx

I also added the wildcard earlier based on https://www.ghostwriter.wiki/getting-started/quickstart#customizing-the-domain-name-or-ip-address.

By default, the Hasura Console contained:

I updated it to match you screenshot:

Same error :(

{"type":"webhook-log","timestamp":"2022-08-04T23:53:06.582+0000","level":"error","detail":{"response":null,"url":"https://nginx:443/api/webhook","method":"GET","http_error":{"type":"http_exception","message":"Internal Exception","request":{"secure":true,"path":"/api/webhook","responseTimeout":"ResponseTimeoutDefault","queryString":"","method":"GET","requestHeaders":{"x-request-id":"808c4bb0-09dc-477c-b424-7b9a5a7539fc","Content-Type":"application/json","cookie":"<REDACTED>","X-Forwarded-Proto":"https","sec-fetch-dest":"empty","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","User-Agent":"hasura-graphql-engine/v2.7.0","X-Real-IP":"172.24.0.1","X-Forwarded-Port":"443","X-Forwarded-For":"172.24.0.1"},"host":"nginx","port":443}},"message":null,"status_code":null}}
{"type":"http-log","timestamp":"2022-08-04T23:53:06.582+0000","level":"error","detail":{"operation":{"error":{"path":"$","error":"webhook authentication request failed","code":"unexpected"},"request_id":"808c4bb0-09dc-477c-b424-7b9a5a7539fc","response_size":80,"request_mode":"error","raw_query":"{\"query\":\"\\n    query IntrospectionQuery {\\n      __schema {\\n        queryType { name }\\n        mutationType { name }\\n        subscriptionType { name }\\n        types {\\n          ...FullType\\n        }\\n        directives {\\n          name\\n          description\\n          locations\\n          args {\\n            ...InputValue\\n          }\\n        }\\n      }\\n    }\\n\\n    fragment FullType on __Type {\\n      kind\\n      name\\n      description\\n      fields(includeDeprecated: true) {\\n        name\\n        description\\n        args {\\n          ...InputValue\\n        }\\n        type {\\n          ...TypeRef\\n        }\\n        isDeprecated\\n        deprecationReason\\n      }\\n      inputFields {\\n        ...InputValue\\n      }\\n      interfaces {\\n        ...TypeRef\\n      }\\n      enumValues(includeDeprecated: true) {\\n        name\\n        description\\n        isDeprecated\\n        deprecationReason\\n      }\\n      possibleTypes {\\n        ...TypeRef\\n      }\\n    }\\n\\n    fragment InputValue on __InputValue {\\n      name\\n      description\\n      type { ...TypeRef }\\n      defaultValue\\n    }\\n\\n    fragment TypeRef on __Type {\\n      kind\\n      name\\n      ofType {\\n        kind\\n        name\\n        ofType {\\n          kind\\n          name\\n          ofType {\\n            kind\\n            name\\n            ofType {\\n              kind\\n              name\\n              ofType {\\n                kind\\n                name\\n                ofType {\\n                  kind\\n                  name\\n                  ofType {\\n                    kind\\n                    name\\n                  }\\n                }\\n              }\\n            }\\n          }\\n        }\\n      }\\n    }\\n  \"}"},"request_id":"808c4bb0-09dc-477c-b424-7b9a5a7539fc","http_info":{"status":200,"http_version":"HTTP/1.1","url":"/v1/graphql","ip":"172.24.0.1","method":"POST","content_encoding":null}}}

[chrismaddalena]

Hey @ljfds,

Not that it helps us right now, but I did contact Hasura to see if they could suggest a better troubleshooting path. I've been trying to reproduce the problem in different ways. So far, I have only managed to reproduce a connection failure if I make an insecure connection and remove the host from the "Insecure TLS Allow List" in Hasura–e.g., Hasura connects to https://nginx:443/api/webhook for authentication and I delete nginx from the allow list. The exact host (be it nginx, host.docker.internal, or anything else) doesn't matter too much because the certificate is self-signed (none of the hosts are trusted). One other person has also encountered this, so it's not just you.

The logs you've shared so far show the request is not received by the Nginx or Django services. That tracks with the above issue. We can verify it:

$ docker-compose -f production.yml exec -ti graphql_engine /bin/sh
# apt install curl -y
# curl https://nginx/api/webhook
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
# curl https://nginx/api/webhook
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

At this point, there should be no logs for nginx or django. We can allow the insecure connection:

# curl -k https://nginx/api/webhook
{"X-Hasura-Role": "public", "X-Hasura-User-Id": "-1", "X-Hasura-User-Name": "anonymous"}
Then the logs should show:
ghostwriter-django-1          | INFO:     172.18.0.6:53964 - "GET /api/webhook HTTP/1.0" 200 OK
ghostwriter-nginx-1           | 172.18.0.7 - - [08/Aug/2022:18:38:15 +0000] "GET /api/webhook HTTP/2.0" 200 88 "-" "curl/7.64.0" "-"

You saw similar results, so we know the containers can communicate and the insecure connection is the issue. That would suggest we're troubleshooting an issue with the Insecure TLS Allow List. That does not make sense to me because a problem there should affect everyone, but I dug up something that might help. It's a long shot, but there is a way to tell Hasura to trust the self-signed certificate (or any certificate installed for Ghostwriter). I just did this (below), deleted entries in the allow list, and the webhook continued functioning. It's worth a try.

1. Stop any running containers
2. Open production.yml
3. Scroll down to line 125 (the volumes section of the graphql_engine container)
4. Add a new line below the metadata volume and add: - ./ssl:/etc/ssl/certs (see block below)
5. Rebuild the containers: docker-compose -f production.yml build
6. Bring containers back up: docker-compose -f production.yml up
7. Try to reproduce the issue

    volumes:
      - ./hasura-docker/metadata:/metadata
      - ./ssl:/etc/ssl/certs

That will mount the contents of the Ghostwriter /ssl directory in the Hasura container's /etc/ssl/certs. The service loads any certificates there into its certificate store.

[ljfds]

Thanks @chrismaddalena for the help once again.

I followed your steps above, and verified your assumption about the logs (i.e., without -k there is nothing in the log output).

I also followed the proposed remediation steps, but I end up with the exact same webhook authentication request failed error message after rebuilding and restarting the containers (bare Ubuntu 22.04 LTS install on Vultr, following the minimum steps in the installation guide to test FYI).

I do have a workaround interim solution which may help other people in a development environment (it's not a secure solution and I don't recommend it..).

The "Interacting via Automation" example script from https://www.ghostwriter.wiki/features/graphql-api can be updated to work with the Hasura secret directly (rather than a user login token):

def post_query(data):
  return requests.post(
    "http://localhost:8080/v1/graphql",
    headers={"Content-Type": "application/json", "x-hasura-admin-secret": <redacted>},
    data=data
  )

[SecurityJon]

Just tagging onto this thread as I have the same identical issue, in a production environment:

Running the commands suggested I got:

./ghostwriter-cli-linux logs all | grep 'api/webhook'
172.30.0.4 - - [17/Aug/2022:15:04:57 +0000] "GET /api/webhook HTTP/1.1" 400 255 "-" "hasura-graphql-engine/v2.7.0" "10.48.208.113"

./ghostwriter-cli-macos config get DJANGO_ALLOWED_HOSTS
DJANGO_ALLOWED_HOSTS    localhost 127.0.0.1 172.20.0.5 django host.docker.internal ghostwriter.local 10.47.208.156

./ghostwriter-cli-linux config get NGINX_HOST
NGINX_HOST  nginx

So I ran the following:

 ./ghostwriter-cli-linux config set DJANGO_ALLOWED_HOSTS 'nginx localhost 127.0.0.1 172.20.0.5 django host.docker.internal ghostwriter.local 10.47.208.156'
./ghostwriter-cli-linux config set NGINX_HOST host.docker.internal
./ghostwriter-cli-linux config set HASURA_GRAPHQL_INSECURE_SKIP_TLS_VERIFY true

I then made the changes to the production.yml file above to add in the SSL certs to the volume, I rebuild and restarted the volumnes but the same issue remained. I'm on version 3.0.0 currently.

[chrismaddalena]

Hey @SecurityJon, thanks for letting me know! I was traveling last week for Black Hat USA, so I couldn't look into this much. I am diving back into this now and am dedicated to finding an answer. I'm trying to work with the Hasura developers to see if I can learn more about the exception.

There is a workaround if you need to interact with the API right now. You can bypass the authentication by using the x-hasura-admin-secret header. You will authenticate as an administrator and Hasura won't use the webhook to authenticate your requests. If you do this, please be very careful. An administrator can do anything. You could delete or change things that should not be deleted or changed or override values that should be set automatically or auto-increment (e.g., id fields).

[chrismaddalena]

@SecurityJon @ljfds To help narrow determine if your issue is related to TLS or some other problem, please try the steps below.

Note: This assumes nginx is still a hostname configured in Hasura console's "Insecure TLS Allow List" for you. Having nginx there avoids any connection issues for me, but we've established that's not working for you. To produce the log example below I had to remove nginx from that list.

1. First, stop any running containers so you can change the Nginx configuration: ./ghostwriter-cli containers down
2. In nginx.conf, change line 4 to set logging to debug: error_log /var/log/nginx/error.log debug;
3. Save that change and restart your containers: ./ghostwriter-cli containers up
4. Connect to the Hasura Console and reproduce the webhook failure
5. Get the last few lines of logs from nginx: ./ghostwriter-cli logs nginx -l 5

Do you see an info log entry that looks anything like what I have below? Do you see something else?

172.21.0.1 - - [23/Aug/2022:23:38:16 +0000] "POST /v1/graphql HTTP/2.0" 200 149867 "https://127.0.0.1/console/api/api-explorer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0" "-"
2022/08/23 23:38:17 [info] 32#32: *37 SSL_do_handshake() failed (SSL: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:SSL alert number 46) while SSL handshaking, client: 172.21.0.6, server: 0.0.0.0:443
172.21.0.1 - - [23/Aug/2022:23:38:17 +0000] "POST /v1/graphql HTTP/2.0" 200 110 "https://127.0.0.1/console/api/api-explorer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0" "-"

Also, could you tell me which versions of Docker and Docker compose you are using and on what OS? Networking does vary slightly between macOS, Linux, and Windows with different versions of Docker.

You can use these commands:

$ docker --version                                                                        
Docker version 20.10.17, build 100c701

$ docker-compose version                 
docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

There is an open issue with the Hasura GraphQL Engine project (https://github.com/hasura/graphql-engine/issues/8746). If they can implement some additional logging that would really help here. In the meantime, I'm looking at different ways I might break the webhook to reproduce this and find a solution.

[zachfey]

Hi @chrismaddalena, jumping in here because I'm having the same issues as the above commenters. I'm able to make requests with the x-hasura-admin-secret header set, but when making requests using the Authorization header or even trying to login, I get "webhook authentication request failed".

Following your steps above, I do see this error in my logs: 2022/09/19 20:31:55 [info] 32#32: *18 SSL_do_handshake() failed (SSL: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:SSL alert number 46) while SSL handshaking, client: 172.23.0.1, server: 0.0.0.0:443 2022/09/19 20:31:56 [info] 32#32: *20 SSL_do_handshake() failed (SSL: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:SSL alert number 46) while SSL handshaking, client: 172.23.0.1, server: 0.0.0.0:443 76.229.131.181 - - [19/Sep/2022:20:31:56 +0000] "POST /v1/graphql HTTP/2.0" 200 110 "https://<REDACTED>.compute-1.amazonaws.com/console/api/api-explorer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" "-"

My versions are as follows:

Docker version 20.10.18, build b40c2f6

docker-compose version 1.29.2, build 5becea4c docker-py version: 5.0.0 CPython version: 3.7.10 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019

I have a pretty barebones instance that I setup on an EC2 instance to test out proxying API requests

[chrismaddalena]

@zachfey Thanks for adding your information to this! I setup an EC2 instance to see if I could experience the issue.

I reproduced the issue, but that was to be expected using an AWS IP and the public DNS name with the default self-signed Ghostwriter certificate. I added the public DNS name (ec2-x-x-x-x.compute-1.amazonaws.com) to Hasura's Insecure TLS Allowlist and the issue was unchanged. I added the IP addresses and private DNS name (ip-x-x-x-x.ec2.internal) to cover everything, but nothing changed.

These changes always resolved the problem for me in my test environments, so deploying in AWS changed something and allowed me to troubleshoot this first hand.

It seems like Hasura's Insecure TLS Allowlist might not work for every configuration. Based on the documentation, I expect it to ignore all SSL warnings/errors (e.g., sslv3 alert certificate unknown:SSL alert number 46) and establish the connection to the authentication webhook, but we know that adding your server's hostname or the Nginx container's hostname to that list doesn't work for everyone.

Theoretically, our Ghostwriter setups should be nearly identical because we're all using the Docker containers, so I've focused on the configuration differences. Those results are inconsistent, so I started to suspect the certificates.

I created a new self-signed certificate for the EC2 instance:

$ cd ssl
$ openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -keyout ghostwriter.key -out ghostwriter.crt
Generating a 4096 bit RSA private key
............++
............................................++
writing new private key to 'ghostwriter.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:WA
Locality Name (eg, city) [Default City]:Seattle
Organization Name (eg, company) [Default Company Ltd]:SpecterOps
Organizational Unit Name (eg, section) []:Ghostwriter
Common Name (eg, your name or your server's hostname) []:ec2-x-x-x-x.compute-1.amazonaws.com
Email Address []:admin@ghostwriter.local

Using installation defaults (i.e., default cofngiuration values set when running ./ghostwriter-cli install), the webhook authentication error disappeared. I didn't even add the EC2 public DNS name to Hasura's Insecure TLS Allowlist.

I need to look into this more to see if it would help to change to how Ghostwriter CLI generates the default certificate. For now, could someone else try generating a new certificate and checking their webhook?

Generate the certificate for whatever hostname you use to connect to Ghostwriter. If it's in EC2, use the EC2 public DNS name.

[chrismaddalena]

I also tried a new certificate with the CN set to nginx. That also worked without issue. The issue returned when I restored the original certificate files. The default certificate seems to be the problem.

To avoid issues with openssl not being in the PATH, Ghostwriter CLI generates the certificate entirely through Golang. Perhaps this causes a variance in the output base don the OS or OS version. I didn't encounter this issue with the certificate on macOS or Debian 11, but see it on Amazon Linux.

Please let me know if a new certificate helps you.

[zachfey]

@chrismaddalena that worked! Thank you very much for your help. Do you need anything else from me to help narrow this down?

[chrismaddalena]

@zachfey Thanks for confirming that helped! I appreciate the offer, but I think I've got everything I need right now. If new certificates help the others that will confirm the certificates are the root cause. I'll leave this open for a while to collect feedback.

[dmarquesdev]

@zachfey Thanks for adding your information to this! I setup an EC2 instance to see if I could experience the issue.

I reproduced the issue, but that was to be expected using an AWS IP and the public DNS name with the default self-signed Ghostwriter certificate. I added the public DNS name (ec2-x-x-x-x.compute-1.amazonaws.com) to Hasura's Insecure TLS Allowlist and the issue was unchanged. I added the IP addresses and private DNS name (ip-x-x-x-x.ec2.internal) to cover everything, but nothing changed.

These changes always resolved the problem for me in my test environments, so deploying in AWS changed something and allowed me to troubleshoot this first hand.

It seems like Hasura's Insecure TLS Allowlist might not work for every configuration. Based on the documentation, I expect it to ignore all SSL warnings/errors (e.g., sslv3 alert certificate unknown:SSL alert number 46) and establish the connection to the authentication webhook, but we know that adding your server's hostname or the Nginx container's hostname to that list doesn't work for everyone.

Theoretically, our Ghostwriter setups should be nearly identical because we're all using the Docker containers, so I've focused on the configuration differences. Those results are inconsistent, so I started to suspect the certificates.

I created a new self-signed certificate for the EC2 instance:

$ cd ssl
$ openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -keyout ghostwriter.key -out ghostwriter.crt
Generating a 4096 bit RSA private key
............++
............................................++
writing new private key to 'ghostwriter.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:WA
Locality Name (eg, city) [Default City]:Seattle
Organization Name (eg, company) [Default Company Ltd]:SpecterOps
Organizational Unit Name (eg, section) []:Ghostwriter
Common Name (eg, your name or your server's hostname) []:ec2-x-x-x-x.compute-1.amazonaws.com
Email Address []:admin@ghostwriter.local

Using installation defaults (i.e., default cofngiuration values set when running ./ghostwriter-cli install), the webhook authentication error disappeared. I didn't even add the EC2 public DNS name to Hasura's Insecure TLS Allowlist.

I need to look into this more to see if it would help to change to how Ghostwriter CLI generates the default certificate. For now, could someone else try generating a new certificate and checking their webhook?

Generate the certificate for whatever hostname you use to connect to Ghostwriter. If it's in EC2, use the EC2 public DNS name.

I was having the same issue and generating a new self-signed certificate worked! Thanks @chrismaddalena

[chrismaddalena]

If regenerating the certificates does not resolve the issue, try adding your CA certificate to the ssl directory. Name it ca-certificates.crt and place it alongside your ghostwriter.key and ghostwriter.crt.

For some certificates, Hasura seems only to want to trust the certificates if it can also import the CA certificate, even if the hostname has been added to the "Insecure TLS Allowlist." Hasura developers mentioned this should help with self-signed certificates, but it also impacted a wildcard certificate I was using for deployment.

[phughesion]

I also tried a new certificate with the CN set to nginx. That also worked without issue. The issue returned when I restored the original certificate files. The default certificate seems to be the problem.

To avoid issues with openssl not being in the PATH, Ghostwriter CLI generates the certificate entirely through Golang. Perhaps this causes a variance in the output base don the OS or OS version. I didn't encounter this issue with the certificate on macOS or Debian 11, but see it on Amazon Linux.

Please let me know if a new certificate helps you.

I had this exact problem and this is the solution that worked for me.

[DemanNL]

@chrismaddalena I have this same issue, however I was unable to solve this with adding a CA. As a workaround I used x-hasura-admin-secret. GraphQL queries work. However mutation queries don't.

Performing the generateReport mutation results in a http exception when calling webhook. With error message Proxy connection to nginx:443 returned response with status code that indicated failure: 502

The error above is the same error which I receive when using the mutation Login query. I don't understand why queries work, but mutations don't.

Additional info: I enabled debug in the nginx.conf as mentioned in https://github.com/GhostManager/Ghostwriter/issues/238#issuecomment-1225016285. This didn't result in any info logging in the error logging sadly.

Last edit and solution: This might be important information for anyone working from within an enterprise environment. Our Docker environment needs to pass through a webproxy. So we edited our docker config.json to include this:

/root/.docker/config.json
{
  "proxies": {
    "default": {
      "httpProxy": "http://PROXY:8080",
      "httpsProxy": "http://PROXY:8080
    }
  }
}

However the config above causes all the internal containers to go over the proxy. Causing nginx webhook calls to fail. In order to fix this the noProxy must be set. I've added the following line to the config.json: "noProxy": "django,postgres,redis,nginx,queue,graphql_engine"

@chrismaddalena I think this can also be fixed by expanding the no_proxy variables set in the https://github.com/GhostManager/Ghostwriter/blob/master/production.yml

After fixing the proxy issue I used these instructions to generate a certificate with the CN nginx https://github.com/GhostManager/Ghostwriter/issues/238#issuecomment-1252889629

[chrismaddalena]

@DemanNL Thanks for that detailed update. If expanding the no_proxy to include the other services worked, we should be able to add that as a default without any negative effects. A web proxy might explain why this has been so difficult to reproduce and such an intermittent issue.

[chrismaddalena]

I believe this issue was successfully narrowed down to some systems and proxies having issues with how Ghostwriter CLI generated certificates (some fields were left blank which led to some systems treating the certificate as bad). Since we made the necessary changes to Ghostwriter CLI, I have not heard of any new issues, so I'm closing this for now. If the issue resurfaces, we can re-open it and investigate.

I appreciate everyone who provided information and helped me resolve the issue!