Analyzer connection : 504 Gateway Timeout

Hi,

I'm trying to discover the skydive-flow-matrix but I have a problem connecting to the analyzer :

skydive-flow-matrix --analyzer X.X.X.X:8082
Traceback (most recent call last):
  File "/usr/local/bin/skydive-flow-matrix", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/matrix/matrix.py", line 451, in main
    csv_output(matrix, args.list_no_peers)
  File "/usr/local/lib/python2.7/dist-packages/matrix/matrix.py", line 266, in csv_output
    result = matrix.get_matrix()
  File "/usr/local/lib/python2.7/dist-packages/matrix/matrix.py", line 107, in get_matrix
    "G.%sV().Has('Type', 'host').HasKey('Sockets')" % self.get_context())
  File "/usr/local/lib/python2.7/dist-packages/skydive/rest/client.py", line 111, in lookup_nodes
    return self.lookup(gremlin, Node)
  File "/usr/local/lib/python2.7/dist-packages/skydive/rest/client.py", line 104, in lookup
    objs = self.request("/api/topology", method="POST", data=data)
  File "/usr/local/lib/python2.7/dist-packages/skydive/rest/client.py", line 85, in request
    raise BadRequest(e.read())
skydive.rest.client.BadRequest: <HTML><TITLE>504 Gateway Timeout</TITLE>
<H1>504 Gateway Timeout</H1>
Gateway timeout expired while waiting for server response</HTML>

I'm trying to use this tool in Openshift. Here is my current configuration :

# docker --version
Docker version 1.12.6, build 3e8e77d/1.12.6

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)

# openshift version
openshift v3.7.54
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

I tried to install the tool in the same image (skydive/skydive:0.20.0) to use it in the same pod or in another pod but the result is the same.

Here is the current configuration :

analyzers:
  - 127.0.0.1:8082

agent:
  topology:
    probes:
      - ovsdb
      - docker
      - socketinfo

analyzer:
  flow:
    backend: myelasticsearch
  listen: 0.0.0.0:8082
  topology:
    backend: myelasticsearch
    #fabric:
    #- TOR1[Name=tor1] -> TOR1_PORT1[Name=port1, MTU=1500]
    #- TOR1_PORT1 -> *[Type=host, Name=xxxx]/ens192
    #- TOR1[Name=tor1] -> TOR1_PORT2[Name=port2, MTU=1500]
    #- TOR1_PORT2 -> *[Type=host, Name=xxxx]/ens192
  auth:
    # auth section for API request
    api:
      # Specify the name of the auth backend definition, see auth section.
      backend: noauth
storage:
  #Elasticsearch backend information.
  myelasticsearch:
    driver: elasticsearch
    host: elasticsearch-skydive:9200

etcd:
  embedded: true
  listen: 0.0.0.0:12379

ui:
  # Specify the extra assets folder. Javascript and CSS files present in this
  # folder will be added to the WebUI.
  # extra_assets: /usr/share/skydive/assets

  # select between light, dark themes
  theme: light

  # Settings specific to the topology view
  topology:
    # Pre-defined Gremlin expression used in the WebUI for Filtering and Highlighting.
    # Note: Key should be in lower case
    favorites:
       #namespaces: "g.V().Has('Type', 'netns').OutE().BothV()"
       #layer2: "g.E().Has('RelationType', 'layer2')"

    # Highlight Gremlin expression used by default and applied on WebUI load.
    #default_highlight: "layer2"
    # Filter Gremlin expression used by default and applied on WebUI load.
    #default_filter: "layer2"

  # update rate of links in seconds
  bandwidth_update_rate: 5

  # 'absolute' - thresholds in Kbit
  # 'relative' - thresholds in % relative to link speed reported by netlink
  bandwidth_threshold: absolute
  bandwidth_absolute_active: 1
  bandwidth_absolute_warning: 10
  bandwidth_absolute_alert: 100
  bandwidth_relative_active: 0.1
  bandwidth_relative_warning: 0.4
  bandwidth_relative_alert: 0.8

  # Enable/disable ssh to hosts
  # ssh_enabled: false

  bpf:
    # Pre-defined BPF filters
    favorites:
      # filter1: ip broadcast
      # filter2: ip multicast

logging:
  level: DEBUG

Will I have an option to activate on the Analyzer ? Do you have any idea ?

Thanks !

I continued my tests, with a second container in the same pod (localhost), I have the following error :

# skydive-flow-matrix --analyzer 127.0.0.1:8082
Traceback (most recent call last):
  File "/usr/local/bin/skydive-flow-matrix", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/matrix/matrix.py", line 451, in main
    csv_output(matrix, args.list_no_peers)
  File "/usr/local/lib/python2.7/dist-packages/matrix/matrix.py", line 266, in csv_output
    result = matrix.get_matrix()
  File "/usr/local/lib/python2.7/dist-packages/matrix/matrix.py", line 107, in get_matrix
    "G.%sV().Has('Type', 'host').HasKey('Sockets')" % self.get_context())
  File "/usr/local/lib/python2.7/dist-packages/skydive/rest/client.py", line 111, in lookup_nodes
    return self.lookup(gremlin, Node)
  File "/usr/local/lib/python2.7/dist-packages/skydive/rest/client.py", line 104, in lookup
    objs = self.request("/api/topology", method="POST", data=data)
  File "/usr/local/lib/python2.7/dist-packages/skydive/rest/client.py", line 85, in request
    raise BadRequest(e.read())
skydive.rest.client.BadRequest: <HTML><TITLE>503 Service Unavailable</TITLE>
<H1>503 Service Unavailable</H1>
Failed to connect to server <B>127.0.0.1</B></HTML>

On the other hand, if I try, for example, the command below, the analyzer answers correctly:

# skydive client status --analyzer 127.0.0.1:8082
{
  "Agents": {
    "XXXX": {
      "ServiceType": "agent",
      "ClientProtocol": "protobuf",
      "Addr": "X.X.X.X",
      "Port": 56328,
      "IsConnected": true,
      "ConnectTime": "2018-10-24T14:22:23.256576159Z",
      "RemoteHost": "XXXX"
    },
    "XXXX": {
      "ServiceType": "agent",
      "ClientProtocol": "protobuf",
      "Addr": "X.X.X.X",
      "Port": 42862,
      "IsConnected": true,
      "ConnectTime": "2018-10-24T14:22:23.300763942Z",
      "RemoteHost": "XXXX"
    }
  },
  "Peers": {
    "Incomers": {},
    "Outgoers": {}
  },
  "Publishers": {},
  "Subscribers": {
    "X.X.X.X:45560": {
      "ServiceType": "webui",
      "ClientProtocol": "json",
      "Addr": "X.X.X.X",
      "Port": 45560,
      "IsConnected": true,
      "ConnectTime": "2018-10-24T14:22:30.724894609Z",
      "RemoteHost": "X.X.X.X:45560"
    }
  },
  "Alerts": {
    "IsMaster": true
  },
  "Captures": {
    "IsMaster": true
  },
  "Probes": [
    "fabric",
    "peering"
  ]
}

I use Skydive version 0.20.0.

Do you have any idea ?

Thanks !

skydive-project / skydive-flow-matrix

Analyzer connection : 504 Gateway Timeout #7