search

haproxytech / vmware-haproxy

Apache License 2.0
51 stars 24 forks source link

Tool for collecting diagnostics information #3

Closed akutz closed 3 years ago

akutz commented 3 years ago

User Stories

As an Operator, I would like a tool that collects diagnostics information related to the HAProxy appliance, Because I need to be able to triage issues when they occur.

Details

Currently there is no good solution for collecting diagnostics information related to the HAProxy appliance that proves helpful when triaging/root-causing issues on or related to the appliance. We need a tool that can do this, and it should exist on the appliance. A tool should collect at least the information we already gather internally when triaging issues:

  1. SSH to the HAProxy appliance.

  2. Save the HAProxy service log to disk:

    sudo journalctl -xu haproxy | tee /var/log/haproxy.log

  3. Create a file with information about the version of HAProxy:

    { { rpm -qa haproxy || true; } &&
  { command -v haproxy >/dev/null 2>&1 && haproxy -vv || /usr/sbin/haproxy -vv; }; \
} | \
  sudo tee /etc/haproxy/haproxy-version

  4. Create a file with information about the version of the Data Plane API:

    { command -v dataplaneapi >/dev/null 2>&1 && dataplaneapi --version || /usr/local/bin/dataplaneapi --version; } | \
  sudo tee /etc/haproxy/dataplaneapi-version

  5. Create a file with information about the network configuration:

    { echo '--- IP TABLES ---' && \
  { iptables-save || iptables -S; } && \
  echo '--- IP ADDRS ---' && \
  ip a && \
  echo '--- IP ROUTES ---' && \
  ip r && \
  echo '--- IP ROUTE TABLE LOCAL ---' && \
  ip r show table local && \
  echo '--- IP ROUTE TABLES ---' && \
  for table_name in $(grep 'rtctl_' /etc/iproute2/rt_tables | awk '{print $2}'); do echo "${table_name}" && ip route show table "${table_name}"; done && \
  echo '--- OPEN PORTS ---' && \
  sudo lsof -noP | grep LISTEN; } | \
  sudo tee /var/log/network-info.log

  6. Create a compressed tarball that includes several files and directories:

    sudo tar -C / -czf "${HOME}/haproxy-diag.tar.gz" \
  /etc/haproxy \
  /var/log/haproxy.log \
  /var/log/network-info.log

    Please note, the above command will return a non-zero exit code if any of the above directories or files do not exist, but a tarball will still be created with the content that does exist.

  7. Validate the über-tarball created above includes the requested content:

    sudo tar tzf "${HOME}/haproxy-diag.tar.gz"

    The tarball should include some or all of the directories and files listed above.

  8. Rename the tarball to include the timestamp and even host name of the VM on which the tarball was created:

    sudo mv "${HOME}/haproxy-diag.tar.gz" "${HOME}/haproxy-diag-$(hostname --fqdn)-$(date +%s).tar.gz"

  9. Copy the tarball from the remote VM to a local location using the scp program.

Requirements

The following requirements are the minimum, known requirements. There could be additional information/requirements that should be added as well:

brakthehack commented 3 years ago

Added with #9