search

sensu / sensu-go

Simple. Scalable. Multi-cloud monitoring.
https://sensu.io
MIT License
1.03k stars 175 forks source link

asset PATH not injected in Debian 10 #3956

Closed davidvilla closed 4 years ago

davidvilla commented 4 years ago

Expected Behavior

Directory {PATH_TO_ASSET}/bin should be injected into the PATH environment variable as documentation states

Current Behavior

Asset path is not in PATH inside the check command context. Variable {{assetPath "<ASSET_PATH>"}} is not there either.

Steps to Reproduce

Create a check with command "echo $PATH $LD_LIBRARY_PATH" to see the PATH_TO_ASSET/bin is not there. Same for LD_LIBRARY_PATH

Context

Basic installation with asset sensu/monitoring-plugins, by means; sensuctl asset add sensu/monitoring-plugins

Your Environment

image

palourde commented 4 years ago

Hi @davidvilla,

Could you make sure the asset is actually getting installed on the agent? You might see something like entity not filtered from any asset builds, not installing asset in the agent logs, which would mean that none of the filters applied to the agent's entity. Turning the logs to the debug level will also provide more information about that.

Alternatively, you could also try to execute one of the binary in sensu/monitoring-plugins to make sure it's actually getting installed.

davidvilla commented 4 years ago

Hi:

Yes, the assets are installed and I can run them from console.

Cheers

palourde commented 4 years ago

Could you provide us some of your configuration to help us investigate this issue?

Here's various information that could be useful:

$ sensuctl entity info $ENTITY_NAME --format yaml

$ sensuctl check info $CHECK_INFO --format yaml

$ sensuctl event info $ENTITY_NAME $CHECK_INFO --format yaml

$ sensu-backend version

# On the related agent node
$ sensu-agent version

Thanks!

davidvilla commented 4 years ago

Hi:

$ sensuctl entity info amy --format yaml
type: Entity
api_version: core/v2
metadata:
  name: amy
  namespace: default
spec:
  deregister: false
  deregistration: {}
  entity_class: agent
  last_seen: 1597219614
  redact:
  - password
  - passwd
  - pass
  - api_key
  - api_token
  - access_key
  - secret_key
  - private_key
  - secret
  sensu_agent_version: 5.21.0
  subscriptions:
  - linux
  - entity:amy
  system:
    arch: amd64
    cloud_provider: ""
    hostname: amy
    libc_type: glibc
    network:
      interfaces:
      - addresses:
        - 127.0.0.1/8
        - ::1/128
        name: lo
      - addresses:
        - 169.254.7.221/16
        mac: 11:22:33:44:55:66
        name: enp0s31f6
      - addresses:
        - 192.168.8.235/24
        - fe80::1111:2222:3333:4444/64
        mac: 11:22:33:44:55:66
        name: wlp1s0
    os: linux
    platform: debian
    platform_family: debian
    platform_version: "10.5"
    processes: null
    vm_role: host
    vm_system: kvm
  user: agent
$ sensuctl check info ping-google --format yaml
type: CheckConfig
api_version: core/v2
metadata:
  created_by: admin
  labels:
    sensu.io/managed_by: sensuctl
  name: ping-google
  namespace: default
spec:
  check_hooks: null
  command: check_ping -H www.google.com -w 10,1% -c 20,2%
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 30
  low_flap_threshold: 0
  output_metric_format: nagios_perfdata
  output_metric_handlers: null
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - monitoring-plugins
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - linux
  timeout: 10
  ttl: 0
$ sensuctl event info amy ping-google --format yaml
type: Event
api_version: core/v2
metadata:
  namespace: default
spec:
  check:
    check_hooks: null
    command: check_ping -H www.google.com -w 10,1% -c 20,2%
    duration: 0.002072679
    env_vars: null
    executed: 1597220613
    handlers: []
    high_flap_threshold: 0
    history:
    - executed: 1597219567
      status: 0
    - executed: 1597219597
      status: 1
    - executed: 1597219627
      status: 0
    - executed: 1597219657
      status: 0
    - executed: 1597219687
      status: 0
    - executed: 1597219717
      status: 1
    - executed: 1597219747
      status: 0
    - executed: 1597219777
      status: 0
    - executed: 1597219807
      status: 1
    - executed: 1597219837
      status: 1
    - executed: 1597220323
      status: 3
    - executed: 1597220353
      status: 0
    - executed: 1597220383
      status: 1
    - executed: 1597220413
      status: 0
    - executed: 1597220443
      status: 1
    - executed: 1597220473
      status: 0
    - executed: 1597220503
      status: 1
    - executed: 1597220528
      status: 1
    - executed: 1597220553
      status: 127
    - executed: 1597220583
      status: 127
    - executed: 1597220613
      status: 127
    interval: 30
    issued: 1597220613
    last_ok: 1597220473
    low_flap_threshold: 0
    metadata:
      labels:
        sensu.io/managed_by: sensuctl
      name: ping-google
      namespace: default
    occurrences: 3
    occurrences_watermark: 3
    output: |
      sh: 1: check_ping: not found
    output_metric_format: nagios_perfdata
    output_metric_handlers: null
    proxy_entity_name: ""
    publish: true
    round_robin: false
    runtime_assets:
    - monitoring-plugins
    secrets: null
    state: failing
    status: 127
    stdin: false
    subdue: null
    subscriptions:
    - linux
    timeout: 10
    total_state_change: 63
    ttl: 0
  entity:
    deregister: false
    deregistration: {}
    entity_class: agent
    last_seen: 1596299033
    metadata:
      name: amy
      namespace: default
    redact:
    - password
    - passwd
    - pass
    - api_key
    - api_token
    - access_key
    - secret_key
    - private_key
    - secret
    sensu_agent_version: 5.21.0
    subscriptions:
    - linux
    - entity:amy
    system:
      arch: amd64
      cloud_provider: ""
      hostname: amy
      libc_type: glibc
      network:
        interfaces:
         [OMITED]
      os: linux
      platform: debian
      platform_family: debian
      platform_version: "10.5"
      processes: null
      vm_role: host
      vm_system: kvm
    user: agent
  id: d56dffb0-7a69-402a-a39f-087e721845a1
  timestamp: 1597220613
$ /usr/sbin/sensu-backend version
sensu-backend version 5.21.0+ee, enterprise edition, build 081a854d483d7881bbcf4cb60c44f87ea5fdf425, built 2020-06-11T19:45:19Z, built with go1.13.7

Agent and backend are at the same computer.

$ /usr/sbin/sensu-agent version
sensu-agent version 5.21.0+ee, enterprise edition, build 081a854d483d7881bbcf4cb60c44f87ea5fdf425, built 2020-06-11T19:48:44Z, built with go1.13.7

Note that next check DOES work:

$ sensuctl check info ping-google2 --format yaml
type: CheckConfig
api_version: core/v2
metadata:
  created_by: admin
  labels:
    sensu.io/managed_by: sensuctl
  name: ping-google2
  namespace: default
spec:
  check_hooks: null
  command: /var/cache/sensu/sensu-agent/774c60cf1f9ffeb54891f71dcb4df7fb8f78d8a2670e89695ec035f5e914bf90f1e8af9fcecccb8ab3c7ac8c34ccf345edc2bc657a4bbe46d5c3a4a768eba99a/bin/check_ping -H www.google.com -w 10,1% -c 20,2%
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 30
  low_flap_threshold: 0
  output_metric_format: nagios_perfdata
  output_metric_handlers: null
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - monitoring-plugins
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - linux
  timeout: 10
  ttl: 0

Thank you

palourde commented 4 years ago

Thanks for the provided information @davidvilla

You could also provide us the full configuration file for the agent running the amy entity?

Also, it would really help us if you could temporarily modify the log-level of the agent to debug, restart the agent and provide us the log entries of sensu-agent for the first few minutes.

davidvilla commented 4 years ago

Hi @palourde

the agent config file (only not commented lines):

---

subscriptions:
  - linux

backend-url:
  - "ws://127.0.0.1:8081"
log-level: "debug" # available log levels: panic, fatal, error, warn, info, debug

user: "agent"
password: "P@ssw0rd!"

keepalive-warning-timeout: 120
keepalive-interval: 20

agent output attached.

Cheers

sensu-agent.log

portertech commented 4 years ago

@davidvilla any chance SELINUX is enforcing some policy?

davidvilla commented 4 years ago

Hi @portertech,

SELINUX is not enabled in that host.

Cheers

palourde commented 4 years ago

Hi @davidvilla

I just realized that you specified the _runtimeassets monitoring-plugins in your check, but you installed the asset via sensuctl asset add sensu/monitoring-plugins, therefore the actual name of your asset should include the sensu/ prefix, e.g.:

runtime_assets:
- sensu/monitoring-plugins

It also looks like we don't emit any logs when this happen, so I'll make sure that we rectify that.

palourde commented 4 years ago

I'm gonna close this issue since I believe the PR above fixed that, but feel free to let us know if you still have this problem.