Nothing in the Splunk index

alithhero commented 1 year ago

Can somebody help me with this problem? I see nothing significant in the em_logs index that can explain me why I see nothing in the netops index ... Here is my value.yaml file : All XXXXX are properly completed in my configuration.

splunk:
  enabled: true
  protocol: http
  host: XXXXXX
  token: XXXXXXXX
  insecureSSL: "true"
  port: "8088"
  eventIndex: "myindex_netops"
  metricsIndex: "myindex_netmetrics"

image:
  pullPolicy: "Always"
traps:
  communities:
    2c:
      - public
      - homelab
  usernameSecrets:
    - XXXXXX
  #  - sc4snmp-hlab-sha-des

  #loadBalancerIP: The IP address in the metallb pool
  loadBalancerIP: 10.128.149.27
worker:
  # There are 3 types of workers 
  trap:
    # replicaCount: number of trap-worker pods which consumes trap tasks
    replicaCount: 2
    #autoscaling: use it instead of replicaCount in order to make pods scalable by itself
    #autoscaling:
    #  enabled: true
    #  minReplicas: 2
    #  maxReplicas: 10
    #  targetCPUUtilizationPercentage: 80
  poller:
    # replicaCount: number of poller-worker pods which consumes polling tasks
    replicaCount: 2
    #autoscaling: use it instead of replicaCount in order to make pods scalable by itself
    #autoscaling:
    #  enabled: true
    #  minReplicas: 2
    #  maxReplicas: 10
    #  targetCPUUtilizationPercentage: 80
  sender:
    # replicaCount: number of sender-worker pods which consumes sending tasks
    replicaCount: 1
    # autoscaling: use it instead of replicaCount in order to make pods scalable by itself
    #autoscaling:
    #  enabled: true
    #  minReplicas: 2
    #  maxReplicas: 10
    #  targetCPUUtilizationPercentage: 80
  # udpConnectionTimeout: timeout in seconds for SNMP operations
  #udpConnectionTimeout: 5
  logLevel: "INFO"
scheduler:
  logLevel: "INFO"
  profiles: |
    generic_switch:
      frequency: 300
      varBinds:
        - ['SNMPv2-MIB', 'sysDescr']
        - ['SNMPv2-MIB', 'sysName', 0]
        - ['TCP-MIB', 'tcpActiveOpens']
        - ['TCP-MIB', 'tcpAttemptFails']
        - ['IF-MIB']
  groups: |
    ciusss_group_commutateur:
      - address: 10.112.104.5
      - address: 10.130.44.1
      - address: 10.130.32.9
      - address: 10.130.32.8
      - address: 10.130.32.1
      - address: 10.130.32.6
      - address: 10.130.32.7
      - address: 10.130.40.9
      - address: 10.130.40.1
      - address: 10.130.40.6
      - address: 10.130.40.10
      - address: 10.130.40.8
      - address: 10.130.40.7
      - address: 10.132.92.1
      - address: 10.132.92.5
      - address: 10.132.92.6
      - address: 10.48.180.1
      - address: 10.133.0.136
      - address: 10.133.0.134
      - address: 10.133.0.1
      - address: 10.133.0.139
      - address: 10.133.0.131
      - address: 10.133.0.132
      - address: 10.133.0.143
      - address: 10.133.0.144
      - address: 10.133.0.145
      - address: 10.133.0.133
      - address: 10.133.0.140
      - address: 10.133.0.148
      - address: 10.133.0.146
      - address: 10.133.0.135
      - address: 10.133.0.138
      - address: 10.133.4.1
      - address: 10.133.8.1
      - address: 10.133.16.1
      - address: 10.133.24.1
      - address: 10.133.20.1
      - address: 10.112.88.1
      - address: 10.49.168.5
      - address: 10.49.168.6
      - address: 10.112.136.1
      - address: 10.130.192.5
      - address: 10.154.69.5
      - address: 10.132.136.5
      - address: 10.132.136.6
      - address: 10.130.164.6
      - address: 10.132.144.1
      - address: 10.133.64.15
      - address: 10.133.64.5
      - address: 10.133.64.1
      - address: 10.133.64.6
      - address: 10.133.64.8
      - address: 10.133.64.13
      - address: 10.132.156.5
      - address: 10.132.184.5
      - address: 10.130.176.5
      - address: 10.130.132.1
      - address: 10.132.140.6
      - address: 10.133.140.5
      - address: 10.131.60.8
      - address: 10.131.60.1
      - address: 10.131.60.9
      - address: 10.130.208.1
      - address: 10.130.180.7
      - address: 10.130.180.6
      - address: 10.133.76.1
      - address: 10.133.76.11
      - address: 10.133.76.12
      - address: 10.133.88.1
      - address: 10.133.84.1
      - address: 10.132.128.8
      - address: 10.132.128.1
      - address: 10.132.128.7
      - address: 10.132.128.9
      - address: 10.133.96.1
      - address: 10.133.128.5
      - address: 10.130.168.1
      - address: 10.133.100.1
      - address: 10.130.160.5
      - address: 10.131.4.6
      - address: 10.130.204.5
      - address: 10.130.240.5
      - address: 10.133.92.1
      - address: 10.131.52.5
      - address: 10.133.104.1
      - address: 10.130.128.5
      - address: 10.133.116.1
      - address: 10.131.20.5
      - address: 10.140.64.97
      - address: 10.140.64.102
      - address: 10.140.64.106
      - address: 10.140.64.123
      - address: 10.140.64.114
      - address: 10.140.64.98
      - address: 10.140.64.99
      - address: 10.140.68.34
      - address: 10.140.68.33
      - address: 10.140.68.35
      - address: 10.140.68.36
      - address: 10.49.100.1
      - address: 10.134.128.1
      - address: 10.134.128.6
      - address: 10.53.166.150
      - address: 10.53.166.66
      - address: 10.53.166.68
      - address: 10.53.166.70
      - address: 10.53.166.72
      - address: 10.53.166.74
      - address: 10.53.166.76
      - address: 10.53.166.78
      - address: 10.53.166.56
      - address: 10.53.166.58
      - address: 10.53.166.62
      - address: 10.53.166.64
      - address: 10.53.166.54
      - address: 10.53.166.52
      - address: 10.53.166.46
      - address: 10.53.166.48
      - address: 10.53.166.50
      - address: 10.53.166.80
      - address: 10.53.166.104
      - address: 10.53.166.106
      - address: 10.53.166.90
      - address: 10.53.166.92
      - address: 10.53.166.94
      - address: 10.53.166.96
      - address: 10.53.166.98
      - address: 10.53.166.100
      - address: 10.53.166.102
      - address: 10.53.166.82
      - address: 10.53.166.83
      - address: 10.53.166.84
      - address: 10.53.166.86
      - address: 10.53.166.81
      - address: 10.53.166.114
      - address: 10.53.166.118
      - address: 10.53.166.120
      - address: 10.53.166.122
      - address: 10.53.166.110
      - address: 10.53.166.124
      - address: 10.53.166.126
      - address: 10.53.166.128
      - address: 10.53.166.130
      - address: 10.53.166.132
      - address: 10.53.166.108
      - address: 10.53.166.151
      - address: 10.53.166.152
      - address: 10.53.166.103
      - address: 10.53.166.112
      - address: 10.53.166.109
      - address: 10.53.166.42
      - address: 10.53.166.40
      - address: 10.53.166.138
      - address: 10.53.166.140
      - address: 10.53.166.142
      - address: 10.53.166.144
      - address: 10.53.166.146
      - address: 10.53.166.136
      - address: 10.53.166.134
      - address: 10.134.224.19
      - address: 10.134.224.17
      - address: 10.134.224.16
      - address: 10.134.224.5
      - address: 10.134.224.14
      - address: 10.134.224.15
      - address: 10.134.224.11
      - address: 10.134.224.12
      - address: 10.134.224.18
      - address: 10.134.224.20
      - address: 10.134.224.7
      - address: 10.134.224.6
      - address: 10.134.224.4
      - address: 10.134.224.21
      - address: 10.134.224.22
      - address: 10.134.224.23
      - address: 10.133.12.1
      - address: 10.140.160.8
      - address: 10.140.160.7
      - address: 10.140.160.6
      - address: 10.140.160.24
      - address: 10.140.160.22
      - address: 10.140.160.23
      - address: 10.140.160.21
      - address: 10.140.160.17
      - address: 10.140.160.9
      - address: 10.140.160.11
      - address: 10.140.160.12
      - address: 10.140.160.13
      - address: 10.140.160.15
      - address: 10.140.160.18
      - address: 10.140.160.14
      - address: 10.140.160.16
      - address: 10.140.160.10
      - address: 10.140.160.31
      - address: 10.140.160.29
      - address: 10.140.160.28
      - address: 10.140.160.30
      - address: 10.140.160.27
      - address: 10.140.160.25
      - address: 10.140.160.26
      - address: 10.140.160.19
      - address: 10.140.160.20
      - address: 10.140.160.33
      - address: 10.140.160.34
      - address: 10.51.0.7
      - address: 10.51.0.8
      - address: 10.51.0.6
      - address: 10.51.0.9
      - address: 10.51.0.1
      - address: 10.51.0.10
      - address: 10.51.0.21
      - address: 10.51.0.20
      - address: 10.133.36.1
      - address: 10.133.36.7
      - address: 10.133.36.6
      - address: 10.139.4.1
      - address: 10.139.4.6
      - address: 10.139.4.8
      - address: 10.139.4.9
      - address: 10.139.4.7
      - address: 10.133.32.1
      - address: 10.133.32.7
      - address: 10.133.32.6
      - address: 10.136.100.1
      - address: 10.141.32.1
      - address: 10.141.32.6
      - address: 10.141.32.7
      - address: 10.141.32.8
      - address: 10.141.36.1
      - address: 10.139.0.1
      - address: 10.139.0.6
      - address: 10.139.0.7
      - address: 10.139.0.8
      - address: 10.51.12.5
      - address: 10.51.12.1
      - address: 10.51.12.6
      - address: 10.51.12.7
      - address: 10.141.40.1
      - address: 10.141.40.3
      - address: 10.141.40.4
      - address: 10.141.40.6
      - address: 10.51.160.6
      - address: 10.51.160.1
      - address: 10.51.160.7
      - address: 10.51.160.8
      - address: 10.51.160.9
      - address: 10.51.164.1
      - address: 10.143.128.3
      - address: 10.143.128.5
      - address: 10.143.128.7
      - address: 10.143.128.6
      - address: 10.143.128.4
      - address: 10.143.128.8
      - address: 10.143.128.1
      - address: 10.51.4.1
      - address: 10.48.127.5
      - address: 10.51.16.1
      - address: 10.48.156.44
      - address: 10.53.20.1
      - address: 10.112.64.10
      - address: 10.112.64.15
      - address: 10.112.64.7
      - address: 10.112.118.4
      - address: 10.112.64.6
      - address: 10.112.64.8
      - address: 10.112.64.1
      - address: 10.112.64.12
      - address: 10.112.65.12
      - address: 10.112.65.14
      - address: 10.112.66.124
      - address: 10.112.66.14
      - address: 10.112.118.7
      - address: 10.112.118.5
      - address: 10.112.240.1
      - address: 10.112.156.5
      - address: 10.112.76.8
      - address: 10.112.76.5
      - address: 10.112.76.6
      - address: 10.112.76.7
      - address: 10.112.148.1
      - address: 10.112.72.9
      - address: 10.112.72.6
      - address: 10.112.72.7
      - address: 10.112.72.8
      - address: 10.112.72.1
      - address: 10.112.68.9
      - address: 10.112.68.8
      - address: 10.112.68.7
      - address: 10.112.68.1
      - address: 10.112.68.6
      - address: 10.112.68.10
      - address: 10.112.68.11
      - address: 10.112.68.12
      - address: 10.112.68.5
      - address: 10.112.152.1
      - address: 10.112.108.1
      - address: 10.113.0.1
      - address: 10.113.0.6
      - address: 10.113.0.7
      - address: 10.112.84.10
      - address: 10.112.84.8
      - address: 10.112.84.7
      - address: 10.112.84.9
      - address: 10.112.84.1
      - address: 10.112.84.6
      - address: 10.112.96.10
      - address: 10.112.96.9
      - address: 10.112.96.6
      - address: 10.112.96.11
      - address: 10.112.96.8
      - address: 10.112.96.7
      - address: 10.112.96.1
      - address: 10.112.96.12
      - address: 10.112.81.7
      - address: 10.112.80.9
      - address: 10.112.81.5
      - address: 10.112.82.14
      - address: 10.112.81.6
      - address: 10.112.80.1
      - address: 10.112.82.15
      - address: 10.112.82.10
      - address: 10.112.82.11
      - address: 10.112.82.12
      - address: 10.112.82.13
      - address: 10.112.82.6
      - address: 10.112.82.7
      - address: 10.112.82.8
      - address: 10.112.82.9
      - address: 10.112.80.20
      - address: 10.112.80.7
      - address: 10.112.80.11
      - address: 10.112.92.7
      - address: 10.112.92.11
      - address: 10.112.92.9
      - address: 10.112.92.10
      - address: 10.112.92.8
      - address: 10.112.92.6
      - address: 10.112.92.1
      - address: 10.112.132.1
      - address: 10.112.112.5
      - address: 10.112.112.1
      - address: 10.112.196.1
      - address: 10.112.184.1
      - address: 10.112.208.1
      - address: 10.112.248.1
      - address: 10.112.216.1
      - address: 10.112.220.1
      - address: 10.112.164.5
      - address: 10.112.192.1
      - address: 10.112.236.1
      - address: 10.112.204.1
      - address: 10.112.224.1
      - address: 10.112.168.1
      - address: 10.112.160.1
      - address: 10.112.172.1
      - address: 10.112.228.1
      - address: 10.112.212.1
      - address: 10.112.176.1
      - address: 10.112.140.1
      - address: 10.112.128.1
      - address: 10.112.100.7
      - address: 10.112.100.6
      - address: 10.112.100.1
      - address: 10.130.68.4
      - address: 10.130.68.1
      - address: 10.137.164.5
      - address: 10.137.164.1
      - address: 10.137.164.12
      - address: 10.130.64.1
      - address: 10.130.64.5
      - address: 10.130.64.13
      - address: 10.137.160.1
      - address: 10.137.160.6
      - address: 10.139.232.1
      - address: 10.139.228.1
      - address: 10.139.228.5
      - address: 10.139.224.1
      - address: 10.139.224.130
      - address: 10.48.196.1
      - address: 10.50.224.1
      - address: 10.20.224.6
      - address: 10.50.224.6
      - address: 10.51.64.2
      - address: 10.51.64.4
      - address: 10.51.64.6
      - address: 10.51.64.5
      - address: 10.51.64.3
      - address: 10.51.64.1
      - address: 10.51.64.8
      - address: 10.51.64.9
      - address: 10.51.68.1
      - address: 10.51.68.6
      - address: 10.130.104.1
      - address: 10.131.97.191
      - address: 10.131.97.192
      - address: 10.131.97.181
      - address: 10.131.97.180
      - address: 10.131.120.5

poller:
  usernameSecrets:
    - XXXXXX
 #   - sc4snmp-hlab-sha-des
  inventory: |
    address,port,version,community,secret,security_engine,walk_interval,profiles,smart_profiles,delete
    ciusss_group_commutateur,,3,,XXXXX,,1800,,,
sim:
  # sim must be enabled if you want to use signalFx
  enabled: false
#  signalfxToken: BCwaJ_Ands4Xh7Nrg
#  signalfxRealm: us0
mongodb:
  pdb:
    create: true
  persistence:
    storageClass: "microk8s-hostpath"
  volumePermissions:
    enabled: true

omrozowicz-splunk commented 1 year ago

Ok, so the problem is that you have no data in Splunk at all? Both myindex_netops and myindex_netmetrics? Did it work with smaller number of devices or you've never tried it? Also, I know you mentioned that logs look fine, but do you have any logs: An error of SNMP isWalk in em_logs?

The only thing that is coming to my mind without knowing any context is that the devices are very big and SC4SNMP cannot walk it before it reaches the timeout. You can configure walk profile to limit the number of data -> https://splunk.github.io/splunk-connect-for-snmp/main/configuration/configuring-profiles/#walk-profile. In such a case you would see many Task splunk_connect_for_snmp.splunk.tasks.walk in logs, but nothing more.

This could be also a problem with snmpv3 secret, but I think If the secret is badly configured, you would see some Error in logs.

Last thing - when you run a helm upgrade command and then microk8s kubectl logs -f job/snmp-splunk-connect-for-snmp-inventory -n sc4snmp, what does it say? You should see many New record or Unchanged record.

alithhero commented 1 year ago

First of all thanks for your help. I just tried with a smaller number of device (7). I got data for 5/7 devices in my personal index, so it left two devices who didn't send data. All those devices are in snmpv3, so I can say that my secret configuration is correct. I see nothing with An error of SNMP isWalk in the em_logs index ...

When you say "configure walk profile to limit the number of data" you mean, increase the walk interval, be more specific in the varBinds section or configure a small_walk ?

Yes, I can see a Unchanged record for my 7 devices in the inventory logs.

omrozowicz-splunk commented 1 year ago

I meant be more specific in the varBinds section or configure a small_walk, if there's no An error of SNMP isWalk that means walk have a hard time to finish itself and that might mean the scope of walk is too big. Overall, it is good idea to limit walk scope as you usually don't need to poll all the data. Try to create such a profile and add it to inventory, and let me know if anything has changed.

alithhero commented 1 year ago

I think it is a part of the the problem, when I only use those two varBinds it works for my 300 appliances : ['SNMPv2-MIB', 'sysName'] ['SNMPv2-MIB', 'sysUpTime',0]

But data returned are not enough for my client so after analysed the MIBS, I would like to pull those varBinds :


          ['SNMPv2-MIB', 'sysUpTime',0]
          ['SNMPv2-MIB', 'sysLocation']
          ['SNMPv2-MIB', 'sysDescr']
          ['IF-MIB', 'ifIndex']
          ['IF-MIB', 'ifDescr']
          ['IF-MIB', 'ifType']
          ['IF-MIB', 'ifSpeed']
          ['IF-MIB', 'ifAdminStatus']
          ['IF-MIB', 'ifInOctets']
          ['IF-MIB', 'ifOutOctets']```
And again, nothing comes to Splunk Index and nothing significant in the `em_log` index ...

wojtekzyla commented 1 year ago

Could you please check if you have any data in the myindex_netmetrics index, provided that you used varBinds which you mentioned? Some of those varBinds are sent to the Metric index https://splunk.github.io/splunk-connect-for-snmp/main/configuration/snmp-data-format/

splunk / splunk-connect-for-snmp

Nothing in the Splunk index #707