No points written on nmon2influxdb_log after config change

[sbengo]

Hi! I started using nmon2influxdb to get data from some host.

Playing with configuration, I changed the parameters different than defaults with the following ones and started to import nmons again:

dashboard_write_file = true
import_build_dashboard =  true

After this change, it seems that the agent stopped to read/write points of the timestamp and checksums measurements on db nmon2influxdb_log, so the agent was parsing and sending the same info whole time.

On the log it appears to do the following query:

https://github.com/adejoux/influxdbclient/blob/4b7a64a3cbe0e19fe7ac08f6ae1df3c393ce9dda/api.go#L279-L302

SELECT last(value) FROM nmon2influxdb_log.log_retention.timestamp WHERE file = 'HOST_YYMMDD_0000.nmon'
SELECT last(value) FROM nmon2influxdb_log.log_retention.checksum WHERE file = 'HOST_YYMMDD_0000.nmon'

And it seems that no error is propagated on the query when the result is 0, so all the checks are false and it keeps adding the data. https://github.com/adejoux/nmon2influxdb/blob/55c83fe089b55141d9038474f5d7e1f8d1654d69/nmon/import.go#L84-L105

We didn't see anything strange on the log and with lack of logging I'm not pretty sure what could be wrong. Regenerating the whole config solved the problem and it started to write points again on the log db nmon2influxdb_log.

[adejoux]

Hello,

Thanks for the detailed description. I will test it and in any case update the log output to catch this kind of issue.

Regards,

Alain

[adejoux]

Hello,

Can you try this attached version?

For me, it seems to work properly here:

nmon2influxdb --debug import ~/server1_171107_0000.nmon
2017/11/07 14:30:40 Using configuration file /home/adejoux/.nmon2influxdb.cfg
2017/11/07 14:30:40 databases: [_internal nmon_reports nmon2influxdb_log nmon2influxdbHMC collectd telegraf vmware_performance aix_stats jmxDB jmxDB2]
2017/11/07 14:30:40 databases: [_internal nmon_reports nmon2influxdb_log nmon2influxdbHMC collectd telegraf vmware_performance aix_stats jmxDB jmxDB2]
2017/11/07 14:30:40 UpdateRetentionPolicy: ALTER RETENTION POLICY "log_retention" ON "nmon2influxdb_log" DURATION 2d REPLICATION 1 DEFAULT
2017/11/07 14:30:40 configuration: {Debug:true DebugFile: Timezone:Europe/Paris InfluxdbUser:secretuser InfluxdbPassword:secret InfluxdbServer:influxsrv InfluxdbPort:8086 InfluxdbSecure:false InfluxdbSkipCertCheck:false InfluxdbDatabase:nmon_reports GrafanaUser:secretuser GrafanaPassword:secret GrafanaURL:https://grafanasrv GrafanaAccess:direct GrafanaDatasource:nmon2influxdb HMCServer: HMCUser:secretuser HMCPassword:secret HMCDatabase:nmon2influxdbHMC2 HMCDataRetention:360d HMCManagedSystem: HMCManagedSystemOnly:false HMCSamples:0 ImportSkipDisks:false ImportAllCpus:false ImportBuildDashboard:true ImportForce:false ImportSkipMetrics:JFSINODE|PCPU|TOP ImportLogDatabase:nmon2influxdb_log ImportLogRetention:2d ImportDataRetention: ImportSSHUser:jenkins ImportSSHKey:/home/user1/.ssh/id_rsa DashboardWriteFile:true StatsLimit:0 StatsSort:mean StatsFilter: StatsFrom: StatsTo: StatsHost: Metric: ListFilter: ListHost: Inputs:[]}
2017/11/07 14:30:40 Adding serie CPU_ALL
2017/11/07 14:30:40 Adding serie DISKBSIZE
2017/11/07 14:30:40 Adding serie DISKBUSY
2017/11/07 14:30:40 Adding serie DISKREAD
2017/11/07 14:30:40 Adding serie DISKRXFER
2017/11/07 14:30:40 Adding serie DISKSERV
2017/11/07 14:30:40 Adding serie DISKWAIT
2017/11/07 14:30:41 Adding serie DISKWRITE
2017/11/07 14:30:41 Adding serie DISKXFER
2017/11/07 14:30:41 Adding serie FCREAD
2017/11/07 14:30:41 Adding serie FCWRITE
2017/11/07 14:30:41 Adding serie FCXFERIN
2017/11/07 14:30:41 Adding serie FCXFEROUT
2017/11/07 14:30:41 Adding serie FILE
2017/11/07 14:30:41 Adding serie IOADAPT
2017/11/07 14:30:41 Adding serie JFSFILE
2017/11/07 14:30:41 Adding serie LPAR
2017/11/07 14:30:41 Adding serie MEM
2017/11/07 14:30:41 Adding serie MEMNEW
2017/11/07 14:30:41 Adding serie MEMUSE
2017/11/07 14:30:41 Adding serie NET
2017/11/07 14:30:41 Adding serie NETERROR
2017/11/07 14:30:41 Adding serie NETPACKET
2017/11/07 14:30:41 Adding serie NETSIZE
2017/11/07 14:30:41 Adding serie PAGE
2017/11/07 14:30:41 Adding serie POOLS
2017/11/07 14:30:41 Adding serie PROC
2017/11/07 14:30:41 Adding serie PROCAIO
2017/11/07 14:30:41 ERROR: parsing the following line : TOP,%CPU Utilisation
2017/11/07 14:30:41 Import file: /home/adejoux/server1_171107_0000.nmon
2017/11/07 14:30:41 NMON file separator: ,
2017/11/07 14:30:41 query: SELECT last("value") FROM "timestamp" WHERE "file" = 'server1_171107_0000.nmon'
2017/11/07 14:30:42 influxdb stored timestamp: 14:28:29,07-NOV-2017
2017/11/07 14:30:42 query: SELECT last("value") FROM "checksum" WHERE "file" = 'server1_171107_0000.nmon'
2017/11/07 14:30:42 influxdb stored checksum: 8992d456990fa4bd1e32a629e68c48df35e5d417
2017/11/07 14:30:42 metric skipped : JFSINODE
2017/11/07 14:30:42 metric skipped : TOP

File /home/adejoux/server1_171107_0000.nmon imported : 0 points !
2017/11/07 14:30:43 configuration: {Debug:true DebugFile: Timezone:Europe/Paris InfluxdbUser:secretuser InfluxdbPassword:secret InfluxdbServer:influxsrv InfluxdbPort:8086 InfluxdbSecure:false InfluxdbSkipCertCheck:false InfluxdbDatabase:nmon_reports GrafanaUser:secretuser GrafanaPassword:secret GrafanaURL:https://grafanasrv GrafanaAccess:direct GrafanaDatasource:nmon2influxdb HMCServer: HMCUser:secretuser HMCPassword:secret HMCDatabase:nmon2influxdbHMC2 HMCDataRetention:360d HMCManagedSystem: HMCManagedSystemOnly:false HMCSamples:0 ImportSkipDisks:false ImportAllCpus:false ImportBuildDashboard:true ImportForce:false ImportSkipMetrics:JFSINODE|PCPU|TOP ImportLogDatabase:nmon2influxdb_log ImportLogRetention:2d ImportDataRetention: ImportSSHUser:jenkins ImportSSHKey:/home/user1/.ssh/id_rsa DashboardWriteFile:true StatsLimit:0 StatsSort:mean StatsFilter: StatsFrom: StatsTo: StatsHost: Metric: ListFilter: ListHost: Inputs:[]}
2017/11/07 14:30:43 Adding serie CPU_ALL
2017/11/07 14:30:43 Adding serie DISKBSIZE
2017/11/07 14:30:43 Adding serie DISKBUSY
2017/11/07 14:30:43 Adding serie DISKREAD
2017/11/07 14:30:43 Adding serie DISKRXFER
2017/11/07 14:30:43 Adding serie DISKSERV
2017/11/07 14:30:44 Adding serie DISKWAIT
2017/11/07 14:30:44 Adding serie DISKWRITE
2017/11/07 14:30:44 Adding serie DISKXFER
2017/11/07 14:30:44 Adding serie FCREAD
2017/11/07 14:30:44 Adding serie FCWRITE
2017/11/07 14:30:44 Adding serie FCXFERIN
2017/11/07 14:30:44 Adding serie FCXFEROUT
2017/11/07 14:30:44 Adding serie FILE
2017/11/07 14:30:44 Adding serie IOADAPT
2017/11/07 14:30:44 Adding serie JFSFILE
2017/11/07 14:30:44 Adding serie LPAR
2017/11/07 14:30:44 Adding serie MEM
2017/11/07 14:30:44 Adding serie MEMNEW
2017/11/07 14:30:44 Adding serie MEMUSE
2017/11/07 14:30:44 Adding serie NET
2017/11/07 14:30:44 Adding serie NETERROR
2017/11/07 14:30:44 Adding serie NETPACKET
2017/11/07 14:30:44 Adding serie NETSIZE
2017/11/07 14:30:44 Adding serie PAGE
2017/11/07 14:30:44 Adding serie POOLS
2017/11/07 14:30:44 Adding serie PROC
2017/11/07 14:30:44 Adding serie PROCAIO
2017/11/07 14:30:44 ERROR: parsing the following line : TOP,%CPU Utilisation
2017/11/07 14:30:44 Writing GRAFANA dashboard: server1_dashboard

nmon2influxdb.gz