search

opendistro-for-elasticsearch / performance-analyzer

📈 OpenDistro Performance Analyzer
https://opendistro.github.io/
Apache License 2.0
146 stars 49 forks source link

sqlite crashes #238

Open opoplawski opened 4 years ago

opoplawski commented 4 years ago

We're seeing crashes like these fairly regularly (a few times a day):

Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: 07:58:00.615 [SF1wX1Q8QWm3JH9VAGYz-w-task-0-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.framework.api.persist.SQLParsingUtil - Eden fails to match any row in field MemType.
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: 07:58:00.616 [SF1wX1Q8QWm3JH9VAGYz-w-task-0-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.framework.api.persist.SQLParsingUtil - Survivor fails to match any row in field MemType.
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: 07:58:00.616 [SF1wX1Q8QWm3JH9VAGYz-w-task-0-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.store.collector.NodeConfigCollector - Metric value is NaN for resource:resource_enum: YOUNG_GEN
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: metric_enum: HEAP_MAX
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: Nov 11, 2020 7:58:00 AM org.jooq.tools.JooqLogger info
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: Nov 11, 2020 7:58:00 AM org.jooq.tools.JooqLogger info
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: Nov 11, 2020 7:58:00 AM org.jooq.tools.JooqLogger info
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: 07:58:00.946 [pa-reader] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ReaderMetricsProcessor - READER PROCESSOR ERROR. NEEDS DEBUGGING Other ExceptionCode: org.jooq.exception.DataAccessException: SQL [select IndexName, ShardID, sum(Cache_Request_Hit) sum_Cache_Request_Hit, avg(Cache_Request_Hit) avg_Cache_Request_Hit, min(Cache_Request_Hit) min_Cache_Request_Hit, max(Cache_Request_Hit) max_Cache_Request_Hit from SHARD_STATS1605110250000 where Cache_Request_Hit is not null group by IndexName, ShardID]; [SQLITE_ERROR] SQL error or missing database (no such table: SHARD_STATS1605110250000).
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: org.jooq.exception.DataAccessException: SQL [select IndexName, ShardID, sum(Cache_Request_Hit) sum_Cache_Request_Hit, avg(Cache_Request_Hit) avg_Cache_Request_Hit, min(Cache_Request_Hit) min_Cache_Request_Hit, max(Cache_Request_Hit) max_Cache_Request_Hit from SHARD_STATS1605110250000 where Cache_Request_Hit is not null group by IndexName, ShardID]; [SQLITE_ERROR] SQL error or missing database (no such table: SHARD_STATS1605110250000)
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.jooq_3.10.8.SQLITE.debug(Unknown Source) ~[?:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.jooq.impl.Tools.translate(Tools.java:2241) ~[jooq-3.10.8.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:704) ~[jooq-3.10.8.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:361) ~[jooq-3.10.8.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:317) ~[jooq-3.10.8.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.jooq.impl.SelectImpl.fetch(SelectImpl.java:2597) ~[jooq-3.10.8.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.MetricsEmitter.emitNodeMetrics(MetricsEmitter.java:818) ~[performance-analyzer-rca-1.10.0-SNAPSHOT.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ReaderMetricsProcessor.emitNodeMetrics(ReaderMetricsProcessor.java:888) ~[performance-analyzer-rca-1.10.0-SNAPSHOT.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ReaderMetricsProcessor.emitMetrics(ReaderMetricsProcessor.java:389) ~[performance-analyzer-rca-1.10.0-SNAPSHOT.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ReaderMetricsProcessor.processMetrics(ReaderMetricsProcessor.java:591) ~[performance-analyzer-rca-1.10.0-SNAPSHOT.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ReaderMetricsProcessor.run(ReaderMetricsProcessor.java:179) [performance-analyzer-rca-1.10.0-SNAPSHOT.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp.lambda$startReaderThread$2(PerformanceAnalyzerApp.java:234) [performance-analyzer-rca-1.10.0-SNAPSHOT.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at com.amazon.opendistro.elasticsearch.performanceanalyzer.threads.ThreadProvider.lambda$createThreadForRunnable$0(ThreadProvider.java:53) [performance-analyzer-rca-1.10.0-SNAPSHOT.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at java.lang.Thread.run(Thread.java:832) [?:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: Caused by: java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (no such table: SHARD_STATS1605110250000)
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.sqlite.core.DB.newSQLException(DB.java:890) ~[sqlite-jdbc-3.8.11.2.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.sqlite.core.DB.newSQLException(DB.java:901) ~[sqlite-jdbc-3.8.11.2.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.sqlite.core.DB.throwex(DB.java:868) ~[sqlite-jdbc-3.8.11.2.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.sqlite.core.NativeDB.prepare(Native Method) ~[sqlite-jdbc-3.8.11.2.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.sqlite.core.DB.prepare(DB.java:211) ~[sqlite-jdbc-3.8.11.2.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.sqlite.core.CorePreparedStatement.<init>(CorePreparedStatement.java:40) ~[sqlite-jdbc-3.8.11.2.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.sqlite.jdbc3.JDBC3PreparedStatement.<init>(JDBC3PreparedStatement.java:29) ~[sqlite-jdbc-3.8.11.2.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.sqlite.jdbc4.JDBC4PreparedStatement.<init>(JDBC4PreparedStatement.java:18) ~[sqlite-jdbc-3.8.11.2.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.sqlite.jdbc4.JDBC4Connection.prepareStatement(JDBC4Connection.java:47) ~[sqlite-jdbc-3.8.11.2.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.sqlite.jdbc3.JDBC3Connection.prepareStatement(JDBC3Connection.java:254) ~[sqlite-jdbc-3.8.11.2.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.sqlite.jdbc3.JDBC3Connection.prepareStatement(JDBC3Connection.java:226) ~[sqlite-jdbc-3.8.11.2.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.jooq.impl.ProviderEnabledConnection.prepareStatement(ProviderEnabledConnection.java:109) ~[jooq-3.10.8.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.jooq.impl.SettingsEnabledConnection.prepareStatement(SettingsEnabledConnection.java:73) ~[jooq-3.10.8.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.jooq.impl.AbstractResultQuery.prepare(AbstractResultQuery.java:233) ~[jooq-3.10.8.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:319) ~[jooq-3.10.8.jar:?]
Nov 11 07:58:00 host performance-analyzer-agent-cli[19637]: ... 10 more
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 07:58:05.601 [pa-reader] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ReaderMetricsProcessor - Connection to the database was closed.
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: *** Error in `/usr/share/elasticsearch/jdk/bin/java': corrupted double-linked list: 0x00007fbf5c0035e0 ***
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: *** Error in `/usr/share/elasticsearch/jdk/bin/java': corrupted double-linked list: 0x00007fbf58002ad0 ***
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 07:58:05.602 [pa-reader] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp - Error in ReaderMetricsProcessor...restarting, ExceptionCode: ReaderRestartProcessing
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: ======= Backtrace: =========
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: ======= Backtrace: =========
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: /lib64/libc.so.6(+0x7f3e4)[0x7fbfde9123e4]
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: /lib64/libc.so.6(+0x82ba8)[0x7fbfde915ba8]
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: /lib64/libc.so.6(__libc_malloc+0x4c)[0x7fbfde9186fc]
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: /usr/share/elasticsearch/jdk/lib/server/libjvm.so(+0x380700)[0x7fbfdd668700]
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: /usr/share/elasticsearch/jdk/lib/server/libjvm.so(+0x5d6afb)[0x7fbfdd8beafb]
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: [0x7fbfcbc31da1]
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: ======= Memory map: ========
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: fc000000-100000000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 800000000-800003000 rwxp 00001000 08:02 268509836                        /usr/share/elasticsearch/jdk/lib/server/classes.jsa
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 800003000-8003e3000 rw-p 00004000 08:02 268509836                        /usr/share/elasticsearch/jdk/lib/server/classes.jsa
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 8003e3000-800b0e000 r--p 003e4000 08:02 268509836                        /usr/share/elasticsearch/jdk/lib/server/classes.jsa
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 800b0e000-800b0f000 rw-p 00b0f000 08:02 268509836                        /usr/share/elasticsearch/jdk/lib/server/classes.jsa
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 800b0f000-800f52000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 800f52000-840b0f000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 56541fcaa000-56541fcab000 r-xp 00000000 08:02 16850                      /usr/share/elasticsearch/jdk/bin/java
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 56541feab000-56541feac000 r--p 00001000 08:02 16850                      /usr/share/elasticsearch/jdk/bin/java
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 56541feac000-56541fead000 rw-p 00002000 08:02 16850                      /usr/share/elasticsearch/jdk/bin/java
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 5654209ce000-565420a06000 rw-p 00000000 00:00 0                          [heap]
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf50000000-7fbf50021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf50021000-7fbf54000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf54000000-7fbf54021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf54021000-7fbf58000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf58000000-7fbf580b0000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf580b0000-7fbf5c000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf5c000000-7fbf5c0ba000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf5c0ba000-7fbf60000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf60000000-7fbf60021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf60021000-7fbf64000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf64000000-7fbf64021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf64021000-7fbf68000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf68000000-7fbf68021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf68021000-7fbf6c000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf6c000000-7fbf6c021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf6c021000-7fbf70000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf70000000-7fbf70021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf70021000-7fbf74000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf74000000-7fbf74021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf74021000-7fbf78000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7a9e2000-7fbf7a9f7000 r-xp 00000000 08:02 32877                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7a9f7000-7fbf7abf6000 ---p 00015000 08:02 32877                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7abf6000-7fbf7abf7000 r--p 00014000 08:02 32877                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7abf7000-7fbf7abf8000 rw-p 00015000 08:02 32877                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7abf8000-7fbf7adb8000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7adb8000-7fbf7adf8000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7adf8000-7fbf7adfc000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7adfc000-7fbf7aef9000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7aef9000-7fbf7aefd000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7aefd000-7fbf7affa000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7affa000-7fbf7affe000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7affe000-7fbf7b0fb000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7b0fb000-7fbf7b0ff000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7b0ff000-7fbf7b1fc000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7b1fc000-7fbf7b200000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7b200000-7fbf7b2fd000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7b2fd000-7fbf7b301000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7b301000-7fbf7b3fe000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7b3fe000-7fbf7b402000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7b402000-7fbf7b6fa000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7b6fa000-7fbf7b6ff000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7b6ff000-7fbf7b703000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7b703000-7fbf7ba00000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7ba00000-7fbf7bc00000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7bc00000-7fbf7be00000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7be00000-7fbf7c000000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7c000000-7fbf7c1e0000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf7c1e0000-7fbf80000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf80000000-7fbf80021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf80021000-7fbf84000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf84000000-7fbf849f4000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf849f4000-7fbf88000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf88000000-7fbf88192000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf88192000-7fbf8c000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf8c000000-7fbf8c021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf8c021000-7fbf90000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf90000000-7fbf90021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf90021000-7fbf94000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf94000000-7fbf94021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf94021000-7fbf98000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf98000000-7fbf98021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf98021000-7fbf9c000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf9c000000-7fbf9c021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbf9c021000-7fbfa0000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfa0000000-7fbfa0021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfa0021000-7fbfa4000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfa4000000-7fbfa4021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfa4021000-7fbfa8000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfa8000000-7fbfa8021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfa8021000-7fbfac000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfac000000-7fbfae2e4000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfae2e4000-7fbfb0000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfb0000000-7fbfb0041000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfb0041000-7fbfb4000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfb4000000-7fbfb4021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfb4021000-7fbfb8000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfb8000000-7fbfb8021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfb8021000-7fbfbc000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfbc000000-7fbfbc021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfbc021000-7fbfc0000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc0000000-7fbfc0021000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc0021000-7fbfc4000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc4000000-7fbfc4173000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc4173000-7fbfc8000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8041000-7fbfc8045000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8045000-7fbfc8342000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8342000-7fbfc852e000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc852e000-7fbfc8542000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8542000-7fbfc8742000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8742000-7fbfc8751000 r-xp 00000000 00:27 2433115                    /tmp/libio_grpc_netty_shaded_netty_transport_native_epoll_x86_6413915815408964866165.so (deleted)
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8751000-7fbfc8950000 ---p 0000f000 00:27 2433115                    /tmp/libio_grpc_netty_shaded_netty_transport_native_epoll_x86_6413915815408964866165.so (deleted)
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8950000-7fbfc8952000 rw-p 0000e000 00:27 2433115                    /tmp/libio_grpc_netty_shaded_netty_transport_native_epoll_x86_6413915815408964866165.so (deleted)
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8952000-7fbfc8b52000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8b52000-7fbfc8c0b000 r-xp 00000000 00:27 2432075                    /tmp/sqlite-3.8.11.2-0f8ebf28-89df-4663-b153-c652ab63ca43-libsqlitejdbc.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8c0b000-7fbfc8e0b000 ---p 000b9000 00:27 2432075                    /tmp/sqlite-3.8.11.2-0f8ebf28-89df-4663-b153-c652ab63ca43-libsqlitejdbc.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8e0b000-7fbfc8e10000 rw-p 000b9000 00:27 2432075                    /tmp/sqlite-3.8.11.2-0f8ebf28-89df-4663-b153-c652ab63ca43-libsqlitejdbc.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc8e10000-7fbfc9010000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9010000-7fbfc9014000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9014000-7fbfc9111000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9111000-7fbfc9115000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9115000-7fbfc9212000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9212000-7fbfc9216000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9216000-7fbfc9313000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9313000-7fbfc9317000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9317000-7fbfc9414000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9414000-7fbfc9418000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9418000-7fbfc9515000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9515000-7fbfc9519000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9519000-7fbfc9616000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9616000-7fbfc9618000 r-xp 00000000 08:02 537615382                  /usr/share/elasticsearch/jdk/lib/libextnet.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9618000-7fbfc9817000 ---p 00002000 08:02 537615382                  /usr/share/elasticsearch/jdk/lib/libextnet.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9817000-7fbfc9818000 r--p 00001000 08:02 537615382                  /usr/share/elasticsearch/jdk/lib/libextnet.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9818000-7fbfc9819000 rw-p 00002000 08:02 537615382                  /usr/share/elasticsearch/jdk/lib/libextnet.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9819000-7fbfc9859000 r-xp 00000000 08:02 537903113                  /usr/share/elasticsearch/jdk/lib/libsunec.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9859000-7fbfc9a59000 ---p 00040000 08:02 537903113                  /usr/share/elasticsearch/jdk/lib/libsunec.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9a59000-7fbfc9a5e000 r--p 00040000 08:02 537903113                  /usr/share/elasticsearch/jdk/lib/libsunec.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9a5e000-7fbfc9a60000 rw-p 00045000 08:02 537903113                  /usr/share/elasticsearch/jdk/lib/libsunec.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9a60000-7fbfc9c60000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9c60000-7fbfc9c71000 r-xp 00000000 08:02 537903155                  /usr/share/elasticsearch/jdk/lib/libverify.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9c71000-7fbfc9e70000 ---p 00011000 08:02 537903155                  /usr/share/elasticsearch/jdk/lib/libverify.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9e70000-7fbfc9e72000 r--p 00010000 08:02 537903155                  /usr/share/elasticsearch/jdk/lib/libverify.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9e72000-7fbfc9e73000 rw-p 00012000 08:02 537903155                  /usr/share/elasticsearch/jdk/lib/libverify.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9e73000-7fbfc9e77000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9e77000-7fbfc9f74000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9f74000-7fbfc9f78000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfc9f78000-7fbfca075000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfca075000-7fbfca079000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfca079000-7fbfca376000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfca376000-7fbfca576000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfca576000-7fbfca5ab000 r--s 00000000 08:02 537380290                  /var/db/nscd/hosts
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfca5ab000-7fbfca7ab000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfca7ab000-7fbfca9ab000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfca9ab000-7fbfca9b1000 r-xp 00000000 08:02 537884803                  /usr/share/elasticsearch/jdk/lib/libmanagement_ext.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfca9b1000-7fbfcabb0000 ---p 00006000 08:02 537884803                  /usr/share/elasticsearch/jdk/lib/libmanagement_ext.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcabb0000-7fbfcabb1000 r--p 00005000 08:02 537884803                  /usr/share/elasticsearch/jdk/lib/libmanagement_ext.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcabb1000-7fbfcabb2000 rw-p 00006000 08:02 537884803                  /usr/share/elasticsearch/jdk/lib/libmanagement_ext.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcabb2000-7fbfcabb6000 r-xp 00000000 08:02 537619779                  /usr/share/elasticsearch/jdk/lib/libmanagement.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcabb6000-7fbfcadb5000 ---p 00004000 08:02 537619779                  /usr/share/elasticsearch/jdk/lib/libmanagement.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcadb5000-7fbfcadb6000 r--p 00003000 08:02 537619779                  /usr/share/elasticsearch/jdk/lib/libmanagement.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcadb6000-7fbfcadb7000 rw-p 00004000 08:02 537619779                  /usr/share/elasticsearch/jdk/lib/libmanagement.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcadb7000-7fbfcadcc000 r-xp 00000000 08:02 537884813                  /usr/share/elasticsearch/jdk/lib/libnet.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcadcc000-7fbfcafcb000 ---p 00015000 08:02 537884813                  /usr/share/elasticsearch/jdk/lib/libnet.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcafcb000-7fbfcafcc000 r--p 00014000 08:02 537884813                  /usr/share/elasticsearch/jdk/lib/libnet.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcafcc000-7fbfcafcd000 rw-p 00015000 08:02 537884813                  /usr/share/elasticsearch/jdk/lib/libnet.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcafcd000-7fbfcafde000 r-xp 00000000 08:02 537885285                  /usr/share/elasticsearch/jdk/lib/libnio.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcafde000-7fbfcb1dd000 ---p 00011000 08:02 537885285                  /usr/share/elasticsearch/jdk/lib/libnio.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb1dd000-7fbfcb1de000 r--p 00010000 08:02 537885285                  /usr/share/elasticsearch/jdk/lib/libnio.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb1de000-7fbfcb1df000 rw-p 00011000 08:02 537885285                  /usr/share/elasticsearch/jdk/lib/libnio.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb1df000-7fbfcb1e3000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb1e3000-7fbfcb2e0000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb2e0000-7fbfcb2e1000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb2e1000-7fbfcb3e2000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb3e2000-7fbfcb3e6000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb3e6000-7fbfcb4e3000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb4e3000-7fbfcb4e7000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb4e7000-7fbfcb5e4000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb5e4000-7fbfcb5e8000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb5e8000-7fbfcb6e5000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb6e5000-7fbfcb6e9000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb6e9000-7fbfcb7e6000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb7e6000-7fbfcb7ea000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb7ea000-7fbfcb8e7000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcb8e7000-7fbfcbae8000 r--p 00000000 08:02 537644825                  /usr/lib/locale/locale-archive
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcbae8000-7fbfcbaec000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcbaec000-7fbfcbbe9000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcbbe9000-7fbfcc10a000 rwxp 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcc10a000-7fbfcebe9000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfcebe9000-7fbfd8000000 r--s 00000000 08:02 537903959                  /usr/share/elasticsearch/jdk/lib/modules
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfd8000000-7fbfd87db000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfd87db000-7fbfdc000000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc01f000-7fbfdc050000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc050000-7fbfdc054000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc054000-7fbfdc151000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc151000-7fbfdc152000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc152000-7fbfdc453000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc453000-7fbfdc570000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc580000-7fbfdc587000 r-xp 00000000 08:02 537903158                  /usr/share/elasticsearch/jdk/lib/libzip.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc587000-7fbfdc786000 ---p 00007000 08:02 537903158                  /usr/share/elasticsearch/jdk/lib/libzip.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc786000-7fbfdc787000 r--p 00006000 08:02 537903158                  /usr/share/elasticsearch/jdk/lib/libzip.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc787000-7fbfdc788000 rw-p 00007000 08:02 537903158                  /usr/share/elasticsearch/jdk/lib/libzip.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc788000-7fbfdc7ac000 r-xp 00000000 08:02 537619775                  /usr/share/elasticsearch/jdk/lib/libjava.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc7ac000-7fbfdc9ab000 ---p 00024000 08:02 537619775                  /usr/share/elasticsearch/jdk/lib/libjava.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc9ab000-7fbfdc9ac000 r--p 00023000 08:02 537619775                  /usr/share/elasticsearch/jdk/lib/libjava.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc9ac000-7fbfdc9ae000 rw-p 00024000 08:02 537619775                  /usr/share/elasticsearch/jdk/lib/libjava.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc9ae000-7fbfdc9ba000 r-xp 00000000 08:02 95178                      /usr/lib64/libnss_files-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdc9ba000-7fbfdcbb9000 ---p 0000c000 08:02 95178                      /usr/lib64/libnss_files-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcbb9000-7fbfdcbba000 r--p 0000b000 08:02 95178                      /usr/lib64/libnss_files-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcbba000-7fbfdcbbb000 rw-p 0000c000 08:02 95178                      /usr/lib64/libnss_files-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcbbb000-7fbfdcbc1000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcbc1000-7fbfdcbdc000 r-xp 00000000 08:02 537619844                  /usr/share/elasticsearch/jdk/lib/libjimage.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcbdc000-7fbfdcddb000 ---p 0001b000 08:02 537619844                  /usr/share/elasticsearch/jdk/lib/libjimage.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcddb000-7fbfdcddd000 r--p 0001a000 08:02 537619844                  /usr/share/elasticsearch/jdk/lib/libjimage.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcddd000-7fbfdcdde000 rw-p 0001c000 08:02 537619844                  /usr/share/elasticsearch/jdk/lib/libjimage.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcdde000-7fbfdcde5000 r-xp 00000000 08:02 103716                     /usr/lib64/librt-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcde5000-7fbfdcfe4000 ---p 00007000 08:02 103716                     /usr/lib64/librt-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcfe4000-7fbfdcfe5000 r--p 00006000 08:02 103716                     /usr/lib64/librt-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcfe5000-7fbfdcfe6000 rw-p 00007000 08:02 103716                     /usr/lib64/librt-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdcfe6000-7fbfdd0e7000 r-xp 00000000 08:02 89404                      /usr/lib64/libm-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdd0e7000-7fbfdd2e6000 ---p 00101000 08:02 89404                      /usr/lib64/libm-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdd2e6000-7fbfdd2e7000 r--p 00100000 08:02 89404                      /usr/lib64/libm-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdd2e7000-7fbfdd2e8000 rw-p 00101000 08:02 89404                      /usr/lib64/libm-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdd2e8000-7fbfde530000 r-xp 00000000 08:02 268509839                  /usr/share/elasticsearch/jdk/lib/server/libjvm.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfde530000-7fbfde730000 ---p 01248000 08:02 268509839                  /usr/share/elasticsearch/jdk/lib/server/libjvm.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfde730000-7fbfde7d8000 r--p 01248000 08:02 268509839                  /usr/share/elasticsearch/jdk/lib/server/libjvm.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfde7d8000-7fbfde812000 rw-p 012f0000 08:02 268509839                  /usr/share/elasticsearch/jdk/lib/server/libjvm.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfde812000-7fbfde893000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfde893000-7fbfdea57000 r-xp 00000000 08:02 39708                      /usr/lib64/libc-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdea57000-7fbfdec56000 ---p 001c4000 08:02 39708                      /usr/lib64/libc-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdec56000-7fbfdec5a000 r--p 001c3000 08:02 39708                      /usr/lib64/libc-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdec5a000-7fbfdec5c000 rw-p 001c7000 08:02 39708                      /usr/lib64/libc-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdec5c000-7fbfdec61000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdec61000-7fbfdec63000 r-xp 00000000 08:02 87138                      /usr/lib64/libdl-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdec63000-7fbfdee63000 ---p 00002000 08:02 87138                      /usr/lib64/libdl-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdee63000-7fbfdee64000 r--p 00002000 08:02 87138                      /usr/lib64/libdl-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdee64000-7fbfdee65000 rw-p 00003000 08:02 87138                      /usr/lib64/libdl-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdee65000-7fbfdee7c000 r-xp 00000000 08:02 95595                      /usr/lib64/libpthread-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdee7c000-7fbfdf07b000 ---p 00017000 08:02 95595                      /usr/lib64/libpthread-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf07b000-7fbfdf07c000 r--p 00016000 08:02 95595                      /usr/lib64/libpthread-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf07c000-7fbfdf07d000 rw-p 00017000 08:02 95595                      /usr/lib64/libpthread-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf07d000-7fbfdf081000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf081000-7fbfdf090000 r-xp 00000000 08:02 537619864                  /usr/share/elasticsearch/jdk/lib/libjli.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf090000-7fbfdf290000 ---p 0000f000 08:02 537619864                  /usr/share/elasticsearch/jdk/lib/libjli.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf290000-7fbfdf291000 r--p 0000f000 08:02 537619864                  /usr/share/elasticsearch/jdk/lib/libjli.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf291000-7fbfdf292000 rw-p 00010000 08:02 537619864                  /usr/share/elasticsearch/jdk/lib/libjli.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf292000-7fbfdf2a7000 r-xp 00000000 08:02 250085                     /usr/lib64/libz.so.1.2.7
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf2a7000-7fbfdf4a6000 ---p 00015000 08:02 250085                     /usr/lib64/libz.so.1.2.7
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf4a6000-7fbfdf4a7000 r--p 00014000 08:02 250085                     /usr/lib64/libz.so.1.2.7
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf4a7000-7fbfdf4a8000 rw-p 00015000 08:02 250085                     /usr/lib64/libz.so.1.2.7
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf4a8000-7fbfdf4ca000 r-xp 00000000 08:02 39687                      /usr/lib64/ld-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf4d9000-7fbfdf551000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf551000-7fbfdf55d000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf55d000-7fbfdf5b2000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf5b2000-7fbfdf5ba000 rw-s 00000000 00:27 2432071                    /tmp/hsperfdata_elasticsearch/19638
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf5ba000-7fbfdf5be000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf5be000-7fbfdf6c0000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf6c3000-7fbfdf6c4000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf6c5000-7fbfdf6c6000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf6c6000-7fbfdf6c7000 r--p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf6c7000-7fbfdf6c8000 ---p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf6c8000-7fbfdf6c9000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf6c9000-7fbfdf6ca000 r--p 00021000 08:02 39687                      /usr/lib64/ld-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf6ca000-7fbfdf6cb000 rw-p 00022000 08:02 39687                      /usr/lib64/ld-2.17.so
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7fbfdf6cb000-7fbfdf6cc000 rw-p 00000000 00:00 0
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7ffce2b2d000-7ffce2b4e000 rw-p 00000000 00:00 0                          [stack]
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: 7ffce2b83000-7ffce2b85000 r-xp 00000000 00:00 0                          [vdso]
Nov 11 07:58:05 host performance-analyzer-agent-cli[19637]: ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Nov 11 07:58:07 host performance-analyzer-agent-cli[19637]: /usr/share/elasticsearch/bin/performance-analyzer-agent-cli: line 11: 19638 Aborted                 (core dumped) ES_MAIN_CLASS="com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp" ES_ADDITIONAL_CLASSPATH_DIRECTORIES=performance-analyzer-rca/lib ES_JAVA_OPTS=$PA_AGENT_JAVA_OPTS $ES_HOME/bin/elasticsearch-cli "$@"
Nov 11 07:58:07 host systemd[1]: opendistro-performance-analyzer.service: main process exited, code=exited, status=134/n/a
Nov 11 07:58:07 host systemd[1]: Unit opendistro-performance-analyzer.service entered failed state.
Nov 11 07:58:07 host systemd[1]: opendistro-performance-analyzer.service failed.
Nov 11 07:58:07 host systemd[1]: opendistro-performance-analyzer.service holdoff time over, scheduling restart.
Nov 11 07:58:07 host systemd[1]: Stopped Opendistro for Elasticsearch Performance Analyzer.
Nov 11 07:58:07 host systemd[1]: Started Opendistro for Elasticsearch Performance Analyzer.
adityaj1107 commented 3 years ago

Hi @opoplawski

Can you try with the latest ODFE version of Performance Analyzer(1.12) and see if the issue persists?

opoplawski commented 3 years ago

Performance analyzer is currently broken on my machine due to #235