elastic / support-diagnostics

Support diagnostics utility for elasticsearch and logstash
Other
290 stars 150 forks source link

Support export/import of monitoring data from/to 8.x #635

Closed sakurai-youhei closed 11 months ago

sakurai-youhei commented 1 year ago

resolves #585

Differences between 7.x and 8.x: - type field is omitted in 8.x - cluster_name moves to elasticsearch.cluster.name in 8.x - cluster_uuid aliases to elasticsearch.cluster.id in 8.x - create action must be used for bulk to data streams in 8.x

Differences in monitoring by Metricbeat 8:

sakurai-youhei commented 1 year ago

List

C:\Users\YouheiSakurai\git\support-diagnostics>diagnostics-8.4.4-SNAPSHOT\export-monitoring.bat --host xxx.found.io --port 443 -u elastic -p --ssl --list
No Java Home was found. Using current path. If execution fails please install Java and make sure it is in the search path or exposed via the JAVA_HOME environment variable.
2023-05-15 23:43:47,522 main ERROR Unable to locate appender "diag" for logger config "root"
Processing diagnosticInputs...
Elasticsearch user password: ************************
Deleted directory: C:\Users\YouheiSakurai\git\support-diagnostics\monitoring-export.
Diagnostic logger reconfigured for inclusion into archive
Displaying a list of available clusters.
Monitored Clusters:
name: xxx   id: ALDn90a2TNeng2DarCn5hw   display name: XXX (xxx)
Closing loggers.
Archiving diagnostic results.
Archive: C:\Users\YouheiSakurai\git\support-diagnostics\monitoring-export-20230515-144355.zip was created
Delete of directory:C:\Users\YouheiSakurai\git\support-diagnostics\monitoring-export failed. Usually this indicates a permission issue
org.apache.commons.io.IOExceptionList: C:\Users\YouheiSakurai\git\support-diagnostics\monitoring-export
...

Export

C:\Users\YouheiSakurai\git\support-diagnostics>diagnostics-8.4.4-SNAPSHOT\export-monitoring.bat --host xxx.found.io --port 443 -u elastic -p --ssl --id ALDn90a2TNeng2DarCn5hw --interval 1
No Java Home was found. Using current path. If execution fails please install Java and make sure it is in the search path or exposed via the JAVA_HOME environment variable.
2023-05-15 23:34:17,011 main ERROR Unable to locate appender "diag" for logger config "root"
Processing diagnosticInputs...
Elasticsearch user password: ************************
Deleted directory: C:\Users\YouheiSakurai\git\support-diagnostics\monitoring-export.
Diagnostic logger reconfigured for inclusion into archive
Now extracting cluster_state...
32610 documents retrieved. Writing to disk.
1000 of 32610 processed.
2000 of 32610 processed.
...
32000 of 32610 processed.
32610 of 32610 processed.
Now extracting cluster_stats...
360 documents retrieved. Writing to disk.
360 of 360 processed.
Now extracting node_stats...
36210 documents retrieved. Writing to disk.
1000 of 36210 processed.
2000 of 36210 processed.
...
36000 of 36210 processed.
36210 of 36210 processed.
Now extracting indices_stats...
360 documents retrieved. Writing to disk.
360 of 360 processed.
Now extracting index_stats...
37740 documents retrieved. Writing to disk.
1000 of 37740 processed.
2000 of 37740 processed.
...
37000 of 37740 processed.
37740 of 37740 processed.
Now extracting shards...
1135290 documents retrieved. Writing to disk.
1000 of 1135290 processed.
2000 of 1135290 processed.
...
1135000 of 1135290 processed.
1135290 of 1135290 processed.
Now extracting job_stats...
1080 documents retrieved. Writing to disk.
1000 of 1080 processed.
1080 of 1080 processed.
Now extracting ccr_stats...
No documents found for: ccr_stats.
Now extracting ccr_auto_follow_stats...
No documents found for: ccr_auto_follow_stats.
Now extracting logstash_stats...
358 documents retrieved. Writing to disk.
358 of 358 processed.
Now extracting logstash_state...
1432 documents retrieved. Writing to disk.
1000 of 1432 processed.
1432 of 1432 processed.
Closing loggers.
Archiving diagnostic results.
Archive: C:\Users\YouheiSakurai\git\support-diagnostics\monitoring-export-20230515-143819.zip was created
Delete of directory:C:\Users\YouheiSakurai\git\support-diagnostics\monitoring-export failed. Usually this indicates a permission issue
org.apache.commons.io.IOExceptionList: C:\Users\YouheiSakurai\git\support-diagnostics\monitoring-export
...

Import

C:\Users\YouheiSakurai\git\support-diagnostics>diagnostics-8.4.4-SNAPSHOT\import-monitoring.bat --host xxx.found.io --port 443 --ssl --user elastic -p --input monitoring-export-20230515-115134.zip
No Java Home was found. Using current path. If execution fails please install Java and make sure it is in the search path or exposed via the JAVA_HOME environment variable.
2023-05-15 23:10:04,390 main ERROR Unable to locate appender "diag" for logger config "root"
Processing diagnosticInputs...
Elasticsearch user password: ************************
Deleted directory: C:\Users\YouheiSakurai\git\support-diagnostics\monitoring-export.
Creating temporary directory C:\Users\YouheiSakurai\git\support-diagnostics\monitoring-export
Configuring log file.
Diagnostic logger reconfigured for inclusion into archive
Extract completed successfully!
Processing: cluster_state.json
Targeting .monitoring-es-8-diag-import-2023-05-15 because no type field is found
Indexing document batch 0 to 500
Indexing document batch 500 to 1000
Indexing document batch 1000 to 1500
...
Indexing document batch 32000 to 32500
Indexing document batch 32500 to 32545
32545 events written from cluster_state.json
Processing: cluster_stats.json
Targeting .monitoring-es-8-diag-import-2023-05-15 because no type field is found
Indexing document batch 0 to 360
360 events written from cluster_stats.json
Processing: index_stats.json
Targeting .monitoring-es-8-diag-import-2023-05-15 because no type field is found
Indexing document batch 0 to 500
Indexing document batch 500 to 1000
...
Indexing document batch 37000 to 37500
Indexing document batch 37500 to 37665
37665 events written from index_stats.json
Processing: indices_stats.json
Targeting .monitoring-es-8-diag-import-2023-05-15 because no type field is found
Indexing document batch 0 to 360
360 events written from indices_stats.json
Processing: job_stats.json
Targeting .monitoring-es-8-diag-import-2023-05-15 because no type field is found
Indexing document batch 0 to 500
Indexing document batch 500 to 1000
Indexing document batch 1000 to 1078
1078 events written from job_stats.json
Processing: logstash_state.json
Targeting .monitoring-logstash-7-diag-import-2023-05-15 because type field is found
Indexing document batch 0 to 500
Indexing document batch 500 to 1000
Indexing document batch 1000 to 1433
1433 events written from logstash_state.json
Processing: logstash_stats.json
Targeting .monitoring-logstash-7-diag-import-2023-05-15 because type field is found
Indexing document batch 0 to 359
359 events written from logstash_stats.json
Processing: node_stats.json
Targeting .monitoring-es-8-diag-import-2023-05-15 because no type field is found
Indexing document batch 0 to 500
Indexing document batch 500 to 1000
...
Indexing document batch 35500 to 36000
Indexing document batch 36000 to 36138
36138 events written from node_stats.json
Processing: shards.json
Targeting .monitoring-es-8-diag-import-2023-05-15 because no type field is found
Indexing document batch 0 to 500
Indexing document batch 500 to 1000
...
Indexing document batch 1132500 to 1133000
Indexing document batch 1133000 to 1133024
1133024 events written from shards.json
Closing loggers.
sakurai-youhei commented 1 year ago

I find type field is omitted in 8.x is suspicious. Will do more research.

sakurai-youhei commented 1 year ago

The characteristics of https://github.com/elastic/support-diagnostics/pull/635#issue-1710235410 are originated not in 7.x or 8.x but in legacy collectors or Metricbeat Metricbeat 8 or not.

pattern collected by type cluster_uuid cluster_name elsaticsearch.cluster.name
.monitoring-*-7-yyyy.MM.dd Legacy collectors (both 7 & 8) keyword keyword _source n/a
.monitoring-*-7-mb-yyyy.MM.dd Metricbeat 7 keyword keyword _source n/a
.monitoring-*-8-mb Metricbeat 8 n/a alias to [*.]elasticsearch.cluster.id n/a keyword
sakurai-youhei commented 1 year ago

@elastic/innovation-eng Could you review this PR? Thanks!