search

allegroai / clearml-server

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Other
385 stars 133 forks source link

Error when upgrading to elastic7 #57

Open orpatashnik opened 4 years ago

orpatashnik commented 4 years ago

I'm trying to upgrade trains-server to the latest version, and therefore performing the data migration. There is an error that occurs during the migration of the data from the old elastic to the new one. This is my log:

2020-08-15 01:33:48,199 Waiting for the upgrade containers to start (60sec). 2020-08-15 01:34:48,269 Starting new HTTP connection (1): localhost 2020-08-15 01:34:48,417 Starting new HTTP connection (1): localhost 2020-08-15 01:34:48,545 Starting new HTTP connection (1): localhost 2020-08-15 01:34:48,581 Starting new HTTP connection (1): localhost 2020-08-15 01:34:48,607 Starting new HTTP connection (1): localhost 2020-08-15 01:34:48,640 Starting new HTTP connection (1): localhost 2020-08-15 01:34:48,660 Copying indices from http://localhost:9200 v5 to http://localhost:9211 v7 2020-08-15 01:34:48,676 4 indices will be copied. 2020-08-15 01:34:49,895 Running with a parallel factor of 1 2020-08-15 01:34:49,895 Copying index events-log-d1bd92a3b039400cbafc60a7a5b1e52b 2020-08-15 01:35:00,280 Error when copying index events-log-d1bd92a3b039400cbafc60a7a5b1e52b: 403 Client Error: Forbidden for url: http://localhost:9211/_reindex 2020-08-15 01:35:00,280 events-log-d1bd92a3b039400cbafc60a7a5b1e52b could not be copied. 2020-08-15 01:35:00,280 Copying index events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b 2020-08-15 01:35:30,393 Error when copying index events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b: 403 Client Error: Forbidden for url: http://localhost:9211/_reindex 2020-08-15 01:35:30,394 events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b could not be copied. 2020-08-15 01:35:30,394 Stopping the upgrade containers. 2020-08-15 01:35:30,457 Stopping: elastic-upgrade 2020-08-15 01:35:31,955 Stopping: elastic-upgrade-7 2020-08-15 01:35:33,462 Error running the upgrade: 403 Client Error: Forbidden for url: http://localhost:9211/_reindex

Not sure if it matters, but my trains directory is not in the default location, it is under my home path.

jkhenning commented 4 years ago

Hi @orpatashnik,

We're checking this - can you please attach:

orpatashnik commented 4 years ago

Hi @jkhenning

  1. The command that I used is: sudo python3 elastic_upgrade.py -s /home/orpat/trains/data/elastic/ -t /home/orpat/trains/data/elastic_7 --no-backup . (I backuped in the first time that I ran the script). I ran the script from the trains directory.

  2. upgrade_to_7_2020-08-15_01-33-44.log

  3. elastic_logs.txt

jkhenning commented 4 years ago

Hi @orpatashnik ,

The reason for the failure seems to be not enough free disk space in your system.

Elastic has a couple of watermarks (by default 85% and 90% of the whole disk space), and I would recommend not to reach any of them.

Please make sure you have enough space on the target location so that even after the migration (which essentially makes 2 copies of the data - one for backup and the other in the new ES7 format), more than 15% of the total disk space will still be available.

The related warning in the Elastic log file:

{"type": "server", "timestamp": "2020-08-14T22:34:29,855Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "docker-cluster", "node.name": "49454e03c114", "message": "flood stage disk watermark [95%] exceeded on [eRq0_ibXTkOEVLNfVYmz4g][49454e03c114][/usr/share/elasticsearch/data/nodes/0] free: 366.1gb[4.9%], all indices on this node will be marked read-only", "cluster.uuid": "oNY6NcydSs6jP2WPArHaZg", "node.id": "eRq0_ibXTkOEVLNfVYmz4g"  }
orpatashnik commented 4 years ago

Hi @jkhenning ,

As can be seen in the warning that you attached, there is more than 350 free GB which is quite a lot. The data in my trains directory weighs about 20GB currently, so there is enough space. Is there something that I can do in order to perform the migration in the current situation?

Thanks

jkhenning commented 4 years ago

@orpatashnik, the issue is not the actual amount of free space available, but the watermarks that act as a fail-safe for Elastic.

If you absolutely can't free up more space, you can override the watermarks. I've uploaded an upgraded migration script to allow providing extra environment arguments for the Elastic containers - please download it here (this is not an official version yet)

To use a different watermark (for example to no less than 100GB of free space), use the updated script with this command-line:

sudo python3 elastic_upgrade.py -s /home/orpat/trains/data/elastic/ -t /home/orpat/trains/data/elastic_7 --extra-target-env cluster.routing.allocation.disk.watermark.low=100gb cluster.routing.allocation.disk.watermark.high=100gb cluster.routing.allocation.disk.watermark.flood_stage=100gb

Also, note that if watermark values are indeed the issue, once the migration is completed and before running Trains Server v0.16, you'll need to set the same setting in Trains Server docker-compose file under services/elasticsearch/environment

lamhoangtung commented 3 years ago

Hi @jkhenning. I'm having kind of the same problem too.

  1. The command I ran:

    sudo python3 elastic_upgrade.py --source $PWD/data/elastic --target $PWD/data/elastic_7

  2. The log file created by the migration script 

    2021-02-08 16:55:06,377 Running on: Linux
2021-02-08 16:55:06,441 Docker allocated memory: 16GB
2021-02-08 16:55:06,442 Docker cores: 8
2021-02-08 16:55:06,442 Creating the backup file: /home/ubuntu/linus/migrate_trains/trains/data/elastic_5.tar.gz
2021-02-08 16:55:06,977 Creating the target directory: /home/ubuntu/linus/migrate_trains/trains/data/elastic_7
2021-02-08 16:55:06,977 Stopping containers.
2021-02-08 16:55:07,482 Starting the upgrade containers.
2021-02-08 16:55:08,958 Waiting for the upgrade containers to start (60sec).
2021-02-08 16:56:09,017 Starting new HTTP connection (1): localhost:9234
2021-02-08 16:56:09,164 http://localhost:9234 "GET / HTTP/1.1" 200 242
2021-02-08 16:56:09,167 Starting new HTTP connection (1): localhost:9235
2021-02-08 16:56:09,285 http://localhost:9235 "GET / HTTP/1.1" 200 329
2021-02-08 16:56:09,286 Starting new HTTP connection (1): localhost:9234
2021-02-08 16:56:09,302 http://localhost:9234 "GET / HTTP/1.1" 200 242
2021-02-08 16:56:09,303 Starting new HTTP connection (1): localhost:9235
2021-02-08 16:56:09,312 http://localhost:9235 "GET / HTTP/1.1" 200 329
2021-02-08 16:56:09,320 Starting new HTTP connection (1): localhost:9234
2021-02-08 16:56:09,438 http://localhost:9234 "GET / HTTP/1.1" 200 242
2021-02-08 16:56:09,439 Starting new HTTP connection (1): localhost:9235
2021-02-08 16:56:09,449 http://localhost:9235 "GET / HTTP/1.1" 200 329
2021-02-08 16:56:09,449 Copying indices from http://localhost:9234 v5 to http://localhost:9235 v7
2021-02-08 16:56:09,465 http://localhost:9234 "GET /events-*,queue_*,worker_*/_settings HTTP/1.1" 200 359
2021-02-08 16:56:09,466 4 indices will be copied.
2021-02-08 16:56:09,466 Applying mappings to http://localhost:9235
2021-02-08 16:56:09,524 http://localhost:9235 "PUT /_template/events_log HTTP/1.1" 200 47
2021-02-08 16:56:09,524 Applied mapping events_log: {'acknowledged': True}
2021-02-08 16:56:09,809 http://localhost:9235 "PUT /_template/queue_metrics HTTP/1.1" 200 47
2021-02-08 16:56:09,810 Applied mapping queue_metrics: {'acknowledged': True}
2021-02-08 16:56:09,909 http://localhost:9235 "PUT /_template/events HTTP/1.1" 200 47
2021-02-08 16:56:09,909 Applied mapping events: {'acknowledged': True}
2021-02-08 16:56:10,028 http://localhost:9235 "PUT /_template/worker_stats HTTP/1.1" 200 47
2021-02-08 16:56:10,029 Applied mapping worker_stats: {'acknowledged': True}
2021-02-08 16:56:10,110 http://localhost:9235 "PUT /_template/events_training_debug_image HTTP/1.1" 200 47
2021-02-08 16:56:10,110 Applied mapping events_training_debug_image: {'acknowledged': True}
2021-02-08 16:56:10,204 http://localhost:9235 "PUT /_template/events_plot HTTP/1.1" 200 47
2021-02-08 16:56:10,205 Applied mapping events_plot: {'acknowledged': True}
2021-02-08 16:56:10,205 Running with a parallel factor of 1
2021-02-08 16:56:10,205 Copying index worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06
2021-02-08 16:56:10,209 http://localhost:9235 "HEAD /worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 HTTP/1.1" 404 0
2021-02-08 16:56:10,209 Reindexing worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 with a batch size 1000
2021-02-08 16:56:10,858 http://localhost:9235 "PUT /worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 HTTP/1.1" 200 114
2021-02-08 16:56:10,862 http://localhost:9234 "GET /worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06/_settings HTTP/1.1" 200 211
2021-02-08 16:56:10,866 http://localhost:9235 "GET /worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06/_settings HTTP/1.1" 200 214
2021-02-08 16:56:10,979 http://localhost:9235 "PUT /worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06/_settings HTTP/1.1" 200 47
2021-02-08 16:56:11,020 http://localhost:9235 "POST /_reindex HTTP/1.1" 400 154
2021-02-08 16:56:11,021 Error when copying index worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"[elastic-upgrade:9200] not whitelisted in reindex.remote.whitelist"}],"type":"illegal_argument_exception","reason":"[elastic-upgrade:9200] not whitelisted in reindex.remote.whitelist"},"status":400}
2021-02-08 16:56:11,131 http://localhost:9235 "DELETE /worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 HTTP/1.1" 200 47
2021-02-08 16:56:11,132 Error when copying index worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06: 400 Client Error: Bad Request for url: http://localhost:9235/_reindex
2021-02-08 16:56:11,132 worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 could not be copied.
2021-02-08 16:56:11,132 Copying index queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06
2021-02-08 16:56:11,134 http://localhost:9235 "HEAD /queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 HTTP/1.1" 404 0
2021-02-08 16:56:11,134 Reindexing queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 with a batch size 1000
2021-02-08 16:56:11,510 http://localhost:9235 "PUT /queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 HTTP/1.1" 200 115
2021-02-08 16:56:11,513 http://localhost:9234 "GET /queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06/_settings HTTP/1.1" 200 213
2021-02-08 16:56:11,516 http://localhost:9235 "GET /queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06/_settings HTTP/1.1" 200 216
2021-02-08 16:56:11,622 http://localhost:9235 "PUT /queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06/_settings HTTP/1.1" 200 47
2021-02-08 16:56:11,625 http://localhost:9235 "POST /_reindex HTTP/1.1" 400 154
2021-02-08 16:56:11,629 Error when copying index queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"[elastic-upgrade:9200] not whitelisted in reindex.remote.whitelist"}],"type":"illegal_argument_exception","reason":"[elastic-upgrade:9200] not whitelisted in reindex.remote.whitelist"},"status":400}
2021-02-08 16:56:11,742 http://localhost:9235 "DELETE /queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 HTTP/1.1" 200 47
2021-02-08 16:56:11,743 Error when copying index queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06: 400 Client Error: Bad Request for url: http://localhost:9235/_reindex
2021-02-08 16:56:11,743 queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06 could not be copied.
2021-02-08 16:56:11,743 Stopping the upgrade containers.
2021-02-08 16:56:11,991 Stopping: elastic-upgrade
2021-02-08 16:56:14,052 Stopping: elastic-upgrade-7
2021-02-08 16:56:15,330 Error running the upgrade: 400 Client Error: Bad Request for url: http://localhost:9235/_reindex
2021-02-08 16:56:15,330 Removing the directory: /home/ubuntu/linus/migrate_trains/trains/data/elastic_7
2021-02-08 16:56:15,331 Resolve the error(s) and run the script again. See the upgrade log for troubleshooting. For assistance, see the Trains documentation.

  3. The ES7 container log:

    OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2021-02-08T16:55:13,987Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/nvme0n1p1)]], net usable_space [154.1gb], net total_space [290.7gb], types [ext4]" }
{"type": "server", "timestamp": "2021-02-08T16:55:13,991Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "heap size [1.9gb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2021-02-08T16:55:14,136Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "node name [c04c06687a9f], node ID [BEWlhaA3QWi5f74u14qO_w], cluster name [docker-cluster]" }
{"type": "server", "timestamp": "2021-02-08T16:55:14,137Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "version[7.6.2], pid[1], build[default/docker/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/5.4.0-1028-aws/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.2/13.0.2+8]" }
{"type": "server", "timestamp": "2021-02-08T16:55:14,137Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "JVM home [/usr/share/elasticsearch/jdk]" }
{"type": "server", "timestamp": "2021-02-08T16:55:14,137Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-12302202030657505764, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms2g, -Xmx2g, -XX:MaxDirectMemorySize=1073741824, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,771Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,771Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,771Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [flattened]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,772Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [frozen-indices]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,772Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,772Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,772Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,772Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,773Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,773Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,773Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,773Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,773Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,773Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,774Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,774Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,774Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [search-business-rules]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,774Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [spatial]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,775Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [transform]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,775Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,775Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [vectors]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,775Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-analytics]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,775Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,776Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,776Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,776Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-enrich]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,776Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,776Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,776Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,776Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,777Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,777Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,777Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,777Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,778Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-voting-only-node]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,778Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2021-02-08T16:55:17,778Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "no plugins loaded" }
{"type": "server", "timestamp": "2021-02-08T16:55:22,633Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "[controller/107] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV" }
{"type": "server", "timestamp": "2021-02-08T16:55:23,840Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2021-02-08T16:55:24,647Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "initialized" }
{"type": "server", "timestamp": "2021-02-08T16:55:24,647Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "starting ..." }
{"type": "server", "timestamp": "2021-02-08T16:55:24,772Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "publish_address {192.168.208.3:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2021-02-08T16:55:25,008Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "setting initial configuration to VotingConfiguration{BEWlhaA3QWi5f74u14qO_w}" }
{"type": "server", "timestamp": "2021-02-08T16:55:25,131Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "elected-as-master ([1] nodes joined)[{c04c06687a9f}{BEWlhaA3QWi5f74u14qO_w}{o_ojRxk7Rz21CVOLKZ6Opg}{192.168.208.3}{192.168.208.3:9300}{dilm}{ml.machine_memory=16187129856, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{c04c06687a9f}{BEWlhaA3QWi5f74u14qO_w}{o_ojRxk7Rz21CVOLKZ6Opg}{192.168.208.3}{192.168.208.3:9300}{dilm}{ml.machine_memory=16187129856, xpack.installed=true, ml.max_open_jobs=20}]}" }
{"type": "server", "timestamp": "2021-02-08T16:55:25,163Z", "level": "INFO", "component": "o.e.c.c.CoordinationState", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "cluster UUID set to [9tpGkHwKRtu6F9I-pHeCLg]" }
{"type": "server", "timestamp": "2021-02-08T16:55:25,187Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "master node changed {previous [], current [{c04c06687a9f}{BEWlhaA3QWi5f74u14qO_w}{o_ojRxk7Rz21CVOLKZ6Opg}{192.168.208.3}{192.168.208.3:9300}{dilm}{ml.machine_memory=16187129856, xpack.installed=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}" }
{"type": "server", "timestamp": "2021-02-08T16:55:25,252Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "publish_address {192.168.208.3:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,252Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "started", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,297Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "recovered [0] indices into cluster_state", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,448Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [.watches] for index patterns [.watches*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,504Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [.watch-history-10] for index patterns [.watcher-history-10*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,540Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [.triggered_watches] for index patterns [.triggered_watches*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,576Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [ilm-history] for index patterns [ilm-history-1*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,610Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [.slm-history] for index patterns [.slm-history-1*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,646Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,694Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,739Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,782Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,822Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,859Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding index lifecycle policy [watch-history-ilm-policy]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,895Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding index lifecycle policy [ilm-history-ilm-policy]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:25,932Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding index lifecycle policy [slm-history-ilm-policy]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:55:26,372Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "license [d84b8411-ddce-4904-8e6e-9cb9af021af3] mode [basic] - valid", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:09,484Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [events_log] for index patterns [events-log-*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:09,534Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [queue_metrics] for index patterns [queue_metrics_*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:09,820Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [events] for index patterns [events-*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:09,927Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [worker_stats] for index patterns [worker_stats_*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:10,040Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [events_training_debug_image] for index patterns [events-training_debug_image-*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:10,120Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "adding template [events_plot] for index patterns [events-plot-*]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:10,221Z", "level": "INFO", "component": "o.e.c.m.MetaDataCreateIndexService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "[worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06] creating index, cause [api], templates [worker_stats], shards [1]/[1], mappings [_doc]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:10,871Z", "level": "INFO", "component": "o.e.c.m.MetaDataUpdateSettingsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "updating number_of_replicas to [0] for indices [worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:10,882Z", "level": "INFO", "component": "o.e.c.s.IndexScopedSettings", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": " [worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06] updating [index.refresh_interval] from [1s] to [-1]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:10,973Z", "level": "INFO", "component": "o.e.c.s.IndexScopedSettings", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": " [worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06] updating [index.refresh_interval] from [1s] to [-1]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "deprecation", "timestamp": "2021-02-08T16:56:11,016Z", "level": "WARN", "component": "o.e.d.i.r.ReindexRequest", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "[types removal] Specifying types in reindex requests is deprecated.", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:11,024Z", "level": "INFO", "component": "o.e.c.m.MetaDataDeleteIndexService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "[worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06/65aaXuPeTymaUdWHeF_kFw] deleting index", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:11,143Z", "level": "INFO", "component": "o.e.c.m.MetaDataCreateIndexService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "[queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06] creating index, cause [api], templates [queue_metrics], shards [1]/[1], mappings [_doc]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:11,519Z", "level": "INFO", "component": "o.e.c.m.MetaDataUpdateSettingsService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "updating number_of_replicas to [0] for indices [queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:11,525Z", "level": "INFO", "component": "o.e.c.s.IndexScopedSettings", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": " [queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06] updating [index.refresh_interval] from [1s] to [-1]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:11,616Z", "level": "INFO", "component": "o.e.c.s.IndexScopedSettings", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": " [queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06] updating [index.refresh_interval] from [1s] to [-1]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:11,630Z", "level": "INFO", "component": "o.e.c.m.MetaDataDeleteIndexService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "[queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06/MruBwTWDRn2neXIDR82Yvw] deleting index", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:14,168Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "stopping ...", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:14,186Z", "level": "INFO", "component": "o.e.x.w.WatcherService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "stopping watch service, reason [shutdown initiated]", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:14,187Z", "level": "INFO", "component": "o.e.x.w.WatcherLifeCycleService", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "watcher has stopped and shutdown", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:14,483Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "[controller/107] [Main.cc@150] Ml controller exiting", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:14,484Z", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "Native controller process has stopped - no new native processes can be started", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:14,493Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "stopped", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:14,493Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "closing ...", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }
{"type": "server", "timestamp": "2021-02-08T16:56:14,501Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c04c06687a9f", "message": "closed", "cluster.uuid": "9tpGkHwKRtu6F9I-pHeCLg", "node.id": "BEWlhaA3QWi5f74u14qO_w"  }

And this seems like not related to the DiskThresholdMonitor problem as you describe about since my machine had over 50% of free disk space:

/dev/nvme0n1p1                             291G  137G  155G  47% /

Can you take a look and help me out here. Thanks <3

jkhenning commented 3 years ago

Hi @lamhoangtung,

It looks like you modified the migration script to use the ports 9234 and 9235 on the host system for mapping to ES upgrade containers instead of the default ones (9200 and 9211). The migration script is designed to work with the specific mapping of ports 9200 and 9211. There's not need to change them since there should not be any conflict with the original version containers being mapped to the same ports since the script is stopping all the containers during the upgrade. If you revert back to the original script implementation, it should work as expected.

lamhoangtung commented 3 years ago

Thanks @jkhenning for the help. After changing back to the default port it had worked as expected

annagrr commented 3 years ago

Hi @jkhenning I am having a similar issue. Would you mind having a look to help me see what's going on?

  1. Command I used python elastic_upgrade.py -s /home/USERNAME/trains-server/server/elastic/ -t /home/USERNAME/trains-server/server/elastic_7/
  2. upgrade_to_7_2021-05-27_10-29-05.log
  3. elastic_logs.txt

Any advice would be greatly appreciated!

jkhenning commented 3 years ago

Hi @annagrr,

What version are you upgrading from, and to which version?

evg-allegro commented 3 years ago

Hi @annagrr It seems that there is an issue connecting to one of the ES containers that the upgrade script runs. The ES log that you attached here from the ES7 container seems fine. Can you please collect the log from the source ES5 container and attach it here? The command would be: docker logs elastic-upgrade >& elastic5_logs.txt