Error 100 : General data error (TransportError(503, 'search_phase_execution_exception')) again

asromahin commented 3 years ago

Hello,

Now i'm using clearml server (v0.17.0). I have my problem on old version and new version.

I have this problem, only for console logging. Another logging, as a scalars, photos working without problems. It's similar with this issue [https://github.com/allegroai/clearml/issues/316], but answer not worked for me.

I suggested this reasons of problem: 1) low available space on disk 2) old version 3) incorrect index in elasticsearch

And my test of this reasons: 1) I have only 25% usage of disk. For test I'm set variables:

cluster.routing.allocation.disk.watermark.low: 1000gb
cluster.routing.allocation.disk.watermark.high: 1000gb
cluster.routing.allocation.disk.watermark.flood_stage: 1000gb

but it not resolve my problem

I'm trying use cleanup_serice from yours rep, but when script trying delete task, i have this error:

WARNING:root:Could not delete Task ID=8edc28b6fd164e35b1ad59abb43cd8d2, General data error (TransportError(503, 'search_phase_execution_exception'))

2) after updating trains and use clearml, problem not solved 3) not tested, because not understand how test

now i have this docker-compose.yml:

version: "3.6" services:

apiserver: command:

apiserver container_name: clearml-apiserver image: allegroai/clearml:latest restart: unless-stopped volumes:

/data/trains/logs:/var/log/trains

/data/trains/config:/opt/trains/config

/data/trains/data/fileserver:/mnt/fileserver depends_on:

redis

mongo

elasticsearch

fileserver environment: TRAINS_ELASTIC_SERVICE_HOST: elasticsearch TRAINS_ELASTIC_SERVICE_PORT: 9200 TRAINS_MONGODB_SERVICE_HOST: mongo TRAINS_MONGODB_SERVICE_PORT: 27017 TRAINS_REDIS_SERVICE_HOST: redis TRAINS_REDIS_SERVICE_PORT: 6379 TRAINS_SERVER_DEPLOYMENT_TYPE: ${TRAINS_SERVER_DEPLOYMENT_TYPE:-linux} TRAINSapiserverpre_populateenabled: "true" TRAINSapiserverpre_populate__zip_files: "/opt/trains/db-pre-populate" TRAINSapiserver__pre_populate__artifacts_path: "/mnt/fileserver" ports:

"8008:8008" networks:

backend

frontend

elasticsearch: networks:

backend container_name: clearml-elastic environment: ES_JAVA_OPTS: -Xms2g -Xmx2g bootstrap.memory_lock: "true" cluster.name: clearml cluster.routing.allocation.disk.threshold_enabled: "false" cluster.routing.allocation.node_initial_primaries_recoveries: "500" discovery.zen.minimum_master_nodes: "1" discovery.type: "single-node" http.compression_level: "7" node.ingest: "true" node.name: clearml reindex.remote.whitelist: '.' thread_pool.get.queue_size: "2000" thread_pool.search.queue_size: "10000" xpack.monitoring.enabled: "false" xpack.security.enabled: "false" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2 restart: unless-stopped volumes:

/data/trains/data/elastic_7:/usr/share/elasticsearch/data
- /data/trains/data/elastic_7_config:/usr/share/elasticsearch/config

ports:

"9200:9200"

fileserver: networks:

backend

frontend command:

fileserver container_name: clearml-fileserver image: allegroai/clearml:latest restart: unless-stopped volumes:

/data/trains/logs:/var/log/trains

/data/trains/data/fileserver:/mnt/fileserver

/data/trains/config:/opt/trains/config ports:

"8082:8081"

mongo: networks:

backend container_name: clearml-mongo image: mongo:3.6.5 restart: unless-stopped command: --setParameter internalQueryExecMaxBlockingSortBytes=196100200 volumes:

/data/trains/data/mongo/db:/data/db

/data/trains/data/mongo/configdb:/data/configdb ports:

"27017:27017"

redis: networks:

backend container_name: clearml-redis image: redis:5.0 restart: unless-stopped volumes:

/data/trains/data/redis:/data ports:

"6379:6379"

webserver: command:

webserver container_name: clearml-webserver image: allegroai/clearml:latest restart: unless-stopped depends_on:

apiserver ports:

"8080:80" networks:

backend

frontend

agent-services: networks:

backend container_name: clearml-agent-services image: allegroai/clearml-agent-services:latest restart: unless-stopped privileged: true environment: CLEARML_HOST_IP: ${TRAINS_HOST_IP} CLEARML_WEB_HOST: ${TRAINS_WEB_HOST:-} CLEARML_API_HOST: http://apiserver:8008 CLEARML_FILES_HOST: ${TRAINS_FILES_HOST:-} CLEARML_API_ACCESS_KEY: ${TRAINS_API_ACCESS_KEY:-} CLEARML_API_SECRET_KEY: ${TRAINS_API_SECRET_KEY:-} CLEARML_AGENT_GIT_USER: ${TRAINS_AGENT_GIT_USER} CLEARML_AGENT_GIT_PASS: ${TRAINS_AGENT_GIT_PASS} CLEARML_AGENT_UPDATE_VERSION: ${TRAINS_AGENT_UPDATE_VERSION:->=0.17.0} CLEARML_AGENT_DEFAULT_BASE_DOCKER: "ubuntu:18.04" AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-} AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-} AZURE_STORAGE_ACCOUNT: ${AZURE_STORAGE_ACCOUNT:-} AZURE_STORAGE_KEY: ${AZURE_STORAGE_KEY:-} GOOGLE_APPLICATION_CREDENTIALS: ${GOOGLE_APPLICATION_CREDENTIALS:-} CLEARML_WORKER_ID: "clearml-services" CLEARML_AGENT_DOCKER_HOST_MOUNT: "/data/trains/agent:/root/.clearml" volumes:

/var/run/docker.sock:/var/run/docker.sock

/data/trains/agent:/root/.clearml depends_on:

apiserver

networks: backend: driver: bridge frontend: driver: bridge

Any idea why this is happening & How can I fix it??

jkhenning commented 3 years ago

Hi @asromahin,

I have only 25% usage of disk. For test I'm set variables:...

What's the current free-space on your disk? Your setting of 1000gb means ES will stop working when you have 1TB or less of storage free...

Can you share the clearml-elastic container logs? Use sudo docker logs clearml-elastic and attach the output...

asromahin commented 3 years ago

I have 8 Tb disk, and my usage ~2Tb

my result of sudo docker logs clearml-elastic:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. {"type": "server", "timestamp": "2021-03-29T08:50:37,090Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "clearml", "node.name": "clearml", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda)]], net usable_space [6.5tb], net total_space [9tb], types [ext4]" } {"type": "server", "timestamp": "2021-03-29T08:50:37,095Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "clearml", "node.name": "clearml", "message": "heap size [1.9gb], compressed ordinary object pointers [true]" } {"type": "server", "timestamp": "2021-03-29T08:50:37,333Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "node name [clearml], node ID [WZM-KXDgSF-SWPCCqP_wKA], cluster name [clearml]" } {"type": "server", "timestamp": "2021-03-29T08:50:37,334Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "version[7.6.2], pid[1], build[default/docker/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/3.10.0-1127.13.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.2/13.0.2+8]" } {"type": "server", "timestamp": "2021-03-29T08:50:37,334Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "JVM home [/usr/share/elasticsearch/jdk]" } {"type": "server", "timestamp": "2021-03-29T08:50:37,334Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-15584557704734214955, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms2g, -Xmx2g, -XX:MaxDirectMemorySize=1073741824, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,305Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [aggs-matrix-stats]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,306Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [analysis-common]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,306Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [flattened]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,307Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [frozen-indices]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,307Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-common]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,307Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-geoip]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,308Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-user-agent]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,309Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-expression]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,309Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-mustache]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,309Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-painless]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,309Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [mapper-extras]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,309Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [parent-join]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,309Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [percolator]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,309Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [rank-eval]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,310Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [reindex]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,310Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [repository-url]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,310Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [search-business-rules]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,310Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [spatial]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,310Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [transform]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,310Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [transport-netty4]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,311Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [vectors]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,311Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-analytics]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,311Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ccr]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,311Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-core]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,311Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-deprecation]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,312Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-enrich]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,315Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-graph]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,315Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ilm]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,315Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-logstash]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,316Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ml]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,316Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-monitoring]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,316Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-rollup]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,316Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-security]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,317Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-sql]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,318Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-voting-only-node]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,318Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-watcher]" } {"type": "server", "timestamp": "2021-03-29T08:50:39,327Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "no plugins loaded" } {"type": "deprecation", "timestamp": "2021-03-29T08:50:42,153Z", "level": "WARN", "component": "o.e.d.c.s.Settings", "cluster.name": "clearml", "node.name": "clearml", "message": "[discovery.zen.minimum_master_nodes] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." } {"type": "server", "timestamp": "2021-03-29T08:50:43,890Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "clearml", "node.name": "clearml", "message": "[controller/116] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV" } {"type": "server", "timestamp": "2021-03-29T08:50:44,772Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "clearml", "node.name": "clearml", "message": "using discovery type [single-node] and seed hosts providers [settings]" } {"type": "server", "timestamp": "2021-03-29T08:50:45,821Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "initialized" } {"type": "server", "timestamp": "2021-03-29T08:50:45,821Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "starting ..." } {"type": "server", "timestamp": "2021-03-29T08:50:46,014Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "clearml", "node.name": "clearml", "message": "publish_address {172.21.0.2:9300}, bound_addresses {0.0.0.0:9300}" } {"type": "server", "timestamp": "2021-03-29T08:50:46,655Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "clearml", "node.name": "clearml", "message": "cluster UUID [Qjrrdd7BQp-kmGOod4Xrog]" } {"type": "server", "timestamp": "2021-03-29T08:50:46,772Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "clearml", "node.name": "clearml", "message": "elected-as-master ([1] nodes joined)[{clearml}{WZM-KXDgSF-SWPCCqP_wKA}{gHKXNt_sTT-zepmPfkKCvg}{172.21.0.2}{172.21.0.2:9300}{dilm}{ml.machine_memory=134958968832, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTERTASK, _FINISHELECTION], term: 22, version: 338, delta: master node changed {previous [], current [{clearml}{WZM-KXDgSF-SWPCCqP_wKA}{gHKXNt_sTT-zepmPfkKCvg}{172.21.0.2}{172.21.0.2:9300}{dilm}{ml.machine_memory=134958968832, xpack.installed=true, ml.max_open_jobs=20}]}" } {"type": "server", "timestamp": "2021-03-29T08:50:46,894Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "clearml", "node.name": "clearml", "message": "master node changed {previous [], current [{clearml}{WZM-KXDgSF-SWPCCqP_wKA}{gHKXNt_sTT-zepmPfkKCvg}{172.21.0.2}{172.21.0.2:9300}{dilm}{ml.machine_memory=134958968832, xpack.installed=true, ml.max_open_jobs=20}]}, term: 22, version: 338, reason: Publication{term=22, version=338}" } {"type": "server", "timestamp": "2021-03-29T08:50:46,977Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "clearml", "node.name": "clearml", "message": "publish_address {172.21.0.2:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:50:46,977Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "started", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:50:47,906Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "clearml", "node.name": "clearml", "message": "license [f487da0d-d874-45f8-ad6b-1ab0ca049261] mode [basic] - valid", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:50:47,910Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "clearml", "node.name": "clearml", "message": "recovered [24] indices into cluster_state", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:51:11,665Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "clearml", "node.name": "clearml", "message": "adding template [events_log] for index patterns [events-log-]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:51:11,735Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "clearml", "node.name": "clearml", "message": "adding template [events_training_debug_image] for index patterns [events-training_debug_image-]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:51:11,806Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "clearml", "node.name": "clearml", "message": "adding template [events_plot] for index patterns [events-plot-]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:51:11,866Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "clearml", "node.name": "clearml", "message": "adding template [events] for index patterns [events-]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:51:11,948Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "clearml", "node.name": "clearml", "message": "adding template [worker_stats] for index patterns [workerstats]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:51:12,070Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "clearml", "node.name": "clearml", "message": "adding template [queue_metrics] for index patterns [queuemetrics]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:52:05,012Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:52:05,013Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T08:52:05,026Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:52:05,026Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T08:52:05,035Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:52:05,036Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T08:52:05,044Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:52:05,045Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T08:52:05,065Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:52:05,065Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T08:52:05,073Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:52:05,074Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T08:52:05,085Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:52:05,086Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T08:52:05,092Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T08:52:05,093Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:00:41,329Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:00:41,330Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:00:41,335Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:00:41,335Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:00:41,339Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:00:41,339Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:00:41,344Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:00:41,344Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:04:53,429Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:04:53,430Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:04:53,434Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:04:53,435Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:04:53,440Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:04:53,440Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:04:53,444Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:04:53,445Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:37:48,470Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:37:48,470Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:37:48,475Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:37:48,475Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:37:48,479Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:37:48,480Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:37:48,483Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:37:48,484Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:39:02,776Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:39:02,777Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:39:02,781Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:39:02,782Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:39:02,786Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:39:02,786Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:39:02,790Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:39:02,791Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:39:12,211Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:39:12,211Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:39:12,216Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:39:12,216Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:39:12,220Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:39:12,220Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:39:12,225Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:39:12,226Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:44:28,448Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:44:28,448Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:44:28,452Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:44:28,453Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:44:28,456Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:44:28,457Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:44:28,460Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:44:28,461Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:56:51,265Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:56:51,265Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:56:51,271Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:56:51,272Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:56:51,276Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:56:51,277Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:56:51,281Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:56:51,281Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:56:51,371Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:56:51,371Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:56:51,375Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:56:51,376Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:56:51,380Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:56:51,380Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-03-29T09:56:51,385Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "clearml", "node.name": "clearml", "message": "All shards failed for phase: [query]", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" } {"type": "server", "timestamp": "2021-03-29T09:56:51,385Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "clearml", "node.name": "clearml", "message": "path: /events-log-d1bd92a3b039400cbafc60a7a5b1e52b/_search, params: {index=events-log-d1bd92a3b039400cbafc60a7a5b1e52b}", "cluster.uuid": "Qjrrdd7BQp-kmGOod4Xrog", "node.id": "WZM-KXDgSF-SWPCCqP_wKA" , "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:545) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:306) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:574) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:386) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:285) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] }

jkhenning commented 3 years ago

Hi @asromahin,

What was your previous ClearML Server version, and how did you upgrade?

jkhenning commented 3 years ago

@asromahin,

Can you please share the results of the following commands?

curl -XGET --header "Content-Type: application/json" "https://localhost:9200/_cluster/health?pretty&level=shards"

curl -XGET --header "Content-Type: application/json" "http://localhost:9200/_cluster/health/events-log-d1bd92a3b039400cbafc60a7a5b1e52b?pretty&level=shards"

asromahin commented 3 years ago

curl -XGET --header "Content-Type: application/json" "https://localhost:9200/_cluster/health?pretty&level=shards" curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

curl -XGET --header "Content-Type: application/json" "http://localhost:9200/_cluster/health/events-log-d1bd92a3b039400cbafc60a7a5b1e52b?pretty&level=shards" { "cluster_name" : "clearml", "status" : "red", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 2, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 47.91666666666667, "indices" : { "events-log-d1bd92a3b039400cbafc60a7a5b1e52b" : { "status" : "red", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 2, "shards" : { "0" : { "status" : "red", "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 2 } } } } }

asromahin commented 3 years ago

Hi @asromahin,

What was your previous ClearML Server version, and how did you upgrade?

I was upgrade from 0.16.1 to 0.17.0, before was upgraded from 0.14.* to 0.16.1, with instruction of upgrade to version >= 0.15.0 with migration database

jkhenning commented 3 years ago

Sorry, the first one should be: curl -XGET --header "Content-Type: application/json" "http://localhost:9200/_cluster/health?pretty&level=shards"

Can you try again?

asromahin commented 3 years ago

curl -XGET --header "Content-Type: application/json" "http://localhost:9200/_cluster/health?pretty&level=shards"

{ "cluster_name" : "clearml", "status" : "red", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 23, "active_shards" : 23, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 25, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 47.91666666666667, "indices" : { "events-log-d1bd92a3b039400cbafc60a7a5b1e52b" : { "status" : "red", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 2, "shards" : { "0" : { "status" : "red", "primary_active" : false, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 2 } } }, "events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-10" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2021-03" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-12" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-11" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2021-01" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2021-02" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2021-01" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-11" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-10" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "events-training_debug_image-d1bd92a3b039400cbafc60a7a5b1e52b" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-07" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-09" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2020-08" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-12" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2021-02" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-08" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-07" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2021-03" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-06" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2020-09" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } }, "events-plot-d1bd92a3b039400cbafc60a7a5b1e52b" : { "status" : "yellow", "number_of_shards" : 1, "number_of_replicas" : 1, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "shards" : { "0" : { "status" : "yellow", "primary_active" : true, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1 } } } } }

evg-allegro commented 3 years ago

Hi @asromahin, From the cluster health commands that you ran it appears that the es index that stores task console logs is corrupted. While the other indices are fine. I would suggest removing this index (and then it is auto recreated by clearml). It means that the console logs stored in the index till now will be lost. But since it is corrupted I think we cannot get them anyway. All other task data should remain intact. Here is the command that deletes the problematic index: curl -XDELETE "http://localhost:9200/events-log-d1bd92a3b039400cbafc60a7a5b1e52b"

jkhenning commented 3 years ago

@asromahin You can also restore from backup, in case you followed the periodic backup recommendation here.

asromahin commented 3 years ago

@asromahin You can also restore from backup, in case you followed the periodic backup recommendation here.

it's problem for me, because was maked many experiments from update, and i need save them. Problem with incorrect migration?

asromahin commented 3 years ago

Hi @asromahin, From the cluster health commands that you ran it appears that the es index that stores task console logs is corrupted. While the other indices are fine. I would suggest removing this index (and then it is auto recreated by clearml). It means that the console logs stored in the index till now will be lost. But since it is corrupted I think we cannot get them anyway. All other task data should remain intact. Here is the command that deletes the problematic index: curl -XDELETE "http://localhost:9200/events-log-d1bd92a3b039400cbafc60a7a5b1e52b"

Thank you, it's worked for me. After delete this index, problem resolved and new console logs are saved.

evg-allegro commented 3 years ago

@asromahin Great! I do not think that it is related to migration. Unfortunately when running a single ES cluster node (without replication) this kind of corruption is possible. Doing periodic backups of data folders may be a good mitigation for this kind of issues.

faye1225 commented 3 years ago

Hi @asromahin , I have the same problem. In addition to regular backup, regular deletes the problematic index.Have you found a way to do it once and for all?

jkhenning commented 3 years ago

Hi @faye1225,

We suspect this might be related to the cleanup service - are you running the cleanup service as part of the agent-services?

faye1225 commented 3 years ago

嗨@faye1225，

我们怀疑这可能与清理服务有关 - 您是否将清理服务作为代理服务的一部分运行？

The way I handle it now is, when it goes wrong, I delete this index. Complete manual cleaning. But this is not a good long-term solution

faye1225 commented 3 years ago

Hi, @asromahin @jkhenning , The same problem has cropped up again recently.After delete index, problem resolved, but old logs are not saved. Is there another way？😭

jkhenning commented 3 years ago

@faye1225,

As I said previously, we suspect this might be related to the cleanup service - are you running the cleanup service as part of the agent-services?

faye1225 commented 3 years ago

@faye1225,

As I said previously, we suspect this might be related to the cleanup service - are you running the cleanup service as part of the agent-services?

I not running cleanup service and not deploy the agent-services

jkhenning commented 3 years ago

@faye1225 is that's the case, do you have any idea when this happens? I mean, once you delete the indexes and everything goes back to normal, when does it get messed up again?

faye1225 commented 3 years ago

@faye1225 is that's the case, do you have any idea when this happens? I mean, once you delete the indexes and everything goes back to normal, when does it get messed up again?

This seems to be irregular . It happens when you use it too much

jkhenning commented 3 years ago

This seems to be irregular . It happens when you use it too much

Use what too much? Any specific functionality?

faye1225 commented 3 years ago

This seems to be irregular . It happens when you use it too much

Use what too much? Any specific functionality?

No.Just the normal upload code and train normally

jkhenning commented 3 years ago

Do you have any additional information on any specific time it happens? This is not something we've ever seen in our servers (and we're running and testing tens of servers... 🙂)

faye1225 commented 3 years ago

We are deployed on a local server. Did the exact time of the error help you to solve the problem?

jkhenning commented 3 years ago

Hi @faye1225 ,

Can you tell me which exact server version?

faye1225 commented 3 years ago

The system version used is CentOS 7.9,clearml version used is 1.0.0

jkhenning commented 3 years ago

clearml version used is 1.0.0

Is that a locally hosted ClearML Server version or the ClearML Python SDK version?

faye1225 commented 3 years ago

We deployed the clearml server on our server based on k8s and use the ClearML Python SDK version

jkhenning commented 3 years ago

Can you try to go to http://<server-address>:8080/version.json in your browser? Let me know what you see there.

faye1225 commented 3 years ago

{"version":"1.0.2-108", "trains-server-treeish":"release/open-v1.0", "trains-server-commit":"df081a28c90a6eb8666a5f3f025e640810a47e5e", "webapp-treeish":"release/community-v1.0", "webapp-commit":"b1bac4b64537b1d9e8a0155c24d2cb1f2176df2e", "webapp-common-treeish":"release/community-v1.0", "webapp-common-commit":"10afec1d542b3d5111d539bd6c150fd8b749d63b", "webserver-config-treeish":"master", "webserver-config-commit":"53d3b05eb4791d528480f8bd06d7c9e8adfd5ffd", "date":"Mon May 24 08:52:32 UTC 2021"}

jkhenning commented 3 years ago

Hi @faye1225,

Thanks for the info. It does seem you're using an updated server version, but I can't understand the cause for these strange events.

Do you restart the server often or on a regular basis?

faye1225 commented 3 years ago

I not restart the server often or on a regular basis

eliorc commented 2 years ago

We had the same issue, only a server restart solved it

faye1225 commented 2 years ago

We had the same issue, only a server restart solved it

It's been a long time since this problem, luckily I modified the mount disk and the problem was fixed. Maybe you can refer to this method

allegroai / clearml

Error 100 : General data error (TransportError(503, 'search_phase_execution_exception')) again #324

- /data/trains/data/elastic_7_config:/usr/share/elasticsearch/config