search

allegroai / clearml-server

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Other
386 stars 134 forks source link

SERVER UNAVAILABLE #239

Closed jingli-wtbox closed 7 months ago

jingli-wtbox commented 8 months ago

Hi,

Any help will be appreciated.

After deployment, got the below error.

Screenshot 2024-04-03 at 3 24 33 PM

part of log (docker logs clearml-elastic) are listed below:


> {"type": "server", "timestamp": "2024-04-03T02:08:27,603Z", "level": "ERROR", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "clearml", "node.name": "clearml", "message": "exception during geoip databases update", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" , 
> "stacktrace": ["org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active",
> "at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:137) ~[ingest-geoip-7.17.18.jar:7.17.18]",
> "at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:284) [ingest-geoip-7.17.18.jar:7.17.18]",
> "at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-7.17.18.jar:7.17.18]",
> "at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-7.17.18.jar:7.17.18]",
> "at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-7.17.18.jar:7.17.18]",
> "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.18.jar:7.17.18]",
> "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.18.jar:7.17.18]",
> "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]",
> "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]",
> "at java.lang.Thread.run(Thread.java:1583) [?:?]"] }
jkhenning commented 8 months ago

Hi @jingli-wtbox , can you please attach the complete ES log?

jingli-wtbox commented 7 months ago

@jkhenning , thanks. please find the full ES log below. 

docker logs clearml-elastic
Apr 03, 2024 2:48:47 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
{"type": "server", "timestamp": "2024-04-03T02:48:49,323Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "version[7.17.18], pid[7], build[default/docker/8682172c2130b9a411b1bd5ff37c9792367de6b0/2024-02-02T12:04:59.691750271Z], OS[Linux/5.15.0-1036-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/21.0.2/21.0.2+13-58]" }
{"type": "server", "timestamp": "2024-04-03T02:48:49,332Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
{"type": "server", "timestamp": "2024-04-03T02:48:49,333Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-9093165662722554014, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms2g, -Xmx2g, -Dlog4j2.formatMsgNoLookups=true, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,037Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,038Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,038Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [constant-keyword]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,038Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [frozen-indices]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,038Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [kibana]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,040Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [legacy-geo]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,040Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,040Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [mapper-version]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,040Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,040Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,041Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,041Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,041Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [repositories-metering-api]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,041Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [repository-encrypted]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,042Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,042Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [runtime-fields-common]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,042Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [search-business-rules]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,042Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [searchable-snapshots]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,043Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [snapshot-repo-test-kit]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,043Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [spatial]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,044Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [transform]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,044Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,044Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [unsigned-long]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,045Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [vector-tile]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,045Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [vectors]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,045Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [wildcard]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,046Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-aggregate-metric]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,046Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-analytics]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,046Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-async]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,047Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-async-search]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,047Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-autoscaling]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,047Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,048Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,048Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-data-streams]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,048Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,049Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-enrich]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,049Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-eql]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,049Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-fleet]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,050Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,050Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-identity-provider]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,050Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,050Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,051Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,051Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,051Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ql]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,052Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,052Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,052Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-shutdown]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,052Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,053Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-stack]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,053Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-text-structure]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,054Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-voting-only-node]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,054Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,055Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "no plugins loaded" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,111Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "clearml", "node.name": "clearml", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/root)]], net usable_space [120.4gb], net total_space [1.9tb], types [ext4]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,112Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "clearml", "node.name": "clearml", "message": "heap size [2gb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,239Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "node name [clearml], node ID [A6LcQvX-RYyaLHo9pDNESA], cluster name [clearml], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
{"type": "server", "timestamp": "2024-04-03T02:49:01,644Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "clearml", "node.name": "clearml", "message": "[controller/217] [Main.cc@122] controller (64 bit): Version 7.17.18 (Build a1014b051a9d63) Copyright (c) 2024 Elasticsearch BV" }
{"type": "server", "timestamp": "2024-04-03T02:49:02,341Z", "level": "INFO", "component": "o.e.i.g.ConfigDatabases", "cluster.name": "clearml", "node.name": "clearml", "message": "initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes" }
{"type": "server", "timestamp": "2024-04-03T02:49:02,346Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "initialized database registry, using geoip-databases directory [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA]" }
{"type": "server", "timestamp": "2024-04-03T02:49:03,034Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "clearml", "node.name": "clearml", "message": "creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]" }
{"type": "server", "timestamp": "2024-04-03T02:49:03,074Z", "level": "INFO", "component": "o.e.i.r.RecoverySettings", "cluster.name": "clearml", "node.name": "clearml", "message": "using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]" }
{"type": "server", "timestamp": "2024-04-03T02:49:03,118Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "clearml", "node.name": "clearml", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2024-04-03T02:49:03,621Z", "level": "INFO", "component": "o.e.g.DanglingIndicesState", "cluster.name": "clearml", "node.name": "clearml", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
{"type": "server", "timestamp": "2024-04-03T02:49:04,486Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "initialized" }
{"type": "server", "timestamp": "2024-04-03T02:49:04,487Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "starting ..." }
{"type": "server", "timestamp": "2024-04-03T02:49:04,505Z", "level": "INFO", "component": "o.e.x.s.c.f.PersistentCache", "cluster.name": "clearml", "node.name": "clearml", "message": "persistent cache index loaded" }
{"type": "server", "timestamp": "2024-04-03T02:49:04,506Z", "level": "INFO", "component": "o.e.x.d.l.DeprecationIndexingComponent", "cluster.name": "clearml", "node.name": "clearml", "message": "deprecation component started" }
{"type": "server", "timestamp": "2024-04-03T02:49:04,645Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "clearml", "node.name": "clearml", "message": "publish_address {192.168.16.3:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2024-04-03T02:49:05,021Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "clearml", "node.name": "clearml", "message": "cluster UUID [TTohLnsDRueenl__VV7vBw]" }
{"type": "server", "timestamp": "2024-04-03T02:49:05,173Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "clearml", "node.name": "clearml", "message": "elected-as-master ([1] nodes joined)[{clearml}{A6LcQvX-RYyaLHo9pDNESA}{PmEz4BOmQvCTiWgtpW8GvQ}{192.168.16.3}{192.168.16.3:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 4, version: 75, delta: master node changed {previous [], current [{clearml}{A6LcQvX-RYyaLHo9pDNESA}{PmEz4BOmQvCTiWgtpW8GvQ}{192.168.16.3}{192.168.16.3:9300}{cdfhilmrstw}]}" }
{"type": "server", "timestamp": "2024-04-03T02:49:05,328Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "clearml", "node.name": "clearml", "message": "master node changed {previous [], current [{clearml}{A6LcQvX-RYyaLHo9pDNESA}{PmEz4BOmQvCTiWgtpW8GvQ}{192.168.16.3}{192.168.16.3:9300}{cdfhilmrstw}]}, term: 4, version: 75, reason: Publication{term=4, version=75}" }
{"type": "server", "timestamp": "2024-04-03T02:49:05,545Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "clearml", "node.name": "clearml", "message": "publish_address {192.168.16.3:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA"  }
{"type": "server", "timestamp": "2024-04-03T02:49:05,546Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "started", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA"  }
{"type": "server", "timestamp": "2024-04-03T02:49:06,125Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "clearml", "node.name": "clearml", "message": "license [0f53cb6c-bba4-4c42-bec6-e47b5365624c] mode [basic] - valid", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA"  }
{"type": "server", "timestamp": "2024-04-03T02:49:06,130Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "clearml", "node.name": "clearml", "message": "recovered [6] indices into cluster_state", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA"  }
{"type": "server", "timestamp": "2024-04-03T02:49:06,715Z", "level": "ERROR", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "clearml", "node.name": "clearml", "message": "exception during geoip databases update", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" , 
"stacktrace": ["org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active",
"at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:137) ~[ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:284) [ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.18.jar:7.17.18]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]",
"at java.lang.Thread.run(Thread.java:1583) [?:?]"] }
{"type": "server", "timestamp": "2024-04-03T02:49:06,735Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "clearml", "node.name": "clearml", "message": "Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.geoip_databases][0], [queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2024-04][0], [events-training_debug_image-][0], [events-plot-][0], [events-log-][0], [events-training_stats_scalar-][0]]]).", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA"  }
{"type": "server", "timestamp": "2024-04-03T02:49:06,769Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-Country.mmdb.tmp.gz]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA"  }
{"type": "server", "timestamp": "2024-04-03T02:49:06,788Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-City.mmdb.tmp.gz]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA"  }
{"type": "server", "timestamp": "2024-04-03T02:49:06,788Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-ASN.mmdb.tmp.gz]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA"  }
{"type": "server", "timestamp": "2024-04-03T02:49:07,136Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-Country.mmdb]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA"  }
{"type": "server", "timestamp": "2024-04-03T02:49:07,195Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-ASN.mmdb]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA"  }
{"type": "server", "timestamp": "2024-04-03T02:49:07,785Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-City.mmdb]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA"  }
jingli-wtbox commented 7 months ago

i fix it by following advice from @evg-allegro

put the following setting in the docker-compose.yaml under elasticsearch->environment: ingest.geoip.downloader.enabled: false

also i need to change elastic search folder to /var/lib/elasticsearch/data in docker-compose.yml.

jingli-wtbox commented 7 months ago

hope it will help anyone who is a beginner to clearML.