search

open-telemetry / opentelemetry-demo

This repository contains the OpenTelemetry Astronomy Shop, a microservice-based distributed system intended to illustrate the implementation of OpenTelemetry in a near real-world environment.
https://opentelemetry.io/docs/demo/
Apache License 2.0
1.88k stars 1.24k forks source link

opensearch won't start in local docker env at first time #1761

Closed sophiaxu0424 closed 1 week ago

sophiaxu0424 commented 1 month ago

Bug Report

Which version of the demo you are using? (please provide either a specific

main branch: https://github.com/open-telemetry/opentelemetry-demo/commits/main

and hash: bafadb0d4261ac519f19d6d69fe393c971586525

Symptom

A clear and concise description of what the bug is.

What is the expected behavior? opensearch container is up and running

What do you expect to see? opensearch container is up and running successfully

What is the actual behavior?

[2024-10-30T21:25:35,091][ERROR][o.o.s.l.BuiltinLogTypeLoader] [demo-node] Failed loading builtin log types from disk!
java.nio.file.FileSystemNotFoundException: null
    at jdk.zipfs@21.0.4/jdk.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:156) ~[?:?]
    at jdk.zipfs@21.0.4/jdk.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:142) ~[?:?]
    at java.base/java.nio.file.Path.of(Path.java:209) ~[?:?]
    at java.base/java.nio.file.Paths.get(Paths.java:98) ~[?:?]
    at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.loadBuiltinLogTypes(BuiltinLogTypeLoader.java:73) ~[opensearch-security-analytics-2.17.1.0.jar:2.17.1.0]
    at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.ensureLogTypesLoaded(BuiltinLogTypeLoader.java:62) [opensearch-security-analytics-2.17.1.0.jar:2.17.1.0]
    at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.doStart(BuiltinLogTypeLoader.java:146) [opensearch-security-analytics-2.17.1.0.jar:2.17.1.0]
    at org.opensearch.common.lifecycle.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:77) [opensearch-common-2.17.1.jar:2.17.1]
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) [?:?]

Please describe the actual behavior experienced.

Reproduce

Could you provide the minimum required steps to resolve the issue you're seeing?

make start

or

docker-compose up --force-recreate --remove-orphans --detach

We will close this issue if:

Additional Context

Please feel free to add any other context about the problem here.

Re-start the the containers again and again, would get a healthy status in opensearch.

julianocosta89 commented 1 month ago

Hello @sophiaxu0424, which OS are you running the demo? Does the error also happen when you run make start? I saw that you also suggested docker-compose, but that won't work. We use docker compose, the version with the - is the old compose version and not supported.

sophiaxu0424 commented 1 month ago

Hi @julianocosta89 i am using a MAC OS with M1, I saw that the docker-compose has depreciated, but it might because of my local setting in /usr/local/bin, it is still using docker-compose command with the latest version.

julianocosta89 commented 4 weeks ago

@sophiaxu0424 if you have the latest version of Docker installed, it already supports docker compose. Could you try the following and let us know if it helps?

docker compose down
docker compose up --force-recreate --remove-orphans --detach
sophiaxu0424 commented 3 weeks ago

Still having this error

sophiaxu@Sophias-MBP-2 opentelemetry-demo % docker compose down
docker compose up --force-recreate --remove-orphans --detach
[+] Running 26/26
 ✔ Container accounting-service       Removed                                                                                                                      0.2s
 ✔ Container prometheus               Removed                                                                                                                      2.6s
 ✔ Container frauddetection-service   Removed                                                                                                                      0.2s
 ✔ Container frontend-proxy           Removed                                                                                                                      0.2s
 ✔ Container grafana                  Removed                                                                                                                      2.9s
 ✔ Container flagd-ui                 Removed                                                                                                                      0.1s
 ✔ Container load-generator           Removed                                                                                                                      0.1s
 ✔ Container frontend                 Removed                                                                                                                      0.1s
 ✔ Container imageprovider            Removed                                                                                                                      0.1s
 ✔ Container checkout-service         Removed                                                                                                                      0.1s
 ✔ Container ad-service               Removed                                                                                                                      0.2s
 ✔ Container recommendation-service   Removed                                                                                                                      0.2s
 ✔ Container quote-service            Removed                                                                                                                      0.2s
 ✔ Container payment-service          Removed                                                                                                                      0.2s
 ✔ Container kafka                    Removed                                                                                                                      3.6s
 ✔ Container currency-service         Removed                                                                                                                      0.2s
 ✔ Container cart-service             Removed                                                                                                                      0.2s
 ✔ Container email-service            Removed                                                                                                                      0.2s
 ✔ Container shipping-service         Removed                                                                                                                      0.2s
 ✔ Container product-catalog-service  Removed                                                                                                                      0.3s
 ✔ Container valkey-cart              Removed                                                                                                                      1.9s
 ✔ Container otel-col                 Removed                                                                                                                      0.2s
 ✔ Container flagd                    Removed                                                                                                                      1.8s
 ✔ Container jaeger                   Removed                                                                                                                      2.0s
 ✔ Container opensearch               Removed                                                                                                                      1.9s
 ✔ Network opentelemetry-demo         Removed                                                                                                                      0.2s
[+] Running 26/26
 ✔ Network opentelemetry-demo         Created                                                                                                                      0.1s
 ✘ Container opensearch               Error                                                                                                                      278.8s
 ✔ Container prometheus               Started                                                                                                                      2.0s
 ✔ Container grafana                  Started                                                                                                                      2.0s
 ✔ Container jaeger                   Started                                                                                                                      2.2s
 ✔ Container kafka                    Started                                                                                                                      2.2s
 ✔ Container valkey-cart              Started                                                                                                                      2.0s
 ✔ Container flagd                    Started                                                                                                                      2.2s
 ✔ Container otel-col                 Created                                                                                                                      0.1s
 ✔ Container accounting-service       Created                                                                                                                      0.2s
 ✔ Container imageprovider            Created                                                                                                                      0.2s
 ✔ Container payment-service          Created                                                                                                                      0.2s
 ✔ Container quote-service            Created                                                                                                                      0.2s
 ✔ Container cart-service             Created                                                                                                                      0.2s
 ✔ Container frauddetection-service   Created                                                                                                                      0.3s
 ✔ Container flagd-ui                 Created                                                                                                                      0.2s
 ✔ Container currency-service         Created                                                                                                                      0.2s
 ✔ Container ad-service               Created                                                                                                                      0.2s
 ✔ Container product-catalog-service  Created                                                                                                                      0.2s
 ✔ Container shipping-service         Created                                                                                                                      0.2s
 ✔ Container email-service            Created                                                                                                                      0.2s
 ✔ Container recommendation-service   Created                                                                                                                      0.1s
 ✔ Container checkout-service         Created                                                                                                                      0.1s
 ✔ Container frontend                 Created                                                                                                                      0.1s
 ✔ Container load-generator           Created                                                                                                                      0.1s
 ✔ Container frontend-proxy           Created                                                                                                                      0.1s
dependency failed to start: container opensearch is unhealthy
sophiaxu0424 commented 3 weeks ago

and docker logs about this opensearch container

sophiaxu@Sophias-MBP-2 opentelemetry-demo % docker logs 549022bed7c3
Disabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.17.1.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Nov 08, 2024 4:02:55 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.17.1.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-11-08T16:02:58,363][INFO ][o.o.n.Node               ] [demo-node] version[2.17.1], pid[1], build[tar/1893d20797e30110e5877170e44d42275ce5951e/2024-09-26T21:59:52.691008096Z], OS[Linux/6.5.0-44-generic/aarch64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.4/21.0.4+7-LTS]
[2024-11-08T16:02:58,382][INFO ][o.o.n.Node               ] [demo-node] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-11-08T16:02:58,383][INFO ][o.o.n.Node               ] [demo-node] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-12179628951096343081, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms300m, -Xmx300m, -XX:MaxDirectMemorySize=157286400, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2024-11-08T16:03:00,047][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [demo-node] Java vector incubator API enabled; uses preferredBitSize=128; FMA enabled
[2024-11-08T16:03:06,826][INFO ][o.o.s.s.t.SSLConfig      ] [demo-node] SSL dual mode is disabled
[2024-11-08T16:03:06,837][WARN ][o.o.s.OpenSearchSecurityPlugin] [demo-node] OpenSearch Security plugin installed but disabled. This can expose your configuration (including passwords) to the public.
[2024-11-08T16:03:09,527][INFO ][o.o.p.c.c.PluginSettings ] [demo-node] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2024-11-08T16:03:12,547][INFO ][o.o.i.r.ReindexPlugin    ] [demo-node] ReindexPlugin reloadSPI called
[2024-11-08T16:03:12,550][INFO ][o.o.i.r.ReindexPlugin    ] [demo-node] Unable to find any implementation for RemoteReindexExtension
[2024-11-08T16:03:12,843][INFO ][o.o.j.JobSchedulerPlugin ] [demo-node] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2024-11-08T16:03:12,845][INFO ][o.o.j.JobSchedulerPlugin ] [demo-node] Loaded scheduler extension: opensearch_time_series_analytics, index: .opendistro-anomaly-detector-jobs
[2024-11-08T16:03:12,861][INFO ][o.o.j.JobSchedulerPlugin ] [demo-node] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2024-11-08T16:03:12,877][INFO ][o.o.j.JobSchedulerPlugin ] [demo-node] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2024-11-08T16:03:12,879][INFO ][o.o.j.JobSchedulerPlugin ] [demo-node] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job
[2024-11-08T16:03:12,890][INFO ][o.o.j.JobSchedulerPlugin ] [demo-node] Loaded scheduler extension: async-query-scheduler, index: .async-query-scheduler
[2024-11-08T16:03:13,005][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [aggs-matrix-stats]
[2024-11-08T16:03:13,005][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [analysis-common]
[2024-11-08T16:03:13,006][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [cache-common]
[2024-11-08T16:03:13,006][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [geo]
[2024-11-08T16:03:13,006][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [ingest-common]
[2024-11-08T16:03:13,006][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [ingest-geoip]
[2024-11-08T16:03:13,006][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [ingest-user-agent]
[2024-11-08T16:03:13,006][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [lang-expression]
[2024-11-08T16:03:13,006][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [lang-mustache]
[2024-11-08T16:03:13,006][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [lang-painless]
[2024-11-08T16:03:13,006][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [mapper-extras]
[2024-11-08T16:03:13,006][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [opensearch-dashboards]
[2024-11-08T16:03:13,006][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [parent-join]
[2024-11-08T16:03:13,007][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [percolator]
[2024-11-08T16:03:13,007][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [rank-eval]
[2024-11-08T16:03:13,008][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [reindex]
[2024-11-08T16:03:13,014][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [repository-url]
[2024-11-08T16:03:13,019][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [search-pipeline-common]
[2024-11-08T16:03:13,019][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [systemd]
[2024-11-08T16:03:13,019][INFO ][o.o.p.PluginsService     ] [demo-node] loaded module [transport-netty4]
[2024-11-08T16:03:13,020][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-alerting]
[2024-11-08T16:03:13,026][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-anomaly-detection]
[2024-11-08T16:03:13,028][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-asynchronous-search]
[2024-11-08T16:03:13,028][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-cross-cluster-replication]
[2024-11-08T16:03:13,028][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-custom-codecs]
[2024-11-08T16:03:13,028][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-flow-framework]
[2024-11-08T16:03:13,028][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-geospatial]
[2024-11-08T16:03:13,028][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-index-management]
[2024-11-08T16:03:13,029][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-job-scheduler]
[2024-11-08T16:03:13,042][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-knn]
[2024-11-08T16:03:13,042][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-ml]
[2024-11-08T16:03:13,042][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-neural-search]
[2024-11-08T16:03:13,043][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-notifications]
[2024-11-08T16:03:13,043][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-notifications-core]
[2024-11-08T16:03:13,043][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-observability]
[2024-11-08T16:03:13,043][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-performance-analyzer]
[2024-11-08T16:03:13,064][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-reports-scheduler]
[2024-11-08T16:03:13,064][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-security]
[2024-11-08T16:03:13,064][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-security-analytics]
[2024-11-08T16:03:13,064][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-skills]
[2024-11-08T16:03:13,064][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-sql]
[2024-11-08T16:03:13,066][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [opensearch-system-templates]
[2024-11-08T16:03:13,066][INFO ][o.o.p.PluginsService     ] [demo-node] loaded plugin [query-insights]
[2024-11-08T16:03:13,496][WARN ][stderr                   ] [demo-node] WARNING: A restricted method in java.lang.foreign.Linker has been called
[2024-11-08T16:03:13,496][WARN ][stderr                   ] [demo-node] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
[2024-11-08T16:03:13,496][WARN ][stderr                   ] [demo-node] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
[2024-11-08T16:03:13,662][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [demo-node] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
[2024-11-08T16:03:13,679][INFO ][o.o.e.NodeEnvironment    ] [demo-node] using [1] data paths, mounts [[/ (overlay)]], net usable_space [38.4gb], net total_space [57gb], types [overlay]
[2024-11-08T16:03:13,679][INFO ][o.o.e.NodeEnvironment    ] [demo-node] heap size [300mb], compressed ordinary object pointers [true]
[2024-11-08T16:03:13,842][INFO ][o.o.n.Node               ] [demo-node] node name [demo-node], node ID [UsVAuE0KSX-0jUCgTlPjYw], cluster name [demo-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-11-08T16:03:14,107][INFO ][o.o.e.ExtensionsManager  ] [demo-node] ExtensionsManager initialized
[2024-11-08T16:03:27,754][DEPRECATION][o.o.d.c.s.Settings       ] [demo-node] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2024-11-08T16:03:46,865][INFO ][o.o.n.p.NeuralSearch     ] [demo-node] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
[2024-11-08T16:04:29,675][INFO ][o.o.t.b.CircuitBreakerService] [demo-node] Registered memory breaker.
[2024-11-08T16:09:36,285][INFO ][o.o.m.b.MLCircuitBreakerService] [demo-node] Registered ML memory breaker.
[2024-11-08T16:09:36,435][INFO ][o.o.m.b.MLCircuitBreakerService] [demo-node] Registered ML disk breaker.
[2024-11-08T16:09:36,435][INFO ][o.o.m.b.MLCircuitBreakerService] [demo-node] Registered ML native memory breaker.
[2024-11-08T16:11:27,707][INFO ][o.r.Reflections          ] [demo-node] Reflections took 7533 ms to scan 1 urls, producing 26 keys and 67 values
[2024-11-08T16:11:32,640][INFO ][o.r.Reflections          ] [demo-node] Reflections took 264 ms to scan 1 urls, producing 3 keys and 5 values
[2024-11-08T16:12:12,530][WARN ][o.o.s.p.SQLPlugin        ] [demo-node] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
Disabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.17.1.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Nov 08, 2024 4:55:29 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
[2024-11-08T16:56:36,633][INFO ][o.o.n.Node               ] [demo-node] version[2.17.1], pid[1], build[tar/1893d20797e30110e5877170e44d42275ce5951e/2024-09-26T21:59:52.691008096Z], OS[Linux/6.5.0-44-generic/aarch64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.4/21.0.4+7-LTS]
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.17.1.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-11-08T16:56:36,811][INFO ][o.o.n.Node               ] [demo-node] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-11-08T16:56:36,811][INFO ][o.o.n.Node               ] [demo-node] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-17850748487669341561, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms300m, -Xmx300m, -XX:MaxDirectMemorySize=157286400, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2024-11-08T16:57:10,789][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [demo-node] Java vector incubator API enabled; uses preferredBitSize=128; FMA enabled
Disabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
............
............
julianocosta89 commented 3 weeks ago

@sophiaxu0424 which OS are you running this? I can't reproduce the error on my end.

Also, we have just release v 1.12.0, could you try pulling the latest changes and run the commands again?

carlyrichmond commented 2 weeks ago

@sophiaxu0424 I had a similar issue today and resolved it by cleaning up dangling volumes using the steps in the below threads:

https://discuss.elastic.co/t/no-space-left-on-device/248400

https://stackoverflow.com/questions/30604846/docker-error-no-space-left-on-device#:~:text=Check%20that%20you%20have%20free,var%2Flib%2Fdocker).&text=If%20you%20edit%20%2Fetc%2Fdefault,the%20change%20to%20take%20effect.

Let me know if that addresses your issue, too.

sophiaxu0424 commented 1 week ago

tried command docker system prune and it helped reclaimed back some space for me. but restarting the demo images/containers still having this error.

Screenshot 2024-11-20 at 10 15 03 AM
sophiaxu0424 commented 1 week ago

Not sure if it was a conflict about my local docker install, I have two local installation and one with brew, another one with dmg install, we can't use docker desktop, and I'm using colima instead.

After i removed the rm -rf ~/.docker/and re installed/upgraded again using brew and re-started colima, it is working for now.

sophiaxu0424 commented 1 week ago

Thanks for the all the investigation till now, a lot of cmd like docker useage cmds are pretty useful for me. Close this for now