[BUG] Fatal glibc error: CPU does not support x86-64-v2 #5120

Closed 4d4ch4u32 closed 2 days ago

Describe the bug

Docker image v2.17.x on GitLab CI is not working, it fails with the error Fatal glibc error: CPU does not support x86-64-v2.

Related component

Build

To Reproduce

Running OpenSearch as a service in a GitLab CI Job:

job_name:
  services:
    - name: opensearchproject/opensearch:2.17.0
      alias: opensearch
      command: ["bash", "-c", "env 'OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m' 'discovery.type=single-node' 'cluster.name=opensearch' ./opensearch-docker-entrypoint.sh"]

Expected behavior

OpenSearch is running in GitLab CI without errors.

Additional Details

GitLab Log:

Running with gitlab-runner 17.4.0 (b92ee590)
  on cur-devtools-svr05 Jjtz6bvP, system ID: s_0c0e0a0e3[2](https://git.xxx.de/xxx/xxx-software/-/jobs/12494/viewer#L2)3c
Preparing the "docker" executor
Using Docker executor with image git.xxx.com:5050/dev-tools/glubuntu-php:latest-php-8.[3](https://git.xxx.de/xxx/xxx-software/-/jobs/12494/viewer#L3) ...
Starting service mysql:8.0.28...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:f2ad9f23df82a3e5efabd157[4](https://git.xxx.de/xxx/xxx-software/-/jobs/12494/viewer#L4)b862a94c0657c73a6179efec07d5cf9ae5a307f for mysql:8.0.28 with digest mysql@sha256:fc77d54cacef90ad3d7[5](https://git.xxx.de/xxx/xxx-software/-/jobs/12494/viewer#L5)964837fad0f2a9a368b69e7d799665a3f4e90e[6](https://git.xxx.de/xxx/xxx-software/-/jobs/12494/viewer#L6)00c2d ...
Starting service opensearchproject/opensearch:2.17.0...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:884999f8490ec8441c36[7](https://git.xxx.de/xxx/xxx-software/-/jobs/12494/viewer#L7)53687f3a5902c4b686c2b31d8eecf90c35e4e1f0c85 for opensearchproject/opensearch:2.17.0 with digest opensearchproject/opensearch@sha256:7b9517dff561f54456fad947f11820fa595a340fd1034fc63d57b77f724b3f8f ...
Waiting for services to be up and running (timeout 30 seconds)...
*** WARNING: Service runner-jjtz6bvp-project-2-concurrent-0-b[8](https://git.xxx.de/xxx/xxx-software/-/jobs/12494/viewer#L8)29bc9726b952fa-opensearchproject__opensearch-1 probably didn't start properly.
Health check error:
start service container: Error response from daemon: Cannot link to a non running container: /runner-jjtz6bvp-project-2-concurrent-0-b82[9](https://git.xxx.de/xxx/xxx-software/-/jobs/12494/viewer#L9)bc9726b952fa-opensearchproject__opensearch-1 AS /runner-jjtz6bvp-project-2-concurrent-0-b829bc9726b952fa-opensearchproject__opensearch-1-wait-for-service/service (services.go:215:0s)
Service container logs:
2024-[10](https://git.xxx.de/xxx/xxx-software/-/jobs/12494/viewer#L10)-15T14:54:11.507195573Z Fatal glibc error: CPU does not support x86-64-v2
*********

This error is not present in version 2.6.0 of the docker image.

Any update?

Moving this to the correct repo. This needs debugging @4d4ch4u32, please help.

@peterzhuamazon , can you please take a look and add your thoughts?

@4d4ch4u32 , can you add details about underlying OS

@4d4ch4u32 , can you add details about underlying OS

This container is used in the GitLab CI: https://git.var-lab.com/dev-tools/glubuntu-php (it's a Ubuntu with pre-installed packages, see the Dockerfile in the repository for details)

But I'm not sure if the GitLab services are also running inside the "Job-Image" which is defined for the GitLab CI-job. It possible that the services are injected with another magic made by GitLab devs...

Moving this to the correct repo. This needs debugging @4d4ch4u32, please help.

Which information do you need?

Redundant of https://github.com/opensearch-project/opensearch-build/issues/4580.

This is more related to the AL2023 more than the build repo.

I think @4d4ch4u32 should check the issues on https://github.com/amazonlinux/container-images or https://github.com/amazonlinux/amazon-linux-2023/.

Thanks.

I did a quick test using docker pull opensearchproject/opensearch:latest and a t2.small instance running in us-west-2 and it works okay:

docker run output

``` docker run -e OPENSEARCH_INITIAL_ADMIN_PASSWORD='someInfoHere' -e OPENSEARCH_JAVA_OPTS='-Xms512m -Xmx512m' -e discovery.type=single-node -it --name openSearch d606c82a3c23 Enabling OpenSearch Security Plugin Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user. Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string. If a password is not provided, the setup will quit. For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/ ### OpenSearch Security Demo Installer ### ** Warning: Do not use on production or public reachable systems ** OpenSearch install type: rpm/deb on Linux 6.1.112-122.189.amzn2023.x86_64 amd64 OpenSearch config dir: /usr/share/opensearch/config/ OpenSearch config file: /usr/share/opensearch/config/opensearch.yml OpenSearch bin dir: /usr/share/opensearch/bin/ OpenSearch plugins dir: /usr/share/opensearch/plugins/ OpenSearch lib dir: /usr/share/opensearch/lib/ Detected OpenSearch Version: 2.17.1 Detected OpenSearch Security Version: 2.17.1.0 Admin password set successfully. ### Success ### Execute this script now on all your nodes and then start all nodes ### OpenSearch Security will be automatically initialized. ### If you like to change the runtime configuration ### change the files in ../../../config/opensearch-security and execute: sudo "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv ### or run ./securityadmin_demo.sh ### To use the Security Plugin ConfigurationGUI ### To access your secured cluster open https://: and log in with admin/. ### (Ignore the SSL certificate warning because we installed self-signed demo certificates) Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin WARNING: Using incubator modules: jdk.incubator.vector WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.17.1.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch WARNING: System::setSecurityManager will be removed in a future release Oct 24, 2024 10:50:22 PM sun.util.locale.provider.LocaleProviderAdapter WARNING: COMPAT locale provider will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.17.1.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security WARNING: System::setSecurityManager will be removed in a future release [2024-10-24T22:50:23,141][INFO ][o.o.n.Node ] [423a42af1e19] version[2.17.1], pid[1], build[tar/1893d20797e30110e5877170e44d42275ce5951e/2024-09-26T21:59:32.078798875Z], OS[Linux/6.1.112-122.189.amzn2023.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.4/21.0.4+7-LTS] [2024-10-24T22:50:23,148][INFO ][o.o.n.Node ] [423a42af1e19] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true] [2024-10-24T22:50:23,148][INFO ][o.o.n.Node ] [423a42af1e19] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-280146287058861371, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true] [2024-10-24T22:50:23,549][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [423a42af1e19] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled [2024-10-24T22:50:25,570][INFO ][o.o.s.s.t.SSLConfig ] [423a42af1e19] SSL dual mode is disabled [2024-10-24T22:50:25,571][INFO ][o.o.s.OpenSearchSecurityPlugin] [423a42af1e19] OpenSearch Config path is /usr/share/opensearch/config [2024-10-24T22:50:26,079][INFO ][o.o.s.s.DefaultSecurityKeyStore] [423a42af1e19] JVM supports TLSv1.3 [2024-10-24T22:50:26,081][INFO ][o.o.s.s.DefaultSecurityKeyStore] [423a42af1e19] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively [2024-10-24T22:50:27,764][INFO ][o.o.s.s.DefaultSecurityKeyStore] [423a42af1e19] TLS Transport Client Provider : JDK [2024-10-24T22:50:27,766][INFO ][o.o.s.s.DefaultSecurityKeyStore] [423a42af1e19] TLS Transport Server Provider : JDK [2024-10-24T22:50:27,766][INFO ][o.o.s.s.DefaultSecurityKeyStore] [423a42af1e19] TLS HTTP Provider : JDK [2024-10-24T22:50:27,766][INFO ][o.o.s.s.DefaultSecurityKeyStore] [423a42af1e19] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2] [2024-10-24T22:50:27,767][INFO ][o.o.s.s.DefaultSecurityKeyStore] [423a42af1e19] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2] [2024-10-24T22:50:27,852][INFO ][o.o.s.OpenSearchSecurityPlugin] [423a42af1e19] Clustername: docker-cluster [2024-10-24T22:50:27,862][WARN ][o.o.s.OpenSearchSecurityPlugin] [423a42af1e19] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700) [2024-10-24T22:50:27,863][WARN ][o.o.s.OpenSearchSecurityPlugin] [423a42af1e19] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600) [2024-10-24T22:50:27,864][WARN ][o.o.s.OpenSearchSecurityPlugin] [423a42af1e19] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600) [2024-10-24T22:50:27,864][WARN ][o.o.s.OpenSearchSecurityPlugin] [423a42af1e19] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600) [2024-10-24T22:50:27,865][WARN ][o.o.s.OpenSearchSecurityPlugin] [423a42af1e19] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600) [2024-10-24T22:50:27,865][WARN ][o.o.s.OpenSearchSecurityPlugin] [423a42af1e19] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600) [2024-10-24T22:50:27,866][WARN ][o.o.s.OpenSearchSecurityPlugin] [423a42af1e19] File /usr/share/opensearch/config/securityadmin_demo.sh has insecure file permissions (should be 0600) [2024-10-24T22:50:29,319][INFO ][o.o.p.c.c.PluginSettings ] [423a42af1e19] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600 [2024-10-24T22:50:30,572][INFO ][o.o.i.r.ReindexPlugin ] [423a42af1e19] ReindexPlugin reloadSPI called [2024-10-24T22:50:30,573][INFO ][o.o.i.r.ReindexPlugin ] [423a42af1e19] Unable to find any implementation for RemoteReindexExtension [2024-10-24T22:50:30,642][INFO ][o.o.j.JobSchedulerPlugin ] [423a42af1e19] Loaded scheduler extension: opensearch_time_series_analytics, index: .opendistro-anomaly-detector-jobs [2024-10-24T22:50:30,696][INFO ][o.o.j.JobSchedulerPlugin ] [423a42af1e19] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions [2024-10-24T22:50:30,699][INFO ][o.o.j.JobSchedulerPlugin ] [423a42af1e19] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config [2024-10-24T22:50:30,700][INFO ][o.o.j.JobSchedulerPlugin ] [423a42af1e19] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource [2024-10-24T22:50:30,701][INFO ][o.o.j.JobSchedulerPlugin ] [423a42af1e19] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job [2024-10-24T22:50:30,703][INFO ][o.o.j.JobSchedulerPlugin ] [423a42af1e19] Loaded scheduler extension: async-query-scheduler, index: .async-query-scheduler [2024-10-24T22:50:30,745][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [aggs-matrix-stats] [2024-10-24T22:50:30,756][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [analysis-common] [2024-10-24T22:50:30,757][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [cache-common] [2024-10-24T22:50:30,757][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [geo] [2024-10-24T22:50:30,758][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [ingest-common] [2024-10-24T22:50:30,758][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [ingest-geoip] [2024-10-24T22:50:30,758][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [ingest-user-agent] [2024-10-24T22:50:30,758][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [lang-expression] [2024-10-24T22:50:30,758][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [lang-mustache] [2024-10-24T22:50:30,759][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [lang-painless] [2024-10-24T22:50:30,759][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [mapper-extras] [2024-10-24T22:50:30,759][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [opensearch-dashboards] [2024-10-24T22:50:30,759][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [parent-join] [2024-10-24T22:50:30,760][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [percolator] [2024-10-24T22:50:30,760][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [rank-eval] [2024-10-24T22:50:30,760][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [reindex] [2024-10-24T22:50:30,760][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [repository-url] [2024-10-24T22:50:30,761][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [search-pipeline-common] [2024-10-24T22:50:30,761][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [systemd] [2024-10-24T22:50:30,762][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded module [transport-netty4] [2024-10-24T22:50:30,762][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-alerting] [2024-10-24T22:50:30,763][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-anomaly-detection] [2024-10-24T22:50:30,763][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-asynchronous-search] [2024-10-24T22:50:30,763][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-cross-cluster-replication] [2024-10-24T22:50:30,763][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-custom-codecs] [2024-10-24T22:50:30,764][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-flow-framework] [2024-10-24T22:50:30,764][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-geospatial] [2024-10-24T22:50:30,764][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-index-management] [2024-10-24T22:50:30,764][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-job-scheduler] [2024-10-24T22:50:30,764][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-knn] [2024-10-24T22:50:30,765][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-ml] [2024-10-24T22:50:30,765][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-neural-search] [2024-10-24T22:50:30,766][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-notifications] [2024-10-24T22:50:30,766][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-notifications-core] [2024-10-24T22:50:30,766][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-observability] [2024-10-24T22:50:30,767][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-performance-analyzer] [2024-10-24T22:50:30,767][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-reports-scheduler] [2024-10-24T22:50:30,767][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-security] [2024-10-24T22:50:30,767][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-security-analytics] [2024-10-24T22:50:30,767][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-skills] [2024-10-24T22:50:30,767][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-sql] [2024-10-24T22:50:30,768][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [opensearch-system-templates] [2024-10-24T22:50:30,768][INFO ][o.o.p.PluginsService ] [423a42af1e19] loaded plugin [query-insights] [2024-10-24T22:50:30,900][INFO ][o.o.s.OpenSearchSecurityPlugin] [423a42af1e19] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml [2024-10-24T22:50:30,927][DEPRECATION][o.o.d.c.s.Settings ] [423a42af1e19] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version. [2024-10-24T22:50:30,957][WARN ][stderr ] [423a42af1e19] WARNING: A restricted method in java.lang.foreign.Linker has been called [2024-10-24T22:50:30,958][WARN ][stderr ] [423a42af1e19] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module [2024-10-24T22:50:30,958][WARN ][stderr ] [423a42af1e19] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module [2024-10-24T22:50:31,122][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [423a42af1e19] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false [2024-10-24T22:50:31,141][INFO ][o.o.e.NodeEnvironment ] [423a42af1e19] using [1] data paths, mounts [[/ (overlay)]], net usable_space [16.7gb], net total_space [19.9gb], types [overlay] [2024-10-24T22:50:31,141][INFO ][o.o.e.NodeEnvironment ] [423a42af1e19] heap size [512mb], compressed ordinary object pointers [true] [2024-10-24T22:50:31,194][INFO ][o.o.n.Node ] [423a42af1e19] node name [423a42af1e19], node ID [0jhQZeQiQfyCRp17kd1euQ], cluster name [docker-cluster], roles [ingest, remote_cluster_client, data, cluster_manager] [2024-10-24T22:50:31,298][INFO ][o.o.e.ExtensionsManager ] [423a42af1e19] ExtensionsManager initialized [2024-10-24T22:50:35,460][DEPRECATION][o.o.d.c.s.Settings ] [423a42af1e19] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version. [2024-10-24T22:50:35,469][DEPRECATION][o.o.d.c.s.Settings ] [423a42af1e19] [plugins.security.ssl.transport.enforce_hostname_verification] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version. [2024-10-24T22:50:36,910][INFO ][o.o.n.p.NeuralSearch ] [423a42af1e19] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled] [2024-10-24T22:50:37,755][WARN ][o.o.s.c.Salt ] [423a42af1e19] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes [2024-10-24T22:50:37,838][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Message routing enabled: true [2024-10-24T22:50:37,893][INFO ][o.o.s.f.SecurityFilter ] [423a42af1e19] indices are made immutable. [2024-10-24T22:50:38,554][INFO ][o.o.t.b.CircuitBreakerService] [423a42af1e19] Registered memory breaker. [2024-10-24T22:50:39,472][INFO ][o.o.m.b.MLCircuitBreakerService] [423a42af1e19] Registered ML memory breaker. [2024-10-24T22:50:39,474][INFO ][o.o.m.b.MLCircuitBreakerService] [423a42af1e19] Registered ML disk breaker. [2024-10-24T22:50:39,474][INFO ][o.o.m.b.MLCircuitBreakerService] [423a42af1e19] Registered ML native memory breaker. [2024-10-24T22:50:39,663][INFO ][o.r.Reflections ] [423a42af1e19] Reflections took 127 ms to scan 1 urls, producing 26 keys and 67 values [2024-10-24T22:50:39,715][INFO ][o.r.Reflections ] [423a42af1e19] Reflections took 7 ms to scan 1 urls, producing 3 keys and 5 values [2024-10-24T22:50:40,059][WARN ][o.o.s.p.SQLPlugin ] [423a42af1e19] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information [2024-10-24T22:50:42,015][INFO ][o.o.t.NettyAllocator ] [423a42af1e19] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}] [2024-10-24T22:50:42,022][INFO ][o.o.s.s.t.SSLConfig ] [423a42af1e19] SSL dual mode is disabled [2024-10-24T22:50:42,295][INFO ][o.o.d.DiscoveryModule ] [423a42af1e19] using discovery type [single-node] and seed hosts providers [settings] [2024-10-24T22:50:43,586][WARN ][o.o.g.DanglingIndicesState] [423a42af1e19] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2024-10-24T22:50:45,244][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [423a42af1e19] PerformanceAnalyzer Enabled: false [2024-10-24T22:50:45,355][INFO ][o.o.n.Node ] [423a42af1e19] initialized [2024-10-24T22:50:45,356][INFO ][o.o.n.Node ] [423a42af1e19] starting ... [2024-10-24T22:50:45,376][ERROR][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Failed loading builtin log types from disk! java.nio.file.FileSystemNotFoundException: null at jdk.zipfs@21.0.4/jdk.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:156) ~[?:?] at jdk.zipfs@21.0.4/jdk.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:142) ~[?:?] at java.base/java.nio.file.Path.of(Path.java:209) ~[?:?] at java.base/java.nio.file.Paths.get(Paths.java:98) ~[?:?] at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.loadBuiltinLogTypes(BuiltinLogTypeLoader.java:73) ~[opensearch-security-analytics-2.17.1.0.jar:2.17.1.0] at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.ensureLogTypesLoaded(BuiltinLogTypeLoader.java:62) [opensearch-security-analytics-2.17.1.0.jar:2.17.1.0] at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.doStart(BuiltinLogTypeLoader.java:146) [opensearch-security-analytics-2.17.1.0.jar:2.17.1.0] at org.opensearch.common.lifecycle.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:77) [opensearch-common-2.17.1.jar:2.17.1] at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) [?:?] at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1116) [?:?] at org.opensearch.node.Node.start(Node.java:1564) [opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) [opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) [opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.1.jar:2.17.1] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.1.jar:2.17.1] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.17.1.jar:2.17.1] at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.1.jar:2.17.1] [2024-10-24T22:50:45,668][INFO ][o.o.t.TransportService ] [423a42af1e19] publish_address {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300} [2024-10-24T22:50:45,671][INFO ][o.o.t.TransportService ] [423a42af1e19] Remote clusters initialized successfully. [2024-10-24T22:50:46,023][INFO ][o.o.c.c.Coordinator ] [423a42af1e19] setting initial configuration to VotingConfiguration{0jhQZeQiQfyCRp17kd1euQ} [2024-10-24T22:50:46,288][INFO ][o.o.c.s.MasterService ] [423a42af1e19] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper and count: 3, term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{423a42af1e19}{0jhQZeQiQfyCRp17kd1euQ}{JIofuu8xRS-VhEHjfPvO6w}{172.17.0.2}{172.17.0.2:9300}{dimr}{shard_indexing_pressure_enabled=true}]} [2024-10-24T22:50:46,360][INFO ][o.o.c.c.CoordinationState] [423a42af1e19] cluster UUID set to [-C_tYSZlQQ25TSExQkTv2Q] [2024-10-24T22:50:46,407][INFO ][o.o.c.s.ClusterApplierService] [423a42af1e19] cluster-manager node changed {previous [], current [{423a42af1e19}{0jhQZeQiQfyCRp17kd1euQ}{JIofuu8xRS-VhEHjfPvO6w}{172.17.0.2}{172.17.0.2:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1} [2024-10-24T22:50:46,420][INFO ][o.o.t.i.IndexManagement ] [423a42af1e19] Candidate custom result indices are empty. [2024-10-24T22:50:46,424][INFO ][o.o.t.i.IndexManagement ] [423a42af1e19] Candidate custom result indices are empty. [2024-10-24T22:50:46,425][INFO ][o.o.t.c.ClusterEventListener] [423a42af1e19] Cluster is not recovered yet. [2024-10-24T22:50:46,497][INFO ][o.o.i.i.ManagedIndexCoordinator] [423a42af1e19] Cache cluster manager node onClusterManager time: 1729810246497 [2024-10-24T22:50:46,510][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [423a42af1e19] Config override setting update called with empty string. Ignoring. [2024-10-24T22:50:46,525][DEPRECATION][o.o.d.c.u.c.ThreadContext] [423a42af1e19] Default access to markAsSystemContext will be removed in a future release. Permission to use markAsSystemContext must be explicitly granted. [2024-10-24T22:50:46,542][INFO ][o.o.d.PeerFinder ] [423a42af1e19] setting findPeersInterval to [1s] as node commission status = [true] for local node [{423a42af1e19}{0jhQZeQiQfyCRp17kd1euQ}{JIofuu8xRS-VhEHjfPvO6w}{172.17.0.2}{172.17.0.2:9300}{dimr}{shard_indexing_pressure_enabled=true}] [2024-10-24T22:50:46,545][WARN ][o.o.c.r.a.AllocationService] [423a42af1e19] Falling back to single shard assignment since batch mode disable or multiple custom allocators set [2024-10-24T22:50:46,550][INFO ][o.o.h.AbstractHttpServerTransport] [423a42af1e19] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200} [2024-10-24T22:50:46,551][INFO ][o.o.n.Node ] [423a42af1e19] started [2024-10-24T22:50:46,552][INFO ][o.o.s.c.ConfigurationRepository] [423a42af1e19] Will attempt to create index .opendistro_security and default configs if they are absent [2024-10-24T22:50:46,561][INFO ][o.o.s.OpenSearchSecurityPlugin] [423a42af1e19] 0 OpenSearch Security modules loaded so far: [] [2024-10-24T22:50:46,563][INFO ][o.o.s.c.ConfigurationRepository] [423a42af1e19] Background init thread started. Install default config?: true [2024-10-24T22:50:46,565][INFO ][o.o.s.c.ConfigurationRepository] [423a42af1e19] Wait for cluster to be available ... [2024-10-24T22:50:46,570][WARN ][o.o.c.r.a.AllocationService] [423a42af1e19] Falling back to single shard assignment since batch mode disable or multiple custom allocators set [2024-10-24T22:50:46,591][INFO ][o.o.t.c.HashRing ] [423a42af1e19] Node added: [0jhQZeQiQfyCRp17kd1euQ] [2024-10-24T22:50:46,597][INFO ][o.o.t.c.HashRing ] [423a42af1e19] Add data node to version hash ring: 0jhQZeQiQfyCRp17kd1euQ [2024-10-24T22:50:46,603][INFO ][o.o.t.c.HashRing ] [423a42af1e19] All nodes with known version: {0jhQZeQiQfyCRp17kd1euQ=ADNodeInfo{version=2.17.1, isEligibleDataNode=true}} [2024-10-24T22:50:46,606][INFO ][o.o.t.c.HashRing ] [423a42af1e19] Rebuild hash ring for realtime with cooldown, nodeChangeEvents size 0 [2024-10-24T22:50:46,607][INFO ][o.o.t.c.HashRing ] [423a42af1e19] Build version hash ring successfully [2024-10-24T22:50:46,609][INFO ][o.o.t.c.ADDataMigrator ] [423a42af1e19] Start migrating AD data [2024-10-24T22:50:46,609][INFO ][o.o.t.c.ADDataMigrator ] [423a42af1e19] AD job index doesn't exist, no need to migrate [2024-10-24T22:50:46,610][INFO ][o.o.t.c.ClusterEventListener] [423a42af1e19] Init version hash ring successfully [2024-10-24T22:50:46,645][INFO ][o.o.g.GatewayService ] [423a42af1e19] recovered [0] indices into cluster_state [2024-10-24T22:50:46,714][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opensearch-observability/IGMAjuO9QQWQ9HzIZqTmdQ] [2024-10-24T22:50:47,022][INFO ][o.o.c.m.MetadataCreateIndexService] [423a42af1e19] [.opensearch-observability] creating index, cause [api], templates [], shards [1]/[0] [2024-10-24T22:50:47,033][WARN ][o.o.c.r.a.AllocationService] [423a42af1e19] Falling back to single shard assignment since batch mode disable or multiple custom allocators set [2024-10-24T22:50:47,110][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opensearch-observability/IGMAjuO9QQWQ9HzIZqTmdQ] [2024-10-24T22:50:47,217][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/md1QlaxBSQ6SuB1jDVF_sg] [2024-10-24T22:50:47,256][INFO ][o.o.c.m.MetadataCreateIndexService] [423a42af1e19] [.opensearch-sap-log-types-config] creating index, cause [auto(sap-logtype api)], templates [], shards [1]/[1] [2024-10-24T22:50:47,266][INFO ][o.o.c.r.a.AllocationService] [423a42af1e19] updating number_of_replicas to [0] for indices [.opensearch-sap-log-types-config] [2024-10-24T22:50:47,270][WARN ][o.o.c.r.a.AllocationService] [423a42af1e19] Falling back to single shard assignment since batch mode disable or multiple custom allocators set [2024-10-24T22:50:47,321][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/md1QlaxBSQ6SuB1jDVF_sg] [2024-10-24T22:50:47,519][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Loading builtin types! [2024-10-24T22:50:47,536][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [windows_logtype.json] log type [2024-10-24T22:50:47,539][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [waf_logtype.json] log type [2024-10-24T22:50:47,548][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [vpcflow_logtype.json] log type [2024-10-24T22:50:47,549][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Loading builtin types! [2024-10-24T22:50:47,550][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [test_windows_logtype.json] log type [2024-10-24T22:50:47,551][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [s3_logtype.json] log type [2024-10-24T22:50:47,552][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_web_logtype.json] log type [2024-10-24T22:50:47,554][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [windows_logtype.json] log type [2024-10-24T22:50:47,554][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_proxy_logtype.json] log type [2024-10-24T22:50:47,569][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_macos_logtype.json] log type [2024-10-24T22:50:47,570][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_compliance_logtype.json] log type [2024-10-24T22:50:47,570][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_cloud_logtype.json] log type [2024-10-24T22:50:47,572][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_apt_logtype.json] log type [2024-10-24T22:50:47,571][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [waf_logtype.json] log type [2024-10-24T22:50:47,573][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [vpcflow_logtype.json] log type [2024-10-24T22:50:47,574][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [test_windows_logtype.json] log type [2024-10-24T22:50:47,576][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [s3_logtype.json] log type [2024-10-24T22:50:47,574][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_application_logtype.json] log type [2024-10-24T22:50:47,577][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [okta_logtype.json] log type [2024-10-24T22:50:47,577][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [network_logtype.json] log type [2024-10-24T22:50:47,577][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_web_logtype.json] log type [2024-10-24T22:50:47,580][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_proxy_logtype.json] log type [2024-10-24T22:50:47,581][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_macos_logtype.json] log type [2024-10-24T22:50:47,582][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_compliance_logtype.json] log type [2024-10-24T22:50:47,582][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_cloud_logtype.json] log type [2024-10-24T22:50:47,583][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_apt_logtype.json] log type [2024-10-24T22:50:47,584][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [others_application_logtype.json] log type [2024-10-24T22:50:47,580][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [netflow_logtype.json] log type [2024-10-24T22:50:47,584][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [m365_logtype.json] log type [2024-10-24T22:50:47,585][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [linux_logtype.json] log type [2024-10-24T22:50:47,585][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [okta_logtype.json] log type [2024-10-24T22:50:47,586][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [network_logtype.json] log type [2024-10-24T22:50:47,586][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [netflow_logtype.json] log type [2024-10-24T22:50:47,587][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [m365_logtype.json] log type [2024-10-24T22:50:47,586][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [gworkspace_logtype.json] log type [2024-10-24T22:50:47,588][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [github_logtype.json] log type [2024-10-24T22:50:47,589][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [linux_logtype.json] log type [2024-10-24T22:50:47,590][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [gworkspace_logtype.json] log type [2024-10-24T22:50:47,590][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [dns_logtype.json] log type [2024-10-24T22:50:47,592][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [cloudtrail_logtype.json] log type [2024-10-24T22:50:47,591][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [github_logtype.json] log type [2024-10-24T22:50:47,593][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [azure_logtype.json] log type [2024-10-24T22:50:47,595][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [dns_logtype.json] log type [2024-10-24T22:50:47,595][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [apache_access_logtype.json] log type [2024-10-24T22:50:47,596][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [ad_ldap_logtype.json] log type [2024-10-24T22:50:47,596][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [cloudtrail_logtype.json] log type [2024-10-24T22:50:47,599][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [azure_logtype.json] log type [2024-10-24T22:50:47,600][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [apache_access_logtype.json] log type [2024-10-24T22:50:47,600][INFO ][o.o.s.l.BuiltinLogTypeLoader] [423a42af1e19] Loaded [ad_ldap_logtype.json] log type [2024-10-24T22:50:47,602][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Indexing [429] fieldMappingDocs from logTypes: 24 [2024-10-24T22:50:47,603][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Indexing [429] fieldMappingDocs from logTypes: 24 [2024-10-24T22:50:47,670][INFO ][o.o.c.r.a.AllocationService] [423a42af1e19] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opensearch-observability][0]]]). [2024-10-24T22:50:47,686][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Loading builtin types! [2024-10-24T22:50:47,687][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Indexing [429] fieldMappingDocs from logTypes: 24 [2024-10-24T22:50:47,694][INFO ][o.o.s.i.DetectorIndexManagementService] [423a42af1e19] info deleteOldIndices [2024-10-24T22:50:47,699][INFO ][o.o.s.i.DetectorIndexManagementService] [423a42af1e19] No Old Correlation Indices to delete [2024-10-24T22:50:47,700][INFO ][o.o.s.i.DetectorIndexManagementService] [423a42af1e19] info deleteOldIndices [2024-10-24T22:50:47,703][INFO ][o.o.s.i.DetectorIndexManagementService] [423a42af1e19] No Old IOC Findings Indices to delete [2024-10-24T22:50:47,718][INFO ][o.o.o.i.ObservabilityIndex] [423a42af1e19] observability:Index .opensearch-observability creation Acknowledged [2024-10-24T22:50:47,722][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:47,739][INFO ][o.o.c.m.MetadataCreateIndexService] [423a42af1e19] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1] [2024-10-24T22:50:47,741][INFO ][o.o.c.r.a.AllocationService] [423a42af1e19] updating number_of_replicas to [0] for indices [.opendistro_security] [2024-10-24T22:50:47,742][WARN ][o.o.c.r.a.AllocationService] [423a42af1e19] Falling back to single shard assignment since batch mode disable or multiple custom allocators set [2024-10-24T22:50:47,798][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:47,844][WARN ][o.o.c.r.a.AllocationService] [423a42af1e19] Falling back to single shard assignment since batch mode disable or multiple custom allocators set [2024-10-24T22:50:47,851][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Indexing [429] fieldMappingDocs [2024-10-24T22:50:47,855][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Indexing [429] fieldMappingDocs [2024-10-24T22:50:47,915][INFO ][o.o.c.r.a.AllocationService] [423a42af1e19] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]). [2024-10-24T22:50:47,978][INFO ][o.o.s.c.ConfigurationRepository] [423a42af1e19] Index .opendistro_security created?: true [2024-10-24T22:50:47,980][INFO ][o.o.s.c.ConfigurationRepository] [423a42af1e19] Node started, try to initialize it. Wait for at least yellow cluster state.... [2024-10-24T22:50:47,985][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Will update 'config' with /usr/share/opensearch/config/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-10-24T22:50:47,990][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Indexing [429] fieldMappingDocs [2024-10-24T22:50:47,986][WARN ][o.o.c.r.a.AllocationService] [423a42af1e19] Falling back to single shard assignment since batch mode disable or multiple custom allocators set [2024-10-24T22:50:48,042][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/md1QlaxBSQ6SuB1jDVF_sg] [2024-10-24T22:50:48,063][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opensearch-sap-log-types-config/md1QlaxBSQ6SuB1jDVF_sg] update_mapping [_doc] [2024-10-24T22:50:48,155][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/md1QlaxBSQ6SuB1jDVF_sg] [2024-10-24T22:50:48,231][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:48,239][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] create_mapping [2024-10-24T22:50:48,293][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/md1QlaxBSQ6SuB1jDVF_sg] [2024-10-24T22:50:48,309][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opensearch-sap-log-types-config/md1QlaxBSQ6SuB1jDVF_sg] update_mapping [_doc] [2024-10-24T22:50:48,414][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/md1QlaxBSQ6SuB1jDVF_sg] [2024-10-24T22:50:48,543][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Doc with id 'config' and version 2 is updated in .opendistro_security index. [2024-10-24T22:50:48,543][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Will update 'roles' with /usr/share/opensearch/config/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-10-24T22:50:48,564][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:48,571][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] update_mapping [_doc] [2024-10-24T22:50:48,747][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Doc with id 'roles' and version 2 is updated in .opendistro_security index. [2024-10-24T22:50:48,748][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Will update 'rolesmapping' with /usr/share/opensearch/config/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-10-24T22:50:48,768][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:48,783][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] update_mapping [_doc] [2024-10-24T22:50:48,874][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Doc with id 'rolesmapping' and version 2 is updated in .opendistro_security index. [2024-10-24T22:50:48,875][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Will update 'internalusers' with /usr/share/opensearch/config/opensearch-security/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-10-24T22:50:48,885][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:48,899][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] update_mapping [_doc] [2024-10-24T22:50:49,028][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Doc with id 'internalusers' and version 2 is updated in .opendistro_security index. [2024-10-24T22:50:49,029][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Will update 'actiongroups' with /usr/share/opensearch/config/opensearch-security/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-10-24T22:50:49,035][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:49,040][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] update_mapping [_doc] [2024-10-24T22:50:49,176][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Doc with id 'actiongroups' and version 2 is updated in .opendistro_security index. [2024-10-24T22:50:49,177][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Will update 'tenants' with /usr/share/opensearch/config/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-10-24T22:50:49,184][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:49,188][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] update_mapping [_doc] [2024-10-24T22:50:49,309][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Doc with id 'tenants' and version 2 is updated in .opendistro_security index. [2024-10-24T22:50:49,310][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Will update 'nodesdn' with /usr/share/opensearch/config/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true [2024-10-24T22:50:49,315][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:49,320][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] update_mapping [_doc] [2024-10-24T22:50:49,446][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Doc with id 'nodesdn' and version 2 is updated in .opendistro_security index. [2024-10-24T22:50:49,447][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Will update 'whitelist' with /usr/share/opensearch/config/opensearch-security/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true [2024-10-24T22:50:49,453][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:49,462][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] update_mapping [_doc] [2024-10-24T22:50:49,490][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Loaded [429] field mapping docs successfully! [2024-10-24T22:50:49,556][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Indexing [23] customLogTypes [2024-10-24T22:50:49,616][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Doc with id 'whitelist' and version 2 is updated in .opendistro_security index. [2024-10-24T22:50:49,617][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Will update 'allowlist' with /usr/share/opensearch/config/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true [2024-10-24T22:50:49,704][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:49,729][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] update_mapping [_doc] [2024-10-24T22:50:49,835][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Doc with id 'allowlist' and version 2 is updated in .opendistro_security index. [2024-10-24T22:50:49,836][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Will update 'audit' with /usr/share/opensearch/config/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-10-24T22:50:49,875][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] [2024-10-24T22:50:49,892][INFO ][o.o.c.m.MetadataMappingService] [423a42af1e19] [.opendistro_security/5SZ5asGPTeOEJu3Ki9tYCg] update_mapping [_doc] [2024-10-24T22:50:49,978][INFO ][o.o.s.s.ConfigHelper ] [423a42af1e19] Doc with id 'audit' and version 2 is updated in .opendistro_security index. [2024-10-24T22:50:50,428][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Loaded [429] field mapping docs successfully! [2024-10-24T22:50:50,435][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Indexing [23] customLogTypes [2024-10-24T22:50:50,511][INFO ][stdout ] [423a42af1e19] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl [2024-10-24T22:50:50,512][INFO ][stdout ] [423a42af1e19] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent [2024-10-24T22:50:50,513][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing on REST API is enabled. [2024-10-24T22:50:50,513][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing. [2024-10-24T22:50:50,514][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing on Transport API is enabled. [2024-10-24T22:50:50,514][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing. [2024-10-24T22:50:50,515][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing of request body is enabled. [2024-10-24T22:50:50,515][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Bulk requests resolution is disabled during request auditing. [2024-10-24T22:50:50,515][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Index resolution is enabled during request auditing. [2024-10-24T22:50:50,516][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Sensitive headers exclusion from auditing is enabled. [2024-10-24T22:50:50,516][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing requests from kibanaserver users is disabled. [2024-10-24T22:50:50,516][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing request headers is disabled. [2024-10-24T22:50:50,516][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing request url params is disabled. [2024-10-24T22:50:50,527][WARN ][o.o.s.a.r.AuditMessageRouter] [423a42af1e19] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint [2024-10-24T22:50:50,527][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing of external configuration is disabled. [2024-10-24T22:50:50,528][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing of internal configuration is enabled. [2024-10-24T22:50:50,528][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing only metadata information for read request is enabled. [2024-10-24T22:50:50,528][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing will watch {} for read requests. [2024-10-24T22:50:50,529][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing read operation requests from kibanaserver users is disabled. [2024-10-24T22:50:50,529][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing only metadata information for write request is enabled. [2024-10-24T22:50:50,529][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing diffs for write requests is disabled. [2024-10-24T22:50:50,531][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing write operation requests from kibanaserver users is disabled. [2024-10-24T22:50:50,532][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Auditing will watch for write requests. [2024-10-24T22:50:50,532][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] .opendistro_security is used as internal security index. [2024-10-24T22:50:50,533][INFO ][o.o.s.a.i.AuditLogImpl ] [423a42af1e19] Internal index used for posting audit logs is null [2024-10-24T22:50:50,533][INFO ][o.o.s.c.ConfigurationRepository] [423a42af1e19] Hot-reloading of audit configuration is enabled [2024-10-24T22:50:50,535][INFO ][o.o.s.c.ConfigurationRepository] [423a42af1e19] Node '423a42af1e19' initialized [2024-10-24T22:50:50,879][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Loaded [429] field mapping docs successfully! [2024-10-24T22:50:50,934][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Indexing [23] customLogTypes [2024-10-24T22:50:51,020][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Loaded [23] customLogType docs successfully! [2024-10-24T22:50:51,151][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Loaded [23] customLogType docs successfully! [2024-10-24T22:50:51,271][INFO ][o.o.s.l.LogTypeService ] [423a42af1e19] Loaded [23] customLogType docs successfully! [2024-10-24T22:50:51,272][INFO ][o.o.s.SecurityAnalyticsPlugin] [423a42af1e19] LogType config index successfully created and builtin log types loaded [2024-10-24T22:50:51,294][INFO ][o.o.s.i.DetectorIndexManagementService] [423a42af1e19] info deleteOldIndices [2024-10-24T22:50:51,295][INFO ][o.o.s.i.DetectorIndexManagementService] [423a42af1e19] No Old Alert Indices to delete [2024-10-24T22:50:51,304][INFO ][o.o.s.i.DetectorIndexManagementService] [423a42af1e19] info deleteOldIndices [2024-10-24T22:50:51,305][INFO ][o.o.s.i.DetectorIndexManagementService] [423a42af1e19] No Old Finding Indices to delete [2024-10-24T22:50:56,504][INFO ][o.o.m.a.MLModelAutoReDeployer] [423a42af1e19] Index not found, not performing auto reloading! [2024-10-24T22:50:56,505][INFO ][o.o.m.c.MLCommonsClusterManagerEventListener] [423a42af1e19] Starting ML sync up job... [2024-10-24T22:51:06,512][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.plugins-ml-config/gRxcZxGKQ26TFMj-LSx9Wg] [2024-10-24T22:51:06,520][INFO ][o.o.c.m.MetadataCreateIndexService] [423a42af1e19] [.plugins-ml-config] creating index, cause [api], templates [], shards [1]/[1] [2024-10-24T22:51:06,529][INFO ][o.o.c.r.a.AllocationService] [423a42af1e19] updating number_of_replicas to [0] for indices [.plugins-ml-config] [2024-10-24T22:51:06,530][WARN ][o.o.c.r.a.AllocationService] [423a42af1e19] Falling back to single shard assignment since batch mode disable or multiple custom allocators set [2024-10-24T22:51:06,570][INFO ][o.o.p.PluginsService ] [423a42af1e19] PluginService:onIndexModule index:[.plugins-ml-config/gRxcZxGKQ26TFMj-LSx9Wg] [2024-10-24T22:51:06,638][INFO ][o.o.c.r.a.AllocationService] [423a42af1e19] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-ml-config][0]]]). [2024-10-24T22:51:06,676][INFO ][o.o.m.e.i.MLIndicesHandler] [423a42af1e19] create index:.plugins-ml-config [2024-10-24T22:51:06,682][WARN ][o.o.c.r.a.AllocationService] [423a42af1e19] Falling back to single shard assignment since batch mode disable or multiple custom allocators set [2024-10-24T22:51:06,719][INFO ][o.o.m.c.MLSyncUpCron ] [423a42af1e19] ML configuration initialized successfully [2024-10-24T22:51:46,499][INFO ][o.o.i.i.ManagedIndexCoordinator] [423a42af1e19] Performing move cluster state metadata. [2024-10-24T22:51:46,500][INFO ][o.o.i.i.MetadataService ] [423a42af1e19] ISM config index not exist, so we cancel the metadata migration job. [2024-10-24T22:51:46,501][INFO ][o.o.i.i.ManagedIndexCoordinator] [423a42af1e19] Performing ISM template migration. [2024-10-24T22:51:46,503][INFO ][o.o.i.i.m.ISMTemplateService] [423a42af1e19] Doing ISM template migration 1 time. [2024-10-24T22:51:46,504][INFO ][o.o.i.i.m.ISMTemplateService] [423a42af1e19] Use 2024-10-24T21:50:46.497Z as migrating ISM template last_updated_time [2024-10-24T22:51:46,505][INFO ][o.o.i.i.m.ISMTemplateService] [423a42af1e19] ISM templates: {} [2024-10-24T22:51:46,506][INFO ][o.o.i.i.m.ISMTemplateService] [423a42af1e19] Policies to update: [] [2024-10-24T22:51:46,515][INFO ][o.o.i.i.m.ISMTemplateService] [423a42af1e19] Failure experienced when migrating ISM Template and update ISM policies: {} [2024-10-24T22:51:46,579][INFO ][o.o.c.s.ClusterSettings ] [423a42af1e19] updating [plugins.index_state_management.template_migration.control] from [0] to [-1] [2024-10-24T22:51:46,594][WARN ][o.o.c.r.a.AllocationService] [423a42af1e19] Falling back to single shard assignment since batch mode disable or multiple custom allocators set [2024-10-24T22:51:46,596][INFO ][o.o.i.i.m.ISMTemplateService] [423a42af1e19] Successfully update template migration setting ```

Now that I've realised my stupidity, I'm not surprised that it works well. With an appropriate CPU. 😆

The CPU of the host I used is not supporting x86-64-v2 🤷‍♂️ 😿

opensearch-project / opensearch-build