DEVELOPER_GUIDE.md is missing required steps for setup

AbhinavGarg90 commented 11 months ago

The DEVELOPER_GUIDE.md is missing the following critical steps:

Generating development certificates via cargo make generate-certs

What is the bug?

Upon trying to clone the repository and run the program, following the directives in DEVELOPER_GUIDE.md, an error ocurred:

error: couldn't read opensearch/tests/../../.ci/certs/root-ca.crt: No such file or directory (os error 2)
  --> opensearch/tests/cert.rs:33:25
   |
33 | static CA_CERT: &[u8] = include_bytes!("../../.ci/certs/root-ca.crt");
   |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: this error originates in the macro `include_bytes` (in Nightly builds, run with -Z macro-backtrace for more info)

error: couldn't read opensearch/tests/../../.ci/certs/esnode.crt: No such file or directory (os error 2)
  --> opensearch/tests/cert.rs:35:29
   |
35 | static ESNODE_CERT: &[u8] = include_bytes!("../../.ci/certs/esnode.crt");
   |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: this error originates in the macro `include_bytes` (in Nightly builds, run with -Z macro-backtrace for more info)

error: couldn't read opensearch/tests/../../.ci/certs/esnode-no-san.crt: No such file or directory (os error 2)
  --> opensearch/tests/cert.rs:36:36
   |
36 | static ESNODE_NO_SAN_CERT: &[u8] = include_bytes!("../../.ci/certs/esnode-no-san.crt");
   |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: this error originates in the macro `include_bytes` (in Nightly builds, run with -Z macro-backtrace for more info)

error: could not compile `opensearch` (test "cert") due to 3 previous errors

How can one reproduce the bug?

Clone repository
run cargo install --force cargo-make
run cargo test

What is the expected behavior?

Program should pass the test. If additional requirements need to be met before the program can be run, can they be added to the DEVELOPER_GUIDE.md?

What is your host/environment?

WSL Ubuntu 22.04.3 LTS libssl-dev installed.

Do you have any additional context?

Despite my best efforts, if there was a mistake made on my end in setup, I regret any time wasted.

Xtansia commented 11 months ago

Hi @AbhinavGarg90,

I must have missed updating the developers guide after I made some changes to how we use development/testing SSL certs.

You should be able to remediate this by running cargo make generate-certs or by directly invoking the .ci/generate-certs.sh script file.

If you'd like to make those documentation changes yourself that'd be awesome otherwise I'll do a run through of the steps in a clean environment and add any missing points when I get a chance next week.

If you hit any other issues please add them here so we can make sure the guide gets fixed, and if others hit the same ones they'll have a reference point.

AbhinavGarg90 commented 11 months ago

additionally I am getting the following error when I run cargo make generate-certs

[cargo-make] ERROR - Error while executing command, error: Os {
    code: 2,
    kind: NotFound,
    message: "No such file or directory",
}

The shell command works

AbhinavGarg90 commented 11 months ago

It also seems that pkg-config is required but not explicitely mentioned in the doumentation.

AbhinavGarg90 commented 11 months ago

Hi @AbhinavGarg90,

I must have missed updating the developers guide after I made some changes to how we use development/testing SSL certs.

You should be able to remediate this by running cargo make generate-certs or by directly invoking the .ci/generate-certs.sh script file.

If you'd like to make those documentation changes yourself that'd be awesome otherwise I'll do a run through of the steps in a clean environment and add any missing points when I get a chance next week.

If you hit any other issues please add them here so we can make sure the guide gets fixed, and if others hit the same ones they'll have a reference point.

I tried using this, it generated certificates though it still seems that there is a problem with the generated certificates.

test default_certificate_validation ... FAILED
test full_certificate_validation ... FAILED
test full_certificate_ca_chain_validation ... FAILED
test certificate_certificate_ca_validation ... FAILED
test full_certificate_ca_validation ... FAILED
test certificate_certificate_validation ... FAILED
test none_certificate_validation ... FAILED
test fail_certificate_certificate_validation ... FAILED

Xtansia commented 11 months ago

additionally I am getting the following error when I run cargo make generate-certs
[cargo-make] ERROR - Error while executing command, error: Os {
    code: 2,
    kind: NotFound,
    message: "No such file or directory",
}
The shell command works

I think this is due to not having bash accessible, which may be due to running on Windows without Git Bash installed and on the PATH. Will need to note this.

It also seems that pkg-config is required but not explicitely mentioned in the doumentation.

I don't believe pkg-config is an explicit dependency of this crate, but might be needed by the openssl implementation crate.

I tried using this, it generated certificates though it still seems that there is a problem with the generated certificates.

test default_certificate_validation ... FAILED
test full_certificate_validation ... FAILED
test full_certificate_ca_chain_validation ... FAILED
test certificate_certificate_ca_validation ... FAILED
test full_certificate_ca_validation ... FAILED
test certificate_certificate_validation ... FAILED
test none_certificate_validation ... FAILED
test fail_certificate_certificate_validation ... FAILED

They should pass, could you please include the full error output so I can see what's actually failing?

AbhinavGarg90 commented 11 months ago

   Compiling semver-parser v0.7.0
   Compiling serde v1.0.188
   Compiling openssl v0.10.57
   Compiling tokio v1.32.0
   Compiling object v0.32.1
   Compiling futures-util v0.3.28
   Compiling tracing v0.1.37
   Compiling darling_core v0.13.4
   Compiling tokio-io v0.1.13
   Compiling mio v0.6.23
   Compiling doc-comment v0.3.3
   Compiling tokio-threadpool v0.1.18
   Compiling base64 v0.11.0
   Compiling num-traits v0.2.16
   Compiling want v0.2.0
   Compiling time-macros v0.2.14
   Compiling sysinfo v0.12.0
   Compiling semver v0.9.0
   Compiling h2 v0.1.26
   Compiling os_type v2.6.0
   Compiling unicode-xid v0.0.4
   Compiling rustc_version v0.2.3
   Compiling synom v0.11.3
   Compiling colored v1.9.4
   Compiling quote v0.3.15
   Compiling number_prefix v0.3.0
   Compiling parking_lot_core v0.6.3
   Compiling parking_lot v0.9.0
   Compiling opensearch v1.0.0 (/home/abhinavgarg/osci/opensearch-rs/opensearch)
   Compiling hyper v0.12.36
   Compiling syn v0.11.11
   Compiling indicatif v0.12.0
   Compiling itertools v0.10.5
   Compiling path-slash v0.1.5
   Compiling linked-hash-map v0.5.6
   Compiling time v0.3.28
   Compiling sysinfo v0.9.6
   Compiling yaml-rust v0.4.5
   Compiling serde_json v1.0.107
   Compiling native-tls v0.2.11
   Compiling serde_urlencoded v0.7.1
   Compiling darling_macro v0.13.4
   Compiling backtrace v0.3.69
   Compiling futures-executor v0.3.28
   Compiling tokio-reactor v0.1.12
   Compiling futures v0.3.28
   Compiling chrono v0.4.31
   Compiling tokio-tcp v0.1.4
   Compiling serde_urlencoded v0.5.5
   Compiling simple_logger v1.16.0
   Compiling tokio v0.1.22
   Compiling toml v0.5.11
   Compiling tokio-util v0.7.9
   Compiling tokio-native-tls v0.3.1
   Compiling failure v0.1.8
   Compiling async-compression v0.4.3
   Compiling serde_yaml v0.8.26
   Compiling cookie_store v0.7.0
   Compiling h2 v0.3.21
   Compiling darling v0.13.4
   Compiling serde_with_macros v1.5.2
   Compiling hyper-tls v0.3.2
   Compiling reqwest v0.9.24
   Compiling serde_with v1.14.0
   Compiling api_generator v1.0.0 (/home/abhinavgarg/osci/opensearch-rs/api_generator)
   Compiling hyper v0.14.27
warning: use of deprecated method `url::Url::into_string`: use Into<String>
   --> api_generator/src/generator/mod.rs:310:19
    |
310 |                 u.into_string()
    |                   ^^^^^^^^^^^
    |
    = note: `#[warn(deprecated)]` on by default

warning: use of deprecated method `itertools::Itertools::fold1`: Use `Iterator::reduce` instead
   --> api_generator/src/generator/mod.rs:717:10
    |
717 |         .fold1(|d1, d2| Deprecated::combine(d1, d2))
    |          ^^^^^

warning: `api_generator` (lib) generated 2 warnings
   Compiling hyper-tls v0.5.0
   Compiling reqwest v0.11.20
warning: unused import: `regex::Regex`
   --> opensearch/src/http/transport.rs:518:9
    |
518 |     use regex::Regex;
    |         ^^^^^^^^^^^^
    |
    = note: `#[warn(unused_imports)]` on by default

warning: `api_generator` (lib test) generated 2 warnings (2 duplicates)
   Compiling yaml_test_runner v1.0.0 (/home/abhinavgarg/osci/opensearch-rs/yaml_test_runner)
warning: unused import: `Component`
  --> yaml_test_runner/src/generator.rs:44:12
   |
44 |     path::{Component, Path, PathBuf},
   |            ^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

warning: field `suite` is never read
  --> yaml_test_runner/src/generator.rs:59:5
   |
55 | struct YamlTests<'a> {
   |        --------- field in this struct
...
59 |     suite: TestSuite,
   |     ^^^^^
   |
   = note: `#[warn(dead_code)]` on by default

warning: `yaml_test_runner` (bin "yaml_test_runner" test) generated 2 warnings (run `cargo fix --bin "yaml_test_runner" --tests` to apply 1 suggestion)
warning: `opensearch` (lib test) generated 1 warning (run `cargo fix --lib -p opensearch --tests` to apply 1 suggestion)
    Finished test [unoptimized + debuginfo] target(s) in 1m 25s
     Running unittests src/lib.rs (target/debug/deps/api_generator-34e7091aab222d56)

running 5 tests
test generator::code_gen::url::enum_builder::tests::generate_parts_enum_from_endpoint ... ignored
test generator::tests::stability_ordering ... ok
test generator::tests::combine_deprecations ... ok
test generator::output::test::nominal_merge ... ok
test generator::output::test::unbalanced_sections ... ok

test result: ok. 4 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.02s

     Running unittests src/bin/run.rs (target/debug/deps/run-26e6a78bab96e9c1)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/opensearch-76a8226a33869e4f)

running 24 tests
test http::request::tests::bodies_into_ndbody_writes_to_bytes ... ok
test http::request::tests::bytes_body_returns_usable_buf ... ok
test http::request::tests::bytes_slice_body_writes_to_bytes_mut ... ok
test http::request::tests::bytes_body_writes_to_bytes_mut ... ok
test http::request::tests::serialize_into_jsonbody_writes_to_bytes ... ok
test http::request::tests::string_body_writes_to_bytes_mut ... ok
test http::request::tests::string_slice_body_writes_to_bytes_mut ... ok
test http::request::tests::vec_body_writes_to_bytes_mut ... ok
test http::response::tests::deserialize_error_string ... ok
test http::response::tests::deserialize_illegal_argument_exception ... ok
test http::response::tests::deserialize_index_not_found_exception ... ok
test http::transport::tests::connection_url_with_trailing_slash ... ok
test http::transport::tests::connection_url_with_no_path_and_trailing_slash ... ok
test params::tests::serialize_slices_auto ... ok
test http::transport::tests::connection_url_with_no_trailing_slash ... ok
test http::transport::tests::connection_url_with_no_path_and_no_trailing_slash ... ok
test tests::build_search_on_all_indices_and_types ... ok
test tests::build_search_on_selected_indices ... ok
test params::tests::serialize_slices_count ... ok
test root::bulk::tests::serialize_bulk_operations_with_different_types_writes_to_bytes ... ok
test tests::build_search_on_selected_indices_and_types ... ok
test tests::percent_encode_characters ... ok
test root::bulk::tests::serialize_bulk_operations_with_same_type_writes_to_bytes ... ok
test http::transport::tests::invalid_pkcs12_cert_credentials ... ok

test result: ok. 24 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

     Running tests/auth.rs (target/debug/deps/auth-e1b8e64d0e5bd673)

running 3 tests
test basic_auth_header ... ok
test bearer_header ... ok
test api_key_header ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.17s

     Running tests/cert.rs (target/debug/deps/cert-da31c5f10722c316)

running 8 tests
test none_certificate_validation ... FAILED
test full_certificate_ca_chain_validation ... FAILED
test certificate_certificate_ca_validation ... FAILED
test full_certificate_ca_validation ... FAILED
test fail_certificate_certificate_validation ... FAILED
test default_certificate_validation ... FAILED
test certificate_certificate_validation ... FAILED
test full_certificate_validation ... FAILED

failures:

---- none_certificate_validation stdout ----
Error: Error { kind: Http(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(9200), path: "/", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }) }

---- full_certificate_ca_chain_validation stdout ----
Error: Error { kind: Http(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(9200), path: "/", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }) }

---- certificate_certificate_ca_validation stdout ----
Error: Error { kind: Http(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(9200), path: "/", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }) }

---- full_certificate_ca_validation stdout ----
Error: Error { kind: Http(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(9200), path: "/", query: None, fragment: None }, source: hyper::Error(IncompleteMessage) }) }

---- fail_certificate_certificate_validation stdout ----
thread 'fail_certificate_certificate_validation' panicked at 'Expected error message to contain 'unable to get local issuer certificate' but was 'error sending request for url (http://localhost:9200/): connection closed before message completed'', opensearch/tests/cert.rs:285:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- default_certificate_validation stdout ----
thread 'default_certificate_validation' panicked at 'Expected error message to contain 'unable to get local issuer certificate' but was 'error sending request for url (http://localhost:9200/): connection closed before message completed'', opensearch/tests/cert.rs:67:13

---- certificate_certificate_validation stdout ----
thread 'certificate_certificate_validation' panicked at 'Expected error message to contain 'unable to get local issuer certificate' but was 'error sending request for url (http://localhost:9200/): connection closed before message completed'', opensearch/tests/cert.rs:241:17

---- full_certificate_validation stdout ----
thread 'full_certificate_validation' panicked at 'Expected error message to contain 'unable to get local issuer certificate' but was 'error sending request for url (http://localhost:9200/): connection closed before message completed'', opensearch/tests/cert.rs:193:17

failures:
    certificate_certificate_ca_validation
    certificate_certificate_validation
    default_certificate_validation
    fail_certificate_certificate_validation
    full_certificate_ca_chain_validation
    full_certificate_ca_validation
    full_certificate_validation
    none_certificate_validation

test result: FAILED. 0 passed; 8 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.42s

error: test failed, to rerun pass `-p opensearch --test cert`

Xtansia commented 11 months ago

@AbhinavGarg90 Could you try and see if cargo make test works for you rather than just cargo test?

AbhinavGarg90 commented 11 months ago

Not working either unfortunately.

~/osci/opensearch-rs$ cargo make test
[cargo-make] INFO - cargo make 0.37.1
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: test
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: legacy-migration
[cargo-make] INFO - Execute Command: "./.ci/run-opensearch.sh"
/home/abhinavgarg/osci/opensearch-rs/.ci/functions/imports.sh: line 18: CLUSTER: unbound variable
[cargo-make] ERROR - Error while executing command, exit code: 1
[cargo-make] WARN - Build Failed.
abhinavgarg@LAPTOP-OAA0RE3F:~/osci/opensearch-rs$ cargo make test
[cargo-make] INFO - cargo make 0.37.1
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: test
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: legacy-migration
[cargo-make] INFO - Execute Command: "./.ci/run-opensearch.sh"
/home/abhinavgarg/osci/opensearch-rs/.ci/functions/imports.sh: line 18: CLUSTER: unbound variable
[cargo-make] ERROR - Error while executing command, exit code: 1
[cargo-make] WARN - Build Failed.

Xtansia commented 11 months ago

Not working either unfortunately.

~/osci/opensearch-rs$ cargo make test
[cargo-make] INFO - cargo make 0.37.1
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: test
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: legacy-migration
[cargo-make] INFO - Execute Command: "./.ci/run-opensearch.sh"
/home/abhinavgarg/osci/opensearch-rs/.ci/functions/imports.sh: line 18: CLUSTER: unbound variable
[cargo-make] ERROR - Error while executing command, exit code: 1
[cargo-make] WARN - Build Failed.
abhinavgarg@LAPTOP-OAA0RE3F:~/osci/opensearch-rs$ cargo make test
[cargo-make] INFO - cargo make 0.37.1
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: test
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: legacy-migration
[cargo-make] INFO - Execute Command: "./.ci/run-opensearch.sh"
/home/abhinavgarg/osci/opensearch-rs/.ci/functions/imports.sh: line 18: CLUSTER: unbound variable
[cargo-make] ERROR - Error while executing command, exit code: 1
[cargo-make] WARN - Build Failed.

That is odd as the CLUSTER variable no longer exists in the current main branch (https://github.com/opensearch-project/opensearch-rs/blob/main/.ci/functions/imports.sh). What commit/branch do you have checked out?

AbhinavGarg90 commented 11 months ago

Not working either unfortunately.

~/osci/opensearch-rs$ cargo make test
[cargo-make] INFO - cargo make 0.37.1
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: test
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: legacy-migration
[cargo-make] INFO - Execute Command: "./.ci/run-opensearch.sh"
/home/abhinavgarg/osci/opensearch-rs/.ci/functions/imports.sh: line 18: CLUSTER: unbound variable
[cargo-make] ERROR - Error while executing command, exit code: 1
[cargo-make] WARN - Build Failed.
abhinavgarg@LAPTOP-OAA0RE3F:~/osci/opensearch-rs$ cargo make test
[cargo-make] INFO - cargo make 0.37.1
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: test
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: legacy-migration
[cargo-make] INFO - Execute Command: "./.ci/run-opensearch.sh"
/home/abhinavgarg/osci/opensearch-rs/.ci/functions/imports.sh: line 18: CLUSTER: unbound variable
[cargo-make] ERROR - Error while executing command, exit code: 1
[cargo-make] WARN - Build Failed.

That is odd as the CLUSTER variable no longer exists in the current main branch (https://github.com/opensearch-project/opensearch-rs/blob/main/.ci/functions/imports.sh). What commit/branch do you have checked out?

I apologize, I was inspecting the 1.0 release to understand the codebase better, forgot to switch back before running the command. I created a clean instance of the repository, following the instructions you mentioned throughout this thread.

Upon running cargo make test it still had 1 error preventing the build. This is confusing as it buiilt successfully the first time I ran it, but gave the following error on the 2nd attempt.:

~/osci/opensearch-rs$ cargo make test
[cargo-make] INFO - cargo make 0.37.2
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: test
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: legacy-migration
[cargo-make] INFO - Skipping Task: set-coverage-vars 
[cargo-make] INFO - Skipping Task: create-test-results-dir 
[cargo-make] INFO - Execute Command: "./.ci/run-opensearch.sh"
INFO: Running opensearch latest
INFO: Creating network search-rest-test if it does not exist already 
359e7e472e59c51f8a6eb1b289a26f958260d38935fa5d12cccc1f2dab42e194
INFO: Take down node if called twice with the same arguments (DETACH=true) or on seperate terminals 
INFO: Removing volume instance-rest-test-data
0: 9200 https://localhost:9200 
INFO: building opensearch container
[+] Building 0.1s (16/16) FINISHED                                                                                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 882B                                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/opensearchproject/opensearch:latest                                                                                                                                                       0.0s
 => [ 1/11] FROM docker.io/opensearchproject/opensearch:latest                                                                                                                                                                       0.0s
 => [internal] load build context                                                                                                                                                                                                    0.0s
 => => transferring context: 131B                                                                                                                                                                                                    0.0s
 => CACHED [ 2/11] WORKDIR /usr/share/opensearch                                                                                                                                                                                     0.0s
 => CACHED [ 3/11] COPY --chown=opensearch:opensearch opensearch.yml ./config/                                                                                                                                                       0.0s
 => CACHED [ 4/11] RUN if [[ -f "./opensearch-onetime-setup.sh" ]]; then ./opensearch-onetime-setup.sh ; fi                                                                                                                          0.0s
 => CACHED [ 5/11] COPY --chown=opensearch:opensearch *.pem ./config/                                                                                                                                                                0.0s
 => CACHED [ 6/11] RUN sed -i'' 's/plugins.security.audit.type:.*/plugins.security.audit.type: log4j/' ./config/opensearch.yml                                                                                                       0.0s
 => CACHED [ 7/11] RUN echo "appender.header_warning.type = HeaderWarningAppender" >> ./config/log4j2.properties                                                                                                                     0.0s
 => CACHED [ 8/11] RUN echo "appender.header_warning.name = header_warning" >> ./config/log4j2.properties                                                                                                                            0.0s
 => CACHED [ 9/11] RUN echo "logger.deprecation.name = org.opensearch.deprecation" >> ./config/log4j2.properties                                                                                                                     0.0s
 => CACHED [10/11] RUN echo "logger.deprecation.level = deprecation" >> ./config/log4j2.properties                                                                                                                                   0.0s
 => CACHED [11/11] RUN echo "logger.deprecation.appenderRef.header_warning.ref = header_warning" >> ./config/log4j2.properties                                                                                                       0.0s
 => exporting to image                                                                                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                                                                              0.0s
 => => writing image sha256:22ecc4c7dcb3e629a43d05976fd481b70a746efdf623b5feed0cc343007d6f83                                                                                                                                         0.0s
 => => naming to docker.io/library/opensearch-secure-true                                                                                                                                                                            0.0s
INFO: Starting container instance 
+ [[ true == \t\r\u\e ]]
+ healthcmd='curl -vvv -s --cacert /usr/share/opensearch/config/root-ca.pem -u admin:admin --fail https://localhost:9200/_cluster/health || exit 1'
++ echo curl -vvv -s --cacert /usr/share/opensearch/config/root-ca.pem -u admin:admin --fail https://localhost:9200/_cluster/health '||' exit 1
+ docker run --name instance --network search-rest-test --env 'OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g' --env node.name=instance --volume instance-rest-test-data:/usr/share/opensearch/data0 --publish 9200:9200 --ulimit nofile=65536:65536 --ulimit memlock=-1:-1 --detach=true '--health-cmd=curl -vvv -s --cacert /usr/share/opensearch/config/root-ca.pem -u admin:admin --fail https://localhost:9200/_cluster/health || exit 1' --health-interval=2s --health-retries=20 --health-timeout=2s --rm -d opensearch-secure-true
125f0cb1ef709ecedb14a46d657c13f0591294379c9b6153011dc607be311a5f
+ set +x

INFO: waiting for node instance to be up

* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to localhost port 9200 after 0 ms: Couldn't connect to server
* Closing connection
* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to localhost port 9200 after 0 ms: Couldn't connect to server
* Closing connection
* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to localhost port 9200 after 0 ms: Couldn't connect to server
* Closing connection
* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to localhost port 9200 after 0 ms: Couldn't connect to server
* Closing connection

INFO: waiting for node instance to be up
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on Amazon Linux release 2023 (Amazon Linux)
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.10.0
Detected OpenSearch Security Version: 2.10.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.10.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.10.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2023-09-28T07:14:46,789][INFO ][o.o.n.Node               ] [instance] version[2.10.0], pid[31], build[tar/eee49cb340edc6c4d489bcd9324dda571fc8dc03/2023-09-20T23:54:29.889267151Z], OS[Linux/5.15.90.1-microsoft-standard-WSL2/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.8/17.0.8+7]
[2023-09-28T07:14:46,792][INFO ][o.o.n.Node               ] [instance] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2023-09-28T07:14:46,792][INFO ][o.o.n.Node               ] [instance] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-8416197821809689957, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2023-09-28T07:14:47,733][INFO ][o.o.s.s.t.SSLConfig      ] [instance] SSL dual mode is disabled
[2023-09-28T07:14:47,733][INFO ][o.o.s.OpenSearchSecurityPlugin] [instance] OpenSearch Config path is /usr/share/opensearch/config
[2023-09-28T07:14:47,968][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] JVM supports TLSv1.3
[2023-09-28T07:14:47,969][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2023-09-28T07:14:48,565][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] TLS Transport Client Provider : JDK
[2023-09-28T07:14:48,565][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] TLS Transport Server Provider : JDK
[2023-09-28T07:14:48,565][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] TLS HTTP Provider             : JDK
[2023-09-28T07:14:48,566][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2023-09-28T07:14:48,566][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
[2023-09-28T07:14:48,594][INFO ][o.o.s.OpenSearchSecurityPlugin] [instance] Clustername: search-rest-test
[2023-09-28T07:14:48,603][WARN ][o.o.s.OpenSearchSecurityPlugin] [instance] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2023-09-28T07:14:48,603][WARN ][o.o.s.OpenSearchSecurityPlugin] [instance] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2023-09-28T07:14:48,603][WARN ][o.o.s.OpenSearchSecurityPlugin] [instance] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2023-09-28T07:14:48,603][WARN ][o.o.s.OpenSearchSecurityPlugin] [instance] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[2023-09-28T07:14:49,536][INFO ][o.o.p.c.c.PluginSettings ] [instance] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2023-09-28T07:14:50,063][INFO ][o.o.i.r.ReindexPlugin    ] [instance] ReindexPlugin reloadSPI called
[2023-09-28T07:14:50,064][INFO ][o.o.i.r.ReindexPlugin    ] [instance] Unable to find any implementation for RemoteReindexExtension
[2023-09-28T07:14:50,093][INFO ][o.o.j.JobSchedulerPlugin ] [instance] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2023-09-28T07:14:50,114][INFO ][o.o.j.JobSchedulerPlugin ] [instance] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-09-28T07:14:50,115][INFO ][o.o.j.JobSchedulerPlugin ] [instance] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-09-28T07:14:50,116][INFO ][o.o.j.JobSchedulerPlugin ] [instance] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2023-09-28T07:14:50,121][INFO ][o.o.p.PluginsService     ] [instance] loaded module [aggs-matrix-stats]
[2023-09-28T07:14:50,121][INFO ][o.o.p.PluginsService     ] [instance] loaded module [analysis-common]
[2023-09-28T07:14:50,122][INFO ][o.o.p.PluginsService     ] [instance] loaded module [geo]
[2023-09-28T07:14:50,122][INFO ][o.o.p.PluginsService     ] [instance] loaded module [ingest-common]
[2023-09-28T07:14:50,122][INFO ][o.o.p.PluginsService     ] [instance] loaded module [ingest-geoip]
[2023-09-28T07:14:50,122][INFO ][o.o.p.PluginsService     ] [instance] loaded module [ingest-user-agent]
[2023-09-28T07:14:50,122][INFO ][o.o.p.PluginsService     ] [instance] loaded module [lang-expression]
[2023-09-28T07:14:50,122][INFO ][o.o.p.PluginsService     ] [instance] loaded module [lang-mustache]
[2023-09-28T07:14:50,122][INFO ][o.o.p.PluginsService     ] [instance] loaded module [lang-painless]
[2023-09-28T07:14:50,122][INFO ][o.o.p.PluginsService     ] [instance] loaded module [mapper-extras]
[2023-09-28T07:14:50,122][INFO ][o.o.p.PluginsService     ] [instance] loaded module [opensearch-dashboards]
[2023-09-28T07:14:50,122][INFO ][o.o.p.PluginsService     ] [instance] loaded module [parent-join]
[2023-09-28T07:14:50,123][INFO ][o.o.p.PluginsService     ] [instance] loaded module [percolator]
[2023-09-28T07:14:50,123][INFO ][o.o.p.PluginsService     ] [instance] loaded module [rank-eval]
[2023-09-28T07:14:50,123][INFO ][o.o.p.PluginsService     ] [instance] loaded module [reindex]
[2023-09-28T07:14:50,123][INFO ][o.o.p.PluginsService     ] [instance] loaded module [repository-url]
[2023-09-28T07:14:50,123][INFO ][o.o.p.PluginsService     ] [instance] loaded module [search-pipeline-common]
[2023-09-28T07:14:50,123][INFO ][o.o.p.PluginsService     ] [instance] loaded module [systemd]
[2023-09-28T07:14:50,123][INFO ][o.o.p.PluginsService     ] [instance] loaded module [transport-netty4]
[2023-09-28T07:14:50,123][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-alerting]
[2023-09-28T07:14:50,124][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-anomaly-detection]
[2023-09-28T07:14:50,124][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-asynchronous-search]
[2023-09-28T07:14:50,124][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-cross-cluster-replication]
[2023-09-28T07:14:50,124][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-custom-codecs]
[2023-09-28T07:14:50,124][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-geospatial]
[2023-09-28T07:14:50,124][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-index-management]
[2023-09-28T07:14:50,124][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-job-scheduler]
[2023-09-28T07:14:50,124][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-knn]
[2023-09-28T07:14:50,125][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-ml]
[2023-09-28T07:14:50,125][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-neural-search]
[2023-09-28T07:14:50,125][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-notifications]
[2023-09-28T07:14:50,125][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-notifications-core]
[2023-09-28T07:14:50,125][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-observability]
[2023-09-28T07:14:50,125][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-performance-analyzer]
[2023-09-28T07:14:50,125][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-reports-scheduler]
[2023-09-28T07:14:50,125][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-security]
[2023-09-28T07:14:50,125][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-security-analytics]
[2023-09-28T07:14:50,125][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-sql]
[2023-09-28T07:14:50,140][INFO ][o.o.s.OpenSearchSecurityPlugin] [instance] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2023-09-28T07:14:50,171][INFO ][o.o.e.ExtensionsManager  ] [instance] ExtensionsManager initialized
[2023-09-28T07:14:50,180][DEPRECATION][o.o.d.c.s.Settings       ] [instance] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2023-09-28T07:14:50,197][INFO ][o.o.e.NodeEnvironment    ] [instance] using [1] data paths, mounts [[/ (overlay)]], net usable_space [933.5gb], net total_space [1006.8gb], types [overlay]
[2023-09-28T07:14:50,197][INFO ][o.o.e.NodeEnvironment    ] [instance] heap size [1gb], compressed ordinary object pointers [true]
[2023-09-28T07:14:50,228][INFO ][o.o.n.Node               ] [instance] node name [instance], node ID [GbQIyJF3SYSW-PES4srk5w], cluster name [search-rest-test], roles [ingest, remote_cluster_client, data, cluster_manager]
[2023-09-28T07:14:51,985][DEPRECATION][o.o.d.c.s.Settings       ] [instance] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2023-09-28T07:14:51,987][DEPRECATION][o.o.d.c.s.Settings       ] [instance] [cluster.initial_master_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2023-09-28T07:14:52,625][INFO ][o.o.n.p.NeuralSearch     ] [instance] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
[2023-09-28T07:14:52,804][WARN ][o.o.s.c.Salt             ] [instance] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2023-09-28T07:14:52,841][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Message routing enabled: true
[2023-09-28T07:14:52,872][INFO ][o.o.s.f.SecurityFilter   ] [instance] <NONE> indices are made immutable.
[2023-09-28T07:14:53,162][INFO ][o.o.a.b.ADCircuitBreakerService] [instance] Registered memory breaker.
[2023-09-28T07:14:53,466][INFO ][o.o.m.b.MLCircuitBreakerService] [instance] Registered ML memory breaker.
[2023-09-28T07:14:53,466][INFO ][o.o.m.b.MLCircuitBreakerService] [instance] Registered ML disk breaker.
[2023-09-28T07:14:53,467][INFO ][o.o.m.b.MLCircuitBreakerService] [instance] Registered ML native memory breaker.
[2023-09-28T07:14:53,552][INFO ][o.r.Reflections          ] [instance] Reflections took 32 ms to scan 1 urls, producing 17 keys and 43 values 
[2023-09-28T07:14:53,615][WARN ][o.o.s.p.SQLPlugin        ] [instance] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2023-09-28T07:14:54,144][INFO ][o.o.t.NettyAllocator     ] [instance] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2023-09-28T07:14:54,241][INFO ][o.o.d.DiscoveryModule    ] [instance] using discovery type [zen] and seed hosts providers [settings]
[2023-09-28T07:14:54,622][WARN ][o.o.g.DanglingIndicesState] [instance] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-09-28T07:14:55,060][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [instance] PerformanceAnalyzer Enabled: false
[2023-09-28T07:14:55,087][INFO ][o.o.n.Node               ] [instance] initialized
[2023-09-28T07:14:55,088][INFO ][o.o.n.Node               ] [instance] starting ...
[2023-09-28T07:14:55,112][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [windows_logtype.json] log type
[2023-09-28T07:14:55,113][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [vpcflow_logtype.json] log type
[2023-09-28T07:14:55,113][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [test_windows_logtype.json] log type
[2023-09-28T07:14:55,114][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [s3_logtype.json] log type
[2023-09-28T07:14:55,114][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_web_logtype.json] log type
[2023-09-28T07:14:55,114][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_proxy_logtype.json] log type
[2023-09-28T07:14:55,115][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_macos_logtype.json] log type
[2023-09-28T07:14:55,115][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_compliance_logtype.json] log type
[2023-09-28T07:14:55,115][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_cloud_logtype.json] log type
[2023-09-28T07:14:55,116][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_apt_logtype.json] log type
[2023-09-28T07:14:55,116][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_application_logtype.json] log type
[2023-09-28T07:14:55,116][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [okta_logtype.json] log type
[2023-09-28T07:14:55,116][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [network_logtype.json] log type
[2023-09-28T07:14:55,117][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [netflow_logtype.json] log type
[2023-09-28T07:14:55,117][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [m365_logtype.json] log type
[2023-09-28T07:14:55,117][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [linux_logtype.json] log type
[2023-09-28T07:14:55,118][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [gworkspace_logtype.json] log type
[2023-09-28T07:14:55,118][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [github_logtype.json] log type
[2023-09-28T07:14:55,119][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [dns_logtype.json] log type
[2023-09-28T07:14:55,119][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [cloudtrail_logtype.json] log type
[2023-09-28T07:14:55,120][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [azure_logtype.json] log type
[2023-09-28T07:14:55,120][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [apache_access_logtype.json] log type
[2023-09-28T07:14:55,121][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [ad_ldap_logtype.json] log type
[2023-09-28T07:14:55,195][INFO ][o.o.t.TransportService   ] [instance] publish_address {172.23.0.2:9300}, bound_addresses {0.0.0.0:9300}
[2023-09-28T07:14:55,196][INFO ][o.o.t.TransportService   ] [instance] Remote clusters initialized successfully.
[2023-09-28T07:14:55,318][INFO ][o.o.b.BootstrapChecks    ] [instance] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-09-28T07:14:55,329][INFO ][o.o.c.c.Coordinator      ] [instance] setting initial configuration to VotingConfiguration{GbQIyJF3SYSW-PES4srk5w}
[2023-09-28T07:14:55,444][INFO ][o.o.c.s.MasterService    ] [instance] elected-as-cluster-manager ([1] nodes joined)[{instance}{GbQIyJF3SYSW-PES4srk5w}{dAdK4UwZQfazsaQyaRIlHA}{172.23.0.2}{172.23.0.2:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{instance}{GbQIyJF3SYSW-PES4srk5w}{dAdK4UwZQfazsaQyaRIlHA}{172.23.0.2}{172.23.0.2:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]}
[2023-09-28T07:14:55,473][INFO ][o.o.c.c.CoordinationState] [instance] cluster UUID set to [JEOVuaJGTmGt1bHKlxki-A]
[2023-09-28T07:14:55,499][INFO ][o.o.c.s.ClusterApplierService] [instance] cluster-manager node changed {previous [], current [{instance}{GbQIyJF3SYSW-PES4srk5w}{dAdK4UwZQfazsaQyaRIlHA}{172.23.0.2}{172.23.0.2:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2023-09-28T07:14:55,505][INFO ][o.o.a.c.ADClusterEventListener] [instance] Cluster is not recovered yet.
[2023-09-28T07:14:55,509][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:55,523][INFO ][o.o.i.i.ManagedIndexCoordinator] [instance] Cache cluster manager node onClusterManager time: 1695885295523
[2023-09-28T07:14:55,533][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [instance] Config override setting update called with empty string. Ignoring.
[2023-09-28T07:14:55,538][INFO ][o.o.d.PeerFinder         ] [instance] setting findPeersInterval to [1s] as node commission status = [true] for local node [{instance}{GbQIyJF3SYSW-PES4srk5w}{dAdK4UwZQfazsaQyaRIlHA}{172.23.0.2}{172.23.0.2:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]
[2023-09-28T07:14:55,542][INFO ][o.o.h.AbstractHttpServerTransport] [instance] publish_address {172.23.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2023-09-28T07:14:55,542][INFO ][o.o.n.Node               ] [instance] started
[2023-09-28T07:14:55,542][INFO ][o.o.s.OpenSearchSecurityPlugin] [instance] Node started
[2023-09-28T07:14:55,543][INFO ][o.o.s.c.ConfigurationRepository] [instance] Will attempt to create index .opendistro_security and default configs if they are absent
[2023-09-28T07:14:55,544][INFO ][o.o.s.c.ConfigurationRepository] [instance] Background init thread started. Install default config?: true
[2023-09-28T07:14:55,544][INFO ][o.o.s.OpenSearchSecurityPlugin] [instance] 0 OpenSearch Security modules loaded so far: []
[2023-09-28T07:14:55,544][INFO ][o.o.s.c.ConfigurationRepository] [instance] Wait for cluster to be available ...
[2023-09-28T07:14:55,557][INFO ][o.o.a.c.HashRing         ] [instance] Node added: [GbQIyJF3SYSW-PES4srk5w]
[2023-09-28T07:14:55,558][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:55,559][INFO ][o.o.a.c.HashRing         ] [instance] Add data node to AD version hash ring: GbQIyJF3SYSW-PES4srk5w
[2023-09-28T07:14:55,561][INFO ][o.o.a.c.HashRing         ] [instance] All nodes with known AD version: {GbQIyJF3SYSW-PES4srk5w=ADNodeInfo{version=2.10.0, isEligibleDataNode=true}}
[2023-09-28T07:14:55,562][INFO ][o.o.a.c.HashRing         ] [instance] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2023-09-28T07:14:55,562][INFO ][o.o.a.c.HashRing         ] [instance] Build AD version hash ring successfully
[2023-09-28T07:14:55,562][INFO ][o.o.a.c.ADDataMigrator   ] [instance] Start migrating AD data
[2023-09-28T07:14:55,562][INFO ][o.o.a.c.ADDataMigrator   ] [instance] AD job index doesn't exist, no need to migrate
[2023-09-28T07:14:55,562][INFO ][o.o.a.c.ADClusterEventListener] [instance] Init AD version hash ring successfully
[2023-09-28T07:14:55,574][INFO ][o.o.g.GatewayService     ] [instance] recovered [0] indices into cluster_state
[2023-09-28T07:14:55,577][DEPRECATION][o.o.d.c.m.MetadataCreateIndexService] [instance] index name [.opensearch-observability] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices
[2023-09-28T07:14:55,594][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-observability/O4LQEiMXTI6NiGz6T0cOTw]
[2023-09-28T07:14:55,700][INFO ][o.o.c.m.MetadataCreateIndexService] [instance] [.opensearch-observability] creating index, cause [api], templates [], shards [1]/[0]
[2023-09-28T07:14:55,750][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-observability/O4LQEiMXTI6NiGz6T0cOTw]
[2023-09-28T07:14:55,791][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:55,795][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/647tZ3HUSvKzpDnmq-jPoA]
[2023-09-28T07:14:55,806][INFO ][o.o.c.m.MetadataCreateIndexService] [instance] [.opensearch-sap-log-types-config] creating index, cause [auto(sap-logtype api)], templates [], shards [1]/[1]
[2023-09-28T07:14:55,807][INFO ][o.o.c.r.a.AllocationService] [instance] updating number_of_replicas to [0] for indices [.opensearch-sap-log-types-config]
[2023-09-28T07:14:55,838][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/647tZ3HUSvKzpDnmq-jPoA]
[2023-09-28T07:14:55,857][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:55,947][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:55,949][INFO ][o.o.s.l.LogTypeService   ] [instance] Loading builtin types!
[2023-09-28T07:14:55,952][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [418] fieldMappingDocs from logTypes: 23
[2023-09-28T07:14:55,982][INFO ][o.o.c.r.a.AllocationService] [instance] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opensearch-observability][0]]]).
[2023-09-28T07:14:56,008][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,009][INFO ][o.o.o.i.ObservabilityIndex] [instance] observability:Index .opensearch-observability creation Acknowledged
[2023-09-28T07:14:56,009][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [instance] observability:createMappingTemplate ss4o_metrics_template API called
[2023-09-28T07:14:56,021][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [418] fieldMappingDocs
[2023-09-28T07:14:56,022][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[validate-template-zefqu83hszqxxcadxyusyq/qN0WRPlkSDCFu16kr9N6gg]
[2023-09-28T07:14:56,047][INFO ][o.o.c.m.MetadataIndexTemplateService] [instance] adding index template [ss4o_metrics_template] for index patterns [ss4o_metrics-*-*]
[2023-09-28T07:14:56,073][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,075][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [instance] observability:Mapping Template ss4o_metrics_template creation Acknowledged
[2023-09-28T07:14:56,075][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [instance] observability:createMappingTemplate ss4o_traces_template API called
[2023-09-28T07:14:56,077][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/647tZ3HUSvKzpDnmq-jPoA]
[2023-09-28T07:14:56,089][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opensearch-sap-log-types-config/647tZ3HUSvKzpDnmq-jPoA] update_mapping [_doc]
[2023-09-28T07:14:56,121][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,125][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[validate-template-wx0vfffvtyg7puq-grgaow/c_N-XBjMRZOfscfD3Q1kWg]
[2023-09-28T07:14:56,137][INFO ][o.o.c.m.MetadataIndexTemplateService] [instance] adding index template [ss4o_traces_template] for index patterns [ss4o_traces-*-*]
[2023-09-28T07:14:56,167][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,168][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [instance] observability:Mapping Template ss4o_traces_template creation Acknowledged
[2023-09-28T07:14:56,170][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/647tZ3HUSvKzpDnmq-jPoA]
[2023-09-28T07:14:56,199][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/647tZ3HUSvKzpDnmq-jPoA]
[2023-09-28T07:14:56,207][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opensearch-sap-log-types-config/647tZ3HUSvKzpDnmq-jPoA] update_mapping [_doc]
[2023-09-28T07:14:56,237][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,513][INFO ][o.o.s.l.LogTypeService   ] [instance] Loaded [418] field mapping docs successfully!
[2023-09-28T07:14:56,527][ERROR][o.o.s.a.BackendRegistry  ] [instance] Not yet initialized (you may need to run securityadmin)
[2023-09-28T07:14:56,532][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [22] customLogTypes
[2023-09-28T07:14:56,549][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:56,554][INFO ][o.o.c.m.MetadataCreateIndexService] [instance] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1]
[2023-09-28T07:14:56,555][INFO ][o.o.c.r.a.AllocationService] [instance] updating number_of_replicas to [0] for indices [.opendistro_security]
[2023-09-28T07:14:56,555][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [22] customLogTypes
[2023-09-28T07:14:56,555][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [22] customLogTypes
[2023-09-28T07:14:56,566][INFO ][o.o.s.l.LogTypeService   ] [instance] Loaded [22] customLogType docs successfully!
[2023-09-28T07:14:56,566][INFO ][o.o.s.SecurityAnalyticsPlugin] [instance] LogType config index successfully created and builtin log types loaded
[2023-09-28T07:14:56,582][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:56,596][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,633][INFO ][o.o.s.l.LogTypeService   ] [instance] Loaded [22] customLogType docs successfully!
[2023-09-28T07:14:56,634][INFO ][o.o.c.r.a.AllocationService] [instance] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]).
[2023-09-28T07:14:56,640][INFO ][o.o.s.l.LogTypeService   ] [instance] Loaded [22] customLogType docs successfully!
[2023-09-28T07:14:56,663][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,664][INFO ][o.o.s.c.ConfigurationRepository] [instance] Index .opendistro_security created?: true
[2023-09-28T07:14:56,664][INFO ][o.o.s.c.ConfigurationRepository] [instance] Node started, try to initialize it. Wait for at least yellow cluster state....
[2023-09-28T07:14:56,672][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'config' with /usr/share/opensearch/config/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-09-28T07:14:56,716][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:56,720][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/QqykQa1uR8ORWPcsaOT38A] create_mapping
[2023-09-28T07:14:56,727][INFO ][o.o.s.i.DetectorIndexManagementService] [instance] No Old Finding Indices to delete
[2023-09-28T07:14:56,727][INFO ][o.o.s.i.DetectorIndexManagementService] [instance] No Old Alert Indices to delete
[2023-09-28T07:14:56,749][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,767][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'config' and version 2 is updated in .opendistro_security index.
[2023-09-28T07:14:56,767][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'roles' with /usr/share/opensearch/config/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-09-28T07:14:56,780][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:56,784][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/QqykQa1uR8ORWPcsaOT38A] update_mapping [_doc]
[2023-09-28T07:14:56,812][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,830][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'roles' and version 2 is updated in .opendistro_security index.
[2023-09-28T07:14:56,830][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'rolesmapping' with /usr/share/opensearch/config/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-09-28T07:14:56,841][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:56,847][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/QqykQa1uR8ORWPcsaOT38A] update_mapping [_doc]
[2023-09-28T07:14:56,872][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,886][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'rolesmapping' and version 2 is updated in .opendistro_security index.
[2023-09-28T07:14:56,886][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'internalusers' with /usr/share/opensearch/config/opensearch-security/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-09-28T07:14:56,899][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:56,904][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/QqykQa1uR8ORWPcsaOT38A] update_mapping [_doc]
[2023-09-28T07:14:56,933][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,947][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'internalusers' and version 2 is updated in .opendistro_security index.
[2023-09-28T07:14:56,947][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'actiongroups' with /usr/share/opensearch/config/opensearch-security/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-09-28T07:14:56,953][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:56,957][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/QqykQa1uR8ORWPcsaOT38A] update_mapping [_doc]
[2023-09-28T07:14:56,985][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:56,998][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'actiongroups' and version 2 is updated in .opendistro_security index.
[2023-09-28T07:14:56,999][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'tenants' with /usr/share/opensearch/config/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-09-28T07:14:57,005][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:57,009][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/QqykQa1uR8ORWPcsaOT38A] update_mapping [_doc]
[2023-09-28T07:14:57,034][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:57,048][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'tenants' and version 2 is updated in .opendistro_security index.
[2023-09-28T07:14:57,048][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'nodesdn' with /usr/share/opensearch/config/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2023-09-28T07:14:57,053][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:57,058][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/QqykQa1uR8ORWPcsaOT38A] update_mapping [_doc]
[2023-09-28T07:14:57,084][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:57,097][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'nodesdn' and version 2 is updated in .opendistro_security index.
[2023-09-28T07:14:57,098][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'whitelist' with /usr/share/opensearch/config/opensearch-security/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2023-09-28T07:14:57,103][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:57,108][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/QqykQa1uR8ORWPcsaOT38A] update_mapping [_doc]
[2023-09-28T07:14:57,131][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:57,145][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'whitelist' and version 2 is updated in .opendistro_security index.
[2023-09-28T07:14:57,146][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'allowlist' with /usr/share/opensearch/config/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2023-09-28T07:14:57,150][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:57,154][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/QqykQa1uR8ORWPcsaOT38A] update_mapping [_doc]
[2023-09-28T07:14:57,178][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:57,191][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'allowlist' and version 2 is updated in .opendistro_security index.
[2023-09-28T07:14:57,192][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'audit' with /usr/share/opensearch/config/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-09-28T07:14:57,215][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/QqykQa1uR8ORWPcsaOT38A]
[2023-09-28T07:14:57,221][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/QqykQa1uR8ORWPcsaOT38A] update_mapping [_doc]
[2023-09-28T07:14:57,245][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-09-28T07:14:57,258][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'audit' and version 2 is updated in .opendistro_security index.
[2023-09-28T07:14:57,380][INFO ][stdout                   ] [instance] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
[2023-09-28T07:14:57,380][INFO ][stdout                   ] [instance] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
[2023-09-28T07:14:57,381][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing on REST API is enabled.
[2023-09-28T07:14:57,381][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2023-09-28T07:14:57,381][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing on Transport API is enabled.
[2023-09-28T07:14:57,382][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2023-09-28T07:14:57,382][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing of request body is enabled.
[2023-09-28T07:14:57,382][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Bulk requests resolution is disabled during request auditing.
[2023-09-28T07:14:57,382][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Index resolution is enabled during request auditing.
[2023-09-28T07:14:57,382][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Sensitive headers auditing is enabled.
[2023-09-28T07:14:57,382][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing requests from kibanaserver users is disabled.
[2023-09-28T07:14:57,386][WARN ][o.o.s.a.r.AuditMessageRouter] [instance] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2023-09-28T07:14:57,386][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing of external configuration is disabled.
[2023-09-28T07:14:57,386][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing of internal configuration is enabled.
[2023-09-28T07:14:57,386][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing only metadata information for read request is enabled.
[2023-09-28T07:14:57,386][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing will watch {} for read requests.
[2023-09-28T07:14:57,386][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing read operation requests from kibanaserver users is disabled.
[2023-09-28T07:14:57,386][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing only metadata information for write request is enabled.
[2023-09-28T07:14:57,386][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing diffs for write requests is disabled.
[2023-09-28T07:14:57,386][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing write operation requests from kibanaserver users is disabled.
[2023-09-28T07:14:57,386][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing will watch <NONE> for write requests.
[2023-09-28T07:14:57,386][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] .opendistro_security is used as internal security index.
[2023-09-28T07:14:57,386][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Internal index used for posting audit logs is null
[2023-09-28T07:14:57,387][INFO ][o.o.s.c.ConfigurationRepository] [instance] Hot-reloading of audit configuration is enabled
[2023-09-28T07:14:57,387][INFO ][o.o.s.c.ConfigurationRepository] [instance] Node 'instance' initialized
[2023-09-28T07:14:58,955][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [instance] this request accesses system indices: [.opendistro_security], but in a future major version, direct access to system indices will be prevented by default

SUCCESS: Detached and healthy: instance on docker network: search-rest-test
SUCCESS: Running on: https://localhost:9200

SUCCESS run-tests
[cargo-make] INFO - Execute Command: "cargo" "test" "--no-fail-fast" "--workspace"
    Finished test [unoptimized + debuginfo] target(s) in 0.10s
     Running unittests src/lib.rs (target/debug/deps/api_generator-f7f3475611d11ba2)

running 5 tests
test generator::tests::combine_deprecations ... ok
test generator::tests::stability_ordering ... ok
test generator::output::test::nominal_merge ... ok
test generator::code_gen::url::enum_builder::tests::generate_parts_enum_from_endpoint ... ok
test generator::output::test::unbalanced_sections ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/bin/run.rs (target/debug/deps/run-e39983ec1effae10)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/opensearch-bad2c56c7efd860a)

running 23 tests
test http::request::tests::bodies_into_ndbody_writes_to_bytes ... ok
test http::request::tests::bytes_body_returns_usable_buf ... ok
test http::request::tests::bytes_body_writes_to_bytes_mut ... ok
test http::request::tests::bytes_slice_body_writes_to_bytes_mut ... ok
test http::request::tests::serialize_into_jsonbody_writes_to_bytes ... ok
test http::request::tests::string_body_writes_to_bytes_mut ... ok
test http::request::tests::string_slice_body_writes_to_bytes_mut ... ok
test http::request::tests::vec_body_writes_to_bytes_mut ... ok
test http::response::tests::deserialize_error_string ... ok
test http::response::tests::deserialize_index_not_found_exception ... ok
test http::transport::tests::connection_url_with_no_path_and_trailing_slash ... ok
test http::response::tests::deserialize_illegal_argument_exception ... ok
test http::transport::tests::connection_url_with_no_trailing_slash ... ok
test http::transport::tests::connection_url_with_no_path_and_no_trailing_slash ... ok
test http::transport::tests::connection_url_with_trailing_slash ... ok
test params::tests::serialize_slices_auto ... ok
test params::tests::serialize_slices_count ... ok
test root::bulk::tests::serialize_bulk_operations_with_different_types_writes_to_bytes ... ok
test root::bulk::tests::serialize_bulk_operations_with_same_type_writes_to_bytes ... ok
test tests::build_search_on_all_indices ... ok
test tests::percent_encode_characters ... ok
test tests::build_search_on_selected_indices ... ok
test http::transport::tests::invalid_pkcs12_cert_credentials ... ok

test result: ok. 23 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/auth.rs (target/debug/deps/auth-82da703a1eee1710)

running 3 tests
test bearer_header ... ok
test api_key_header ... ok
test basic_auth_header ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.90s

     Running tests/aws_auth.rs (target/debug/deps/aws_auth-60c2e2afabd72a70)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/cert.rs (target/debug/deps/cert-08b46affb57dbdf9)

running 8 tests
test full_certificate_ca_validation ... ok
test certificate_certificate_validation ... ok
test full_certificate_validation ... ok
test fail_certificate_certificate_validation ... ok
test full_certificate_ca_chain_validation ... ok
test default_certificate_validation ... ok
test certificate_certificate_ca_validation ... ok
test none_certificate_validation ... ok

test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.13s

     Running tests/client.rs (target/debug/deps/client-c150842d904337ae)

running 17 tests
test default_user_agent_content_type_accept_headers ... ok
test override_default_header ... ok
test cat_health_format_json ... ok
test search_with_body ... ok
test deprecation_warning_headers ... ok
test cat_health_header_json ... ok
test read_response_as_bytes ... ok
test cat_health_text ... ok
test clone_search_with_body ... FAILED
test byte_slice_body ... ok
test default_header ... ok
test serialize_querystring ... ok
test x_opaque_id_header ... ok
test uses_global_request_timeout ... ok
test search_with_no_body ... ok
test call_request_timeout_supersedes_global_timeout ... ok
test uses_call_request_timeout ... ok

failures:

---- clone_search_with_body stdout ----
thread 'clone_search_with_body' panicked at 'assertion failed: `(left == right)`
  left: `0`,
 right: `1`', opensearch/tests/client.rs:444:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

failures:
    clone_search_with_body

test result: FAILED. 16 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.58s

error: test failed, to rerun pass `-p opensearch --test client`
     Running tests/error.rs (target/debug/deps/error-132d74f411efdcce)

running 2 tests
test bad_request_returns_response ... ok
test deserialize_exception ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.64s

     Running unittests src/main.rs (target/debug/deps/yaml_test_runner-fd1b3f71059b6b29)

running 1 test
test rusty_json::test::from_set_value_correctly_interpolates_strings ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests api_generator

running 1 test
test src/generator/output.rs - generator::output::merge_file (line 90) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.31s

   Doc-tests opensearch

running 23 tests
test src/lib.rs - (line 115) - compile ... ok
test src/cat.rs - cat (line 103) - compile ... ok
test src/lib.rs - (line 103) - compile ... ok
test src/cat.rs - cat (line 60) - compile ... ok
test src/indices.rs - indices::Indices<'a>::put_mapping (line 9221) - compile ... ok
test src/lib.rs - (line 123) - compile ... ok
test src/cert.rs - cert::CertificateValidation (line 93) - compile ... ok
test src/cat.rs - cat (line 81) - compile ... ok
test src/cert.rs - cert::CertificateValidation (line 59) - compile ... ok
test src/lib.rs - (line 139) - compile ... ok
test src/cat.rs - cat (line 38) - compile ... ok
test src/indices.rs - indices::Indices<'a>::create (line 9067) - compile ... ok
test src/lib.rs - (line 162) - compile ... ok
test src/lib.rs - (line 217) - compile ... ok
test src/lib.rs - (line 191) - compile ... ok
test src/cert.rs - cert::CertificateValidation (line 125) - compile ... ok
test src/lib.rs - (line 93) - compile ... ok
test src/lib.rs - (line 307) - compile ... ok
test src/lib.rs - (line 260) - compile ... ok
test src/lib.rs - (line 342) - compile ... ok
test src/root/bulk.rs - root::bulk::BulkOperation (line 116) - compile ... ok
test src/root/bulk.rs - root::bulk::BulkOperations (line 582) - compile ... ok
test src/root/mod.rs - root::OpenSearch::scroll (line 9263) - compile ... ok

test result: ok. 23 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.38s

error: 1 target failed:
    `-p opensearch --test client`
[cargo-make] ERROR - Error while executing command, exit code: 101
[cargo-make] WARN - Build Failed.

Xtansia commented 11 months ago

@AbhinavGarg90 That is interesting that that one test is flakey for you, I'm not sure I've seen it fail before.

I've run through the setup from scratch to try to determine missing steps and what needs noting in the guide, Pre-reqs on Ubuntu:

install build-essential before cargo make
install pkg-config
install libssl-dev Will check MacOS and Windows/WSL at some point too.

Need to change advice regarding using cargo test to instead prefer cargo make test.

Am making a PR to fix running cargo make generate-certs.

AbhinavGarg90 commented 11 months ago

@AbhinavGarg90 That is interesting that that one test is flakey for you, I'm not sure I've seen it fail before.

I've run through the setup from scratch to try to determine missing steps and what needs noting in the guide, Pre-reqs on Ubuntu:

install build-essential before cargo make

install pkg-config

install libssl-dev Will check MacOS and Windows/WSL at some point too.

Need to change advice regarding using cargo test to instead prefer cargo make test.

Am making a PR to fix running cargo make generate-certs.

Are there any reccomendations on how to set it up on WSL for the moment?

Would it be ok to make pull requests/ changes without being able to run all the tests?

Xtansia commented 11 months ago

Assuming you use the Ubuntu variant of WSL2, then it should be essentially the same as setting up on Ubuntu normally. Just make sure you set up Docker Desktop for Windows with the WSL2 integration enabled if you plan on running tests locally.

Yes it's 100% okay to make PRs and contribute while still figuring out getting tests to work, we have CI tests for a reason so that we can catch issues on PRs. Local testing is more so important for faster-feedback while developing & iterating rather than waiting on the larger amount of tests that run on PRs.

AbhinavGarg90 commented 11 months ago

Assuming you use the Ubuntu variant of WSL2, then it should be essentially the same as setting up on Ubuntu normally. Just make sure you set up Docker Desktop for Windows with the WSL2 integration enabled if you plan on running tests locally.

Yes it's 100% okay to make PRs and contribute while still figuring out getting tests to work, we have CI tests for a reason so that we can catch issues on PRs. Local testing is more so important for faster-feedback while developing & iterating rather than waiting on the larger amount of tests that run on PRs.

Unfortunately I seem to be having errors that should not be occurring when trying to setup on WSL2. I tried following the exact steps you mentioned, but received test clone_search_with_body ... FAILED.

Would you have any idea why this would fail?

F:~/osci/opensearch-rs$ cargo make test
[cargo-make] INFO - cargo make 0.37.2
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: test
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: legacy-migration
[cargo-make] INFO - Skipping Task: set-coverage-vars 
[cargo-make] INFO - Skipping Task: create-test-results-dir 
[cargo-make] INFO - Execute Command: "./.ci/run-opensearch.sh"
INFO: Running opensearch latest
INFO: Creating network search-rest-test if it does not exist already 
INFO: Take down node if called twice with the same arguments (DETACH=true) or on seperate terminals 
INFO: Removing volume instance-rest-test-data
INFO: Removing volume instance-rest-test-data
instance-rest-test-data
0: 9200 https://localhost:9200 
INFO: building opensearch container
[+] Building 0.1s (16/16) FINISHED                                                                                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 882B                                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/opensearchproject/opensearch:latest                                                                                                                                                       0.0s
 => [internal] load build context                                                                                                                                                                                                    0.0s
 => => transferring context: 131B                                                                                                                                                                                                    0.0s
 => [ 1/11] FROM docker.io/opensearchproject/opensearch:latest                                                                                                                                                                       0.0s
 => CACHED [ 2/11] WORKDIR /usr/share/opensearch                                                                                                                                                                                     0.0s
 => CACHED [ 3/11] COPY --chown=opensearch:opensearch opensearch.yml ./config/                                                                                                                                                       0.0s
 => CACHED [ 4/11] RUN if [[ -f "./opensearch-onetime-setup.sh" ]]; then ./opensearch-onetime-setup.sh ; fi                                                                                                                          0.0s
 => CACHED [ 5/11] COPY --chown=opensearch:opensearch *.pem ./config/                                                                                                                                                                0.0s
 => CACHED [ 6/11] RUN sed -i'' 's/plugins.security.audit.type:.*/plugins.security.audit.type: log4j/' ./config/opensearch.yml                                                                                                       0.0s
 => CACHED [ 7/11] RUN echo "appender.header_warning.type = HeaderWarningAppender" >> ./config/log4j2.properties                                                                                                                     0.0s
 => CACHED [ 8/11] RUN echo "appender.header_warning.name = header_warning" >> ./config/log4j2.properties                                                                                                                            0.0s
 => CACHED [ 9/11] RUN echo "logger.deprecation.name = org.opensearch.deprecation" >> ./config/log4j2.properties                                                                                                                     0.0s
 => CACHED [10/11] RUN echo "logger.deprecation.level = deprecation" >> ./config/log4j2.properties                                                                                                                                   0.0s
 => CACHED [11/11] RUN echo "logger.deprecation.appenderRef.header_warning.ref = header_warning" >> ./config/log4j2.properties                                                                                                       0.0s
 => exporting to image                                                                                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                                                                              0.0s
 => => writing image sha256:9abe0b0bb8b8a2abac9bad0a35a88cb5acacb0ece3c4ef5417112a4951d8f4e3                                                                                                                                         0.0s
 => => naming to docker.io/library/opensearch-secure-true                                                                                                                                                                            0.0s
INFO: Starting container instance 
+ [[ true == \t\r\u\e ]]
+ healthcmd='curl -vvv -s --cacert /usr/share/opensearch/config/root-ca.pem -u admin:admin --fail https://localhost:9200/_cluster/health || exit 1'
++ echo curl -vvv -s --cacert /usr/share/opensearch/config/root-ca.pem -u admin:admin --fail https://localhost:9200/_cluster/health '||' exit 1
+ docker run --name instance --network search-rest-test --env 'OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g' --env node.name=instance --volume instance-rest-test-data:/usr/share/opensearch/data0 --publish 9200:9200 --ulimit nofile=65536:65536 --ulimit memlock=-1:-1 --detach=true '--health-cmd=curl -vvv -s --cacert /usr/share/opensearch/config/root-ca.pem -u admin:admin --fail https://localhost:9200/_cluster/health || exit 1' --health-interval=2s --health-retries=20 --health-timeout=2s --rm -d opensearch-secure-true
622ace4ee096c9018dd9a24732b6830b0f614e11b0e01bd200e9227df62e2899
+ set +x

INFO: waiting for node instance to be up

* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to localhost port 9200 after 2 ms: Couldn't connect to server
* Closing connection
* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to localhost port 9200 after 0 ms: Couldn't connect to server
* Closing connection
* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to localhost port 9200 after 0 ms: Couldn't connect to server
* Closing connection

INFO: waiting for node instance to be up

* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to localhost port 9200 after 0 ms: Couldn't connect to server
* Closing connection
* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to localhost port 9200 after 0 ms: Couldn't connect to server
* Closing connection
* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to localhost port 9200 after 0 ms: Couldn't connect to server
* Closing connection
* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to localhost port 9200 after 0 ms: Couldn't connect to server
* Closing connection
* processing: https://localhost:9200/_cluster/health
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* Connected to localhost (127.0.0.1) port 9200
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /usr/share/opensearch/config/root-ca.pem
*  CApath: none
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [32 bytes data]
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
{ [237 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1300 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
} [8 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: DC=de; L=test; O=node; OU=node; CN=node-0.example.com
*  start date: Oct  3 21:29:13 2023 GMT
*  expire date: Sep  9 21:29:13 2123 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
*  SSL certificate verify ok.
* using HTTP/1.x
* Server auth using Basic with user 'admin'
} [5 bytes data]
> GET /_cluster/health HTTP/1.1
> Host: localhost:9200
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/8.2.1
> Accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [1561 bytes data]
< HTTP/1.1 503 Service Unavailable
< content-type: text/plain; charset=UTF-8
< content-length: 36
* The requested URL returned error: 503
* Closing connection
} [5 bytes data]
* TLSv1.3 (OUT), TLS alert, close notify (256):
} [2 bytes data]

INFO: waiting for node instance to be up
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on Amazon Linux release 2023 (Amazon Linux)
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.10.0
Detected OpenSearch Security Version: 2.10.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.10.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.10.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2023-10-03T21:53:17,209][INFO ][o.o.n.Node               ] [instance] version[2.10.0], pid[31], build[tar/eee49cb340edc6c4d489bcd9324dda571fc8dc03/2023-09-20T23:54:29.889267151Z], OS[Linux/5.15.90.1-microsoft-standard-WSL2/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.8/17.0.8+7]
[2023-10-03T21:53:17,211][INFO ][o.o.n.Node               ] [instance] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2023-10-03T21:53:17,211][INFO ][o.o.n.Node               ] [instance] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-13229438871382891455, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2023-10-03T21:53:18,571][INFO ][o.o.s.s.t.SSLConfig      ] [instance] SSL dual mode is disabled
[2023-10-03T21:53:18,571][INFO ][o.o.s.OpenSearchSecurityPlugin] [instance] OpenSearch Config path is /usr/share/opensearch/config
[2023-10-03T21:53:18,925][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] JVM supports TLSv1.3
[2023-10-03T21:53:18,927][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2023-10-03T21:53:19,507][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] TLS Transport Client Provider : JDK
[2023-10-03T21:53:19,507][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] TLS Transport Server Provider : JDK
[2023-10-03T21:53:19,507][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] TLS HTTP Provider             : JDK
[2023-10-03T21:53:19,507][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2023-10-03T21:53:19,507][INFO ][o.o.s.s.DefaultSecurityKeyStore] [instance] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
[2023-10-03T21:53:19,553][INFO ][o.o.s.OpenSearchSecurityPlugin] [instance] Clustername: search-rest-test
[2023-10-03T21:53:19,562][WARN ][o.o.s.OpenSearchSecurityPlugin] [instance] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2023-10-03T21:53:19,562][WARN ][o.o.s.OpenSearchSecurityPlugin] [instance] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2023-10-03T21:53:19,562][WARN ][o.o.s.OpenSearchSecurityPlugin] [instance] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2023-10-03T21:53:19,563][WARN ][o.o.s.OpenSearchSecurityPlugin] [instance] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[2023-10-03T21:53:20,713][INFO ][o.o.p.c.c.PluginSettings ] [instance] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2023-10-03T21:53:21,393][INFO ][o.o.i.r.ReindexPlugin    ] [instance] ReindexPlugin reloadSPI called
[2023-10-03T21:53:21,394][INFO ][o.o.i.r.ReindexPlugin    ] [instance] Unable to find any implementation for RemoteReindexExtension
[2023-10-03T21:53:21,437][INFO ][o.o.j.JobSchedulerPlugin ] [instance] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2023-10-03T21:53:21,474][INFO ][o.o.j.JobSchedulerPlugin ] [instance] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-10-03T21:53:21,476][INFO ][o.o.j.JobSchedulerPlugin ] [instance] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-10-03T21:53:21,477][INFO ][o.o.j.JobSchedulerPlugin ] [instance] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2023-10-03T21:53:21,483][INFO ][o.o.p.PluginsService     ] [instance] loaded module [aggs-matrix-stats]
[2023-10-03T21:53:21,483][INFO ][o.o.p.PluginsService     ] [instance] loaded module [analysis-common]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [geo]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [ingest-common]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [ingest-geoip]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [ingest-user-agent]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [lang-expression]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [lang-mustache]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [lang-painless]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [mapper-extras]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [opensearch-dashboards]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [parent-join]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [percolator]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [rank-eval]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [reindex]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [repository-url]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [search-pipeline-common]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [systemd]
[2023-10-03T21:53:21,484][INFO ][o.o.p.PluginsService     ] [instance] loaded module [transport-netty4]
[2023-10-03T21:53:21,485][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-alerting]
[2023-10-03T21:53:21,485][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-anomaly-detection]
[2023-10-03T21:53:21,485][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-asynchronous-search]
[2023-10-03T21:53:21,485][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-cross-cluster-replication]
[2023-10-03T21:53:21,485][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-custom-codecs]
[2023-10-03T21:53:21,485][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-geospatial]
[2023-10-03T21:53:21,485][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-index-management]
[2023-10-03T21:53:21,485][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-job-scheduler]
[2023-10-03T21:53:21,485][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-knn]
[2023-10-03T21:53:21,486][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-ml]
[2023-10-03T21:53:21,486][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-neural-search]
[2023-10-03T21:53:21,486][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-notifications]
[2023-10-03T21:53:21,486][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-notifications-core]
[2023-10-03T21:53:21,486][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-observability]
[2023-10-03T21:53:21,486][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-performance-analyzer]
[2023-10-03T21:53:21,486][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-reports-scheduler]
[2023-10-03T21:53:21,486][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-security]
[2023-10-03T21:53:21,486][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-security-analytics]
[2023-10-03T21:53:21,486][INFO ][o.o.p.PluginsService     ] [instance] loaded plugin [opensearch-sql]
[2023-10-03T21:53:21,506][INFO ][o.o.s.OpenSearchSecurityPlugin] [instance] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2023-10-03T21:53:21,548][INFO ][o.o.e.ExtensionsManager  ] [instance] ExtensionsManager initialized
[2023-10-03T21:53:21,558][DEPRECATION][o.o.d.c.s.Settings       ] [instance] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2023-10-03T21:53:21,572][INFO ][o.o.e.NodeEnvironment    ] [instance] using [1] data paths, mounts [[/ (overlay)]], net usable_space [932.3gb], net total_space [1006.8gb], types [overlay]
[2023-10-03T21:53:21,572][INFO ][o.o.e.NodeEnvironment    ] [instance] heap size [1gb], compressed ordinary object pointers [true]
[2023-10-03T21:53:21,612][INFO ][o.o.n.Node               ] [instance] node name [instance], node ID [Q8tSGVgyQ9KgU7P4zVXjvA], cluster name [search-rest-test], roles [ingest, remote_cluster_client, data, cluster_manager]
[2023-10-03T21:53:23,998][DEPRECATION][o.o.d.c.s.Settings       ] [instance] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2023-10-03T21:53:24,001][DEPRECATION][o.o.d.c.s.Settings       ] [instance] [cluster.initial_master_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2023-10-03T21:53:24,863][INFO ][o.o.n.p.NeuralSearch     ] [instance] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
[2023-10-03T21:53:25,092][WARN ][o.o.s.c.Salt             ] [instance] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2023-10-03T21:53:25,145][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Message routing enabled: true
[2023-10-03T21:53:25,189][INFO ][o.o.s.f.SecurityFilter   ] [instance] <NONE> indices are made immutable.
[2023-10-03T21:53:25,593][INFO ][o.o.a.b.ADCircuitBreakerService] [instance] Registered memory breaker.
[2023-10-03T21:53:26,119][INFO ][o.o.m.b.MLCircuitBreakerService] [instance] Registered ML memory breaker.
[2023-10-03T21:53:26,120][INFO ][o.o.m.b.MLCircuitBreakerService] [instance] Registered ML disk breaker.
[2023-10-03T21:53:26,120][INFO ][o.o.m.b.MLCircuitBreakerService] [instance] Registered ML native memory breaker.
[2023-10-03T21:53:26,240][INFO ][o.r.Reflections          ] [instance] Reflections took 47 ms to scan 1 urls, producing 17 keys and 43 values 
[2023-10-03T21:53:26,327][WARN ][o.o.s.p.SQLPlugin        ] [instance] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2023-10-03T21:53:26,987][INFO ][o.o.t.NettyAllocator     ] [instance] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2023-10-03T21:53:27,096][INFO ][o.o.d.DiscoveryModule    ] [instance] using discovery type [zen] and seed hosts providers [settings]
[2023-10-03T21:53:27,514][WARN ][o.o.g.DanglingIndicesState] [instance] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-10-03T21:53:28,048][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [instance] PerformanceAnalyzer Enabled: false
[2023-10-03T21:53:28,083][INFO ][o.o.n.Node               ] [instance] initialized
[2023-10-03T21:53:28,083][INFO ][o.o.n.Node               ] [instance] starting ...
[2023-10-03T21:53:28,119][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [windows_logtype.json] log type
[2023-10-03T21:53:28,120][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [vpcflow_logtype.json] log type
[2023-10-03T21:53:28,120][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [test_windows_logtype.json] log type
[2023-10-03T21:53:28,121][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [s3_logtype.json] log type
[2023-10-03T21:53:28,121][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_web_logtype.json] log type
[2023-10-03T21:53:28,122][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_proxy_logtype.json] log type
[2023-10-03T21:53:28,122][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_macos_logtype.json] log type
[2023-10-03T21:53:28,123][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_compliance_logtype.json] log type
[2023-10-03T21:53:28,123][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_cloud_logtype.json] log type
[2023-10-03T21:53:28,124][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_apt_logtype.json] log type
[2023-10-03T21:53:28,124][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [others_application_logtype.json] log type
[2023-10-03T21:53:28,125][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [okta_logtype.json] log type
[2023-10-03T21:53:28,126][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [network_logtype.json] log type
[2023-10-03T21:53:28,126][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [netflow_logtype.json] log type
[2023-10-03T21:53:28,127][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [m365_logtype.json] log type
[2023-10-03T21:53:28,128][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [linux_logtype.json] log type
[2023-10-03T21:53:28,128][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [gworkspace_logtype.json] log type
[2023-10-03T21:53:28,129][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [github_logtype.json] log type
[2023-10-03T21:53:28,129][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [dns_logtype.json] log type
[2023-10-03T21:53:28,130][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [cloudtrail_logtype.json] log type
[2023-10-03T21:53:28,131][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [azure_logtype.json] log type
[2023-10-03T21:53:28,131][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [apache_access_logtype.json] log type
[2023-10-03T21:53:28,132][INFO ][o.o.s.l.BuiltinLogTypeLoader] [instance] Loaded [ad_ldap_logtype.json] log type
[2023-10-03T21:53:28,236][INFO ][o.o.t.TransportService   ] [instance] publish_address {172.23.0.2:9300}, bound_addresses {0.0.0.0:9300}
[2023-10-03T21:53:28,238][INFO ][o.o.t.TransportService   ] [instance] Remote clusters initialized successfully.
[2023-10-03T21:53:28,384][INFO ][o.o.b.BootstrapChecks    ] [instance] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-10-03T21:53:28,395][INFO ][o.o.c.c.Coordinator      ] [instance] setting initial configuration to VotingConfiguration{Q8tSGVgyQ9KgU7P4zVXjvA}
[2023-10-03T21:53:28,525][INFO ][o.o.c.s.MasterService    ] [instance] elected-as-cluster-manager ([1] nodes joined)[{instance}{Q8tSGVgyQ9KgU7P4zVXjvA}{U-7huJGVRKqVBDdejeNDUg}{172.23.0.2}{172.23.0.2:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{instance}{Q8tSGVgyQ9KgU7P4zVXjvA}{U-7huJGVRKqVBDdejeNDUg}{172.23.0.2}{172.23.0.2:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]}
[2023-10-03T21:53:28,557][INFO ][o.o.c.c.CoordinationState] [instance] cluster UUID set to [XluiDt9hSKKlYyNh2RJbKw]
[2023-10-03T21:53:28,581][INFO ][o.o.c.s.ClusterApplierService] [instance] cluster-manager node changed {previous [], current [{instance}{Q8tSGVgyQ9KgU7P4zVXjvA}{U-7huJGVRKqVBDdejeNDUg}{172.23.0.2}{172.23.0.2:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2023-10-03T21:53:28,587][INFO ][o.o.a.c.ADClusterEventListener] [instance] Cluster is not recovered yet.
[2023-10-03T21:53:28,593][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:28,612][INFO ][o.o.i.i.ManagedIndexCoordinator] [instance] Cache cluster manager node onClusterManager time: 1696370008612
[2023-10-03T21:53:28,621][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [instance] Config override setting update called with empty string. Ignoring.
[2023-10-03T21:53:28,626][INFO ][o.o.d.PeerFinder         ] [instance] setting findPeersInterval to [1s] as node commission status = [true] for local node [{instance}{Q8tSGVgyQ9KgU7P4zVXjvA}{U-7huJGVRKqVBDdejeNDUg}{172.23.0.2}{172.23.0.2:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]
[2023-10-03T21:53:28,641][INFO ][o.o.h.AbstractHttpServerTransport] [instance] publish_address {172.23.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2023-10-03T21:53:28,641][INFO ][o.o.n.Node               ] [instance] started
[2023-10-03T21:53:28,641][INFO ][o.o.s.OpenSearchSecurityPlugin] [instance] Node started
[2023-10-03T21:53:28,642][INFO ][o.o.s.c.ConfigurationRepository] [instance] Will attempt to create index .opendistro_security and default configs if they are absent
[2023-10-03T21:53:28,642][INFO ][o.o.s.c.ConfigurationRepository] [instance] Background init thread started. Install default config?: true
[2023-10-03T21:53:28,643][INFO ][o.o.s.OpenSearchSecurityPlugin] [instance] 0 OpenSearch Security modules loaded so far: []
[2023-10-03T21:53:28,643][INFO ][o.o.s.c.ConfigurationRepository] [instance] Wait for cluster to be available ...
[2023-10-03T21:53:28,646][INFO ][o.o.a.c.HashRing         ] [instance] Node added: [Q8tSGVgyQ9KgU7P4zVXjvA]
[2023-10-03T21:53:28,647][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:28,648][INFO ][o.o.a.c.HashRing         ] [instance] Add data node to AD version hash ring: Q8tSGVgyQ9KgU7P4zVXjvA
[2023-10-03T21:53:28,650][INFO ][o.o.a.c.HashRing         ] [instance] All nodes with known AD version: {Q8tSGVgyQ9KgU7P4zVXjvA=ADNodeInfo{version=2.10.0, isEligibleDataNode=true}}
[2023-10-03T21:53:28,650][INFO ][o.o.a.c.HashRing         ] [instance] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2023-10-03T21:53:28,650][INFO ][o.o.a.c.HashRing         ] [instance] Build AD version hash ring successfully
[2023-10-03T21:53:28,651][INFO ][o.o.a.c.ADDataMigrator   ] [instance] Start migrating AD data
[2023-10-03T21:53:28,651][INFO ][o.o.a.c.ADDataMigrator   ] [instance] AD job index doesn't exist, no need to migrate
[2023-10-03T21:53:28,651][INFO ][o.o.a.c.ADClusterEventListener] [instance] Init AD version hash ring successfully
[2023-10-03T21:53:28,665][INFO ][o.o.g.GatewayService     ] [instance] recovered [0] indices into cluster_state
[2023-10-03T21:53:28,684][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/noDbrBpxSt6e3kgwAeGFtQ]
[2023-10-03T21:53:28,799][INFO ][o.o.c.m.MetadataCreateIndexService] [instance] [.opensearch-sap-log-types-config] creating index, cause [auto(sap-logtype api)], templates [], shards [1]/[1]
[2023-10-03T21:53:28,806][INFO ][o.o.c.r.a.AllocationService] [instance] updating number_of_replicas to [0] for indices [.opensearch-sap-log-types-config]
[2023-10-03T21:53:28,849][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/noDbrBpxSt6e3kgwAeGFtQ]
[2023-10-03T21:53:28,895][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:28,899][DEPRECATION][o.o.d.c.m.MetadataCreateIndexService] [instance] index name [.opensearch-observability] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices
[2023-10-03T21:53:28,900][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-observability/iMGfXfySR3Sk7_bQ7b2zLQ]
[2023-10-03T21:53:28,914][INFO ][o.o.c.m.MetadataCreateIndexService] [instance] [.opensearch-observability] creating index, cause [api], templates [], shards [1]/[0]
[2023-10-03T21:53:28,949][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-observability/iMGfXfySR3Sk7_bQ7b2zLQ]
[2023-10-03T21:53:28,973][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:29,044][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:29,046][INFO ][o.o.s.l.LogTypeService   ] [instance] Loading builtin types!
[2023-10-03T21:53:29,048][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [418] fieldMappingDocs from logTypes: 23
[2023-10-03T21:53:29,075][INFO ][o.o.c.r.a.AllocationService] [instance] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opensearch-observability][0]]]).
[2023-10-03T21:53:29,102][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:29,103][INFO ][o.o.o.i.ObservabilityIndex] [instance] observability:Index .opensearch-observability creation Acknowledged
[2023-10-03T21:53:29,104][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [instance] observability:createMappingTemplate ss4o_metrics_template API called
[2023-10-03T21:53:29,112][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [418] fieldMappingDocs
[2023-10-03T21:53:29,117][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[validate-template-cgbgrrr1rly9ilemqcu0og/NWHK3rIjRvW1VIu0du8R2A]
[2023-10-03T21:53:29,146][INFO ][o.o.c.m.MetadataIndexTemplateService] [instance] adding index template [ss4o_metrics_template] for index patterns [ss4o_metrics-*-*]
[2023-10-03T21:53:29,183][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:29,185][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [instance] observability:Mapping Template ss4o_metrics_template creation Acknowledged
[2023-10-03T21:53:29,185][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [instance] observability:createMappingTemplate ss4o_traces_template API called
[2023-10-03T21:53:29,189][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/noDbrBpxSt6e3kgwAeGFtQ]
[2023-10-03T21:53:29,206][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opensearch-sap-log-types-config/noDbrBpxSt6e3kgwAeGFtQ] update_mapping [_doc]
[2023-10-03T21:53:29,254][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:29,260][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[validate-template-asi33jfirtajrti5ljlg2q/p5ERrzwyQ8WJSwf5DkP9QA]
[2023-10-03T21:53:29,281][INFO ][o.o.c.m.MetadataIndexTemplateService] [instance] adding index template [ss4o_traces_template] for index patterns [ss4o_traces-*-*]
[2023-10-03T21:53:29,318][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:29,320][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [instance] observability:Mapping Template ss4o_traces_template creation Acknowledged
[2023-10-03T21:53:29,323][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/noDbrBpxSt6e3kgwAeGFtQ]
[2023-10-03T21:53:29,370][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/noDbrBpxSt6e3kgwAeGFtQ]
[2023-10-03T21:53:29,381][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opensearch-sap-log-types-config/noDbrBpxSt6e3kgwAeGFtQ] update_mapping [_doc]
[2023-10-03T21:53:29,428][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:29,539][ERROR][o.o.s.a.BackendRegistry  ] [instance] Not yet initialized (you may need to run securityadmin)
[2023-10-03T21:53:29,624][INFO ][o.o.s.l.LogTypeService   ] [instance] Loading builtin types!
[2023-10-03T21:53:29,626][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [418] fieldMappingDocs from logTypes: 23
[2023-10-03T21:53:29,631][INFO ][o.o.s.l.LogTypeService   ] [instance] Loading builtin types!
[2023-10-03T21:53:29,632][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [418] fieldMappingDocs from logTypes: 23
[2023-10-03T21:53:29,643][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [418] fieldMappingDocs
[2023-10-03T21:53:29,644][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [418] fieldMappingDocs
[2023-10-03T21:53:29,653][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:29,658][INFO ][o.o.c.m.MetadataCreateIndexService] [instance] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1]
[2023-10-03T21:53:29,659][INFO ][o.o.c.r.a.AllocationService] [instance] updating number_of_replicas to [0] for indices [.opendistro_security]
[2023-10-03T21:53:29,708][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:29,730][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:29,786][INFO ][o.o.c.r.a.AllocationService] [instance] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]).
[2023-10-03T21:53:29,824][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:29,825][INFO ][o.o.s.c.ConfigurationRepository] [instance] Index .opendistro_security created?: true
[2023-10-03T21:53:29,825][INFO ][o.o.s.c.ConfigurationRepository] [instance] Node started, try to initialize it. Wait for at least yellow cluster state....
[2023-10-03T21:53:29,843][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'config' with /usr/share/opensearch/config/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-10-03T21:53:29,904][INFO ][o.o.s.l.LogTypeService   ] [instance] Loaded [418] field mapping docs successfully!
[2023-10-03T21:53:29,909][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:29,914][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ] create_mapping
[2023-10-03T21:53:29,937][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [22] customLogTypes
[2023-10-03T21:53:29,950][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:29,976][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'config' and version 2 is updated in .opendistro_security index.
[2023-10-03T21:53:29,976][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'roles' with /usr/share/opensearch/config/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-10-03T21:53:29,992][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:29,998][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ] update_mapping [_doc]
[2023-10-03T21:53:30,023][INFO ][o.o.s.l.LogTypeService   ] [instance] Loaded [418] field mapping docs successfully!
[2023-10-03T21:53:30,031][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:30,032][INFO ][o.o.s.l.LogTypeService   ] [instance] Indexing [22] customLogTypes
[2023-10-03T21:53:30,050][INFO ][o.o.s.l.LogTypeService   ] [instance] Loaded [418] field mapping docs successfully!
[2023-10-03T21:53:30,055][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'roles' and version 2 is updated in .opendistro_security index.
[2023-10-03T21:53:30,055][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'rolesmapping' with /usr/share/opensearch/config/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-10-03T21:53:30,075][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:30,083][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ] update_mapping [_doc]
[2023-10-03T21:53:30,085][INFO ][o.o.s.l.LogTypeService   ] [instance] Loaded [22] customLogType docs successfully!
[2023-10-03T21:53:30,088][INFO ][o.o.s.l.LogTypeService   ] [instance] Loaded [22] customLogType docs successfully!
[2023-10-03T21:53:30,088][INFO ][o.o.s.SecurityAnalyticsPlugin] [instance] LogType config index successfully created and builtin log types loaded
[2023-10-03T21:53:30,116][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:30,138][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'rolesmapping' and version 2 is updated in .opendistro_security index.
[2023-10-03T21:53:30,138][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'internalusers' with /usr/share/opensearch/config/opensearch-security/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-10-03T21:53:30,157][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:30,165][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ] update_mapping [_doc]
[2023-10-03T21:53:30,201][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:30,204][INFO ][o.o.s.i.DetectorIndexManagementService] [instance] No Old Alert Indices to delete
[2023-10-03T21:53:30,204][INFO ][o.o.s.i.DetectorIndexManagementService] [instance] No Old Finding Indices to delete
[2023-10-03T21:53:30,222][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'internalusers' and version 2 is updated in .opendistro_security index.
[2023-10-03T21:53:30,222][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'actiongroups' with /usr/share/opensearch/config/opensearch-security/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-10-03T21:53:30,232][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:30,238][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ] update_mapping [_doc]
[2023-10-03T21:53:30,270][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:30,293][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'actiongroups' and version 2 is updated in .opendistro_security index.
[2023-10-03T21:53:30,294][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'tenants' with /usr/share/opensearch/config/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-10-03T21:53:30,304][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:30,312][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ] update_mapping [_doc]
[2023-10-03T21:53:30,345][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:30,365][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'tenants' and version 2 is updated in .opendistro_security index.
[2023-10-03T21:53:30,365][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'nodesdn' with /usr/share/opensearch/config/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2023-10-03T21:53:30,372][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:30,378][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ] update_mapping [_doc]
[2023-10-03T21:53:30,408][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:30,426][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'nodesdn' and version 2 is updated in .opendistro_security index.
[2023-10-03T21:53:30,426][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'whitelist' with /usr/share/opensearch/config/opensearch-security/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2023-10-03T21:53:30,434][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:30,440][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ] update_mapping [_doc]
[2023-10-03T21:53:30,476][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:30,494][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'whitelist' and version 2 is updated in .opendistro_security index.
[2023-10-03T21:53:30,495][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'allowlist' with /usr/share/opensearch/config/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2023-10-03T21:53:30,503][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:30,510][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ] update_mapping [_doc]
[2023-10-03T21:53:30,548][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:30,573][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'allowlist' and version 2 is updated in .opendistro_security index.
[2023-10-03T21:53:30,574][INFO ][o.o.s.s.ConfigHelper     ] [instance] Will update 'audit' with /usr/share/opensearch/config/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-10-03T21:53:30,609][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ]
[2023-10-03T21:53:30,617][INFO ][o.o.c.m.MetadataMappingService] [instance] [.opendistro_security/mD4UkLcKQIqblc_zFdkaSQ] update_mapping [_doc]
[2023-10-03T21:53:30,651][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:30,673][INFO ][o.o.s.s.ConfigHelper     ] [instance] Doc with id 'audit' and version 2 is updated in .opendistro_security index.
[2023-10-03T21:53:30,888][INFO ][stdout                   ] [instance] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
[2023-10-03T21:53:30,889][INFO ][stdout                   ] [instance] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
[2023-10-03T21:53:30,889][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing on REST API is enabled.
[2023-10-03T21:53:30,890][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2023-10-03T21:53:30,891][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing on Transport API is enabled.
[2023-10-03T21:53:30,891][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2023-10-03T21:53:30,891][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing of request body is enabled.
[2023-10-03T21:53:30,891][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Bulk requests resolution is disabled during request auditing.
[2023-10-03T21:53:30,891][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Index resolution is enabled during request auditing.
[2023-10-03T21:53:30,891][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Sensitive headers auditing is enabled.
[2023-10-03T21:53:30,891][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing requests from kibanaserver users is disabled.
[2023-10-03T21:53:30,896][WARN ][o.o.s.a.r.AuditMessageRouter] [instance] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2023-10-03T21:53:30,897][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing of external configuration is disabled.
[2023-10-03T21:53:30,897][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing of internal configuration is enabled.
[2023-10-03T21:53:30,897][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing only metadata information for read request is enabled.
[2023-10-03T21:53:30,897][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing will watch {} for read requests.
[2023-10-03T21:53:30,897][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing read operation requests from kibanaserver users is disabled.
[2023-10-03T21:53:30,897][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing only metadata information for write request is enabled.
[2023-10-03T21:53:30,897][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing diffs for write requests is disabled.
[2023-10-03T21:53:30,897][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing write operation requests from kibanaserver users is disabled.
[2023-10-03T21:53:30,898][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Auditing will watch <NONE> for write requests.
[2023-10-03T21:53:30,898][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] .opendistro_security is used as internal security index.
[2023-10-03T21:53:30,898][INFO ][o.o.s.a.i.AuditLogImpl   ] [instance] Internal index used for posting audit logs is null
[2023-10-03T21:53:30,898][INFO ][o.o.s.c.ConfigurationRepository] [instance] Hot-reloading of audit configuration is enabled
[2023-10-03T21:53:30,899][INFO ][o.o.s.c.ConfigurationRepository] [instance] Node 'instance' initialized
[2023-10-03T21:53:31,952][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [instance] this request accesses system indices: [.opendistro_security], but in a future major version, direct access to system indices will be prevented by default
[2023-10-03T21:53:38,623][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.plugins-ml-config/neKucf6FTX-VM9mXUp3Qqw]
[2023-10-03T21:53:38,628][INFO ][o.o.c.m.MetadataCreateIndexService] [instance] [.plugins-ml-config] creating index, cause [api], templates [], shards [1]/[1]
[2023-10-03T21:53:38,628][INFO ][o.o.c.r.a.AllocationService] [instance] updating number_of_replicas to [0] for indices [.plugins-ml-config]
[2023-10-03T21:53:38,649][INFO ][o.o.p.PluginsService     ] [instance] PluginService:onIndexModule index:[.plugins-ml-config/neKucf6FTX-VM9mXUp3Qqw]
[2023-10-03T21:53:38,663][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:38,697][INFO ][o.o.c.r.a.AllocationService] [instance] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-ml-config][0]]]).
[2023-10-03T21:53:38,716][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [instance] Detected cluster change event for destination migration
[2023-10-03T21:53:38,717][INFO ][o.o.m.i.MLIndicesHandler ] [instance] create index:.plugins-ml-config
[2023-10-03T21:53:38,741][INFO ][o.o.m.c.MLSyncUpCron     ] [instance] ML configuration initialized successfully

SUCCESS: Detached and healthy: instance on docker network: search-rest-test
SUCCESS: Running on: https://localhost:9200

SUCCESS run-tests
[cargo-make] INFO - Execute Command: "cargo" "test" "--no-fail-fast" "--workspace"
    Finished test [unoptimized + debuginfo] target(s) in 0.40s
     Running unittests src/lib.rs (target/debug/deps/api_generator-2b8e9008c5dddf6f)

running 5 tests
test generator::tests::stability_ordering ... ok
test generator::tests::combine_deprecations ... ok
test generator::output::test::nominal_merge ... ok
test generator::output::test::unbalanced_sections ... ok
test generator::code_gen::url::enum_builder::tests::generate_parts_enum_from_endpoint ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s

     Running unittests src/bin/run.rs (target/debug/deps/run-9957c04ed84e750f)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/opensearch-33fa565355b101bb)

running 23 tests
test http::request::tests::string_slice_body_writes_to_bytes_mut ... ok
test http::request::tests::bytes_body_returns_usable_buf ... ok
test http::request::tests::bytes_slice_body_writes_to_bytes_mut ... ok
test http::request::tests::string_body_writes_to_bytes_mut ... ok
test http::request::tests::bytes_body_writes_to_bytes_mut ... ok
test http::request::tests::vec_body_writes_to_bytes_mut ... ok
test tests::build_search_on_all_indices ... ok
test tests::build_search_on_selected_indices ... ok
test params::tests::serialize_slices_auto ... ok
test params::tests::serialize_slices_count ... ok
test tests::percent_encode_characters ... ok
test http::response::tests::deserialize_error_string ... ok
test http::request::tests::serialize_into_jsonbody_writes_to_bytes ... ok
test http::request::tests::bodies_into_ndbody_writes_to_bytes ... ok
test http::response::tests::deserialize_index_not_found_exception ... ok
test root::bulk::tests::serialize_bulk_operations_with_different_types_writes_to_bytes ... ok
test root::bulk::tests::serialize_bulk_operations_with_same_type_writes_to_bytes ... ok
test http::transport::tests::connection_url_with_no_path_and_trailing_slash ... ok
test http::transport::tests::connection_url_with_trailing_slash ... ok
test http::transport::tests::connection_url_with_no_path_and_no_trailing_slash ... ok
test http::transport::tests::connection_url_with_no_trailing_slash ... ok
test http::response::tests::deserialize_illegal_argument_exception ... ok
test http::transport::tests::invalid_pkcs12_cert_credentials ... ok

test result: ok. 23 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s

     Running tests/auth.rs (target/debug/deps/auth-ce5ef59b9b424b40)

running 3 tests
test basic_auth_header ... ok
test bearer_header ... ok
test api_key_header ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.91s

     Running tests/aws_auth.rs (target/debug/deps/aws_auth-d0c7a2b58fea83da)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/cert.rs (target/debug/deps/cert-5a27819082c3c2c4)

running 8 tests
test certificate_certificate_validation ... ok
test fail_certificate_certificate_validation ... ok
test default_certificate_validation ... ok
test full_certificate_validation ... ok
test certificate_certificate_ca_validation ... ok
test full_certificate_ca_chain_validation ... ok
test none_certificate_validation ... ok
test full_certificate_ca_validation ... ok

test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.49s

     Running tests/client.rs (target/debug/deps/client-e524c653e93db373)

running 17 tests
test byte_slice_body ... ok
test cat_health_format_json ... ok
test cat_health_text ... ok
test serialize_querystring ... ok
test default_user_agent_content_type_accept_headers ... ok
test default_header ... ok
test override_default_header ... ok
test read_response_as_bytes ... ok
test cat_health_header_json ... ok
test clone_search_with_body ... FAILED
test search_with_body ... ok
test deprecation_warning_headers ... ok
test x_opaque_id_header ... ok
test uses_call_request_timeout ... ok
test call_request_timeout_supersedes_global_timeout ... ok
test search_with_no_body ... ok
test uses_global_request_timeout ... ok

failures:

---- clone_search_with_body stdout ----
thread 'clone_search_with_body' panicked at 'assertion failed: `(left == right)`
  left: `0`,
 right: `1`', opensearch/tests/client.rs:444:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

failures:
    clone_search_with_body

test result: FAILED. 16 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.83s

error: test failed, to rerun pass `-p opensearch --test client`
     Running tests/error.rs (target/debug/deps/error-d4e4b73af4d96280)

running 2 tests
test deserialize_exception ... ok
test bad_request_returns_response ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.90s

     Running unittests src/main.rs (target/debug/deps/yaml_test_runner-92a0269b27f980e3)

running 1 test
test rusty_json::test::from_set_value_correctly_interpolates_strings ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s

   Doc-tests api_generator

running 1 test
test src/generator/output.rs - generator::output::merge_file (line 90) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.65s

   Doc-tests opensearch

running 23 tests
test src/lib.rs - (line 103) - compile ... ok
test src/lib.rs - (line 139) - compile ... ok
test src/lib.rs - (line 123) - compile ... ok
test src/lib.rs - (line 115) - compile ... ok
test src/lib.rs - (line 260) - compile ... ok
test src/cert.rs - cert::CertificateValidation (line 59) - compile ... ok
test src/lib.rs - (line 191) - compile ... ok
test src/lib.rs - (line 162) - compile ... ok
test src/indices.rs - indices::Indices<'a>::put_mapping (line 9221) - compile ... ok
test src/cat.rs - cat (line 81) - compile ... ok
test src/cat.rs - cat (line 103) - compile ... ok
test src/lib.rs - (line 307) - compile ... ok
test src/cert.rs - cert::CertificateValidation (line 125) - compile ... ok
test src/cat.rs - cat (line 38) - compile ... ok
test src/cat.rs - cat (line 60) - compile ... ok
test src/lib.rs - (line 217) - compile ... ok
test src/cert.rs - cert::CertificateValidation (line 93) - compile ... ok
test src/indices.rs - indices::Indices<'a>::create (line 9067) - compile ... ok
test src/root/bulk.rs - root::bulk::BulkOperation (line 116) - compile ... ok
test src/lib.rs - (line 93) - compile ... ok
test src/lib.rs - (line 342) - compile ... ok
test src/root/bulk.rs - root::bulk::BulkOperations (line 582) - compile ... ok
test src/root/mod.rs - root::OpenSearch::scroll (line 9263) - compile ... ok

test result: ok. 23 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.06s

error: 1 target failed:
    `-p opensearch --test client`
[cargo-make] ERROR - Error while executing command, exit code: 101
[cargo-make] WARN - Build Failed.

Xtansia commented 11 months ago

I don't believe it's a setup issue, it think it's just flakiness in the test for some reason. Can you run the following command to see if it fails on its own:

RUST_BACKTRACE=full cargo make test -p opensearch --test client -- clone_search_with_body

AbhinavGarg90 commented 11 months ago

RUST_BACKTRACE=full cargo make test -p opensearch --test client -- clone_search_with_body

Just tried this. It worked without any errors. It could be the flakiness as you mentioned, but that seems a little problematic. If CI/CD is running the same tests, it may undeterministically reject working pull requests.

AbhinavGarg90 commented 10 months ago

What should we be looking to do to resolve this issue?

Xtansia commented 10 months ago

@AbhinavGarg90 The solution here is just to update the DEVELOPER_GUIDE.md to have correct and complete instructions

AbhinavGarg90 commented 10 months ago

Are we looking to make any changes regarding the flakiness of the tests? I'll start working on a proposed change for the documentation.

opensearch-project / opensearch-rs