search

wazuh / wazuh-indexer

Wazuh indexer, the Wazuh search engine
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
11 stars 17 forks source link

Generate production packages with Wazuh changes and config files #4

Closed AlexRuiz7 closed 10 months ago

AlexRuiz7 commented 1 year ago

Description

We need to generate production packages of the indexer with our changes and default configuration in it.

These packages will contain our security configuration.

Tasks

Related issues

AlexRuiz7 commented 1 year ago

The necessary tooling to build packages is located under the distribution folder, which is a Gradle subproject.

https://github.com/wazuh/wazuh-indexer/tree/4.8.0/distribution

I'll continue investigating how to use these tools.

AlexRuiz7 commented 1 year ago

Successfully run OpenSearch locally:

Details

```java ./gradlew run Welcome to Gradle 8.2.1! Here are the highlights of this release: - Kotlin DSL: new reference documentation, assignment syntax by default - Kotlin DSL is now the default with Gradle init - Improved suggestions to resolve errors in console output - Reduced sync memory consumption For more details see https://docs.gradle.org/8.2.1/release-notes.html ======================================= OpenSearch Build Hamster says Hello! Gradle Version : 8.2.1 OS Info : Linux 5.19.0-50-generic (amd64) JDK Version : 11 (Ubuntu JDK) JAVA_HOME : /usr/lib/jvm/java-11-openjdk-amd64 Random Testing Seed : B998A4CA8A75E61 In FIPS 140 mode : false ======================================= > Task :libs:opensearch-common:compileJava Note: /wazuh-indexer/libs/common/src/main/java/org/opensearch/common/util/set/Sets.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :libs:opensearch-core:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :libs:opensearch-x-content:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :server:compileJava Note: Processing Log4j annotations Note: Annotations processed Note: Processing Log4j annotations Note: No elements to process Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :modules:systemd:compileJava Note: /wazuh-indexer/modules/systemd/src/main/java/org/opensearch/systemd/SystemdPlugin.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:aggs-matrix-stats:compileJava Note: /wazuh-indexer/modules/aggs-matrix-stats/src/main/java/org/opensearch/search/aggregations/matrix/MatrixAggregationPlugin.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:parent-join:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:lang-expression:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:percolator:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:lang-mustache:compileJava Note: /wazuh-indexer/modules/lang-mustache/src/main/java/org/opensearch/script/mustache/MustacheScriptEngine.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:geo:compileJava Note: /wazuh-indexer/modules/geo/src/main/java/org/opensearch/geo/GeoModulePlugin.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :modules:transport-netty4:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:reindex:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:lang-painless:compileJava Note: /wazuh-indexer/modules/lang-painless/src/main/java/org/opensearch/painless/antlr/Walker.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :modules:analysis-common:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :run [2023-08-07T13:00:47.166255Z] [BUILD] Copying additional config files from distro [/wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/jvm.options.d, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/opensearch.prod.yml, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/jvm.prod.options, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/security, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/log4j2.properties, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/opensearch.yml, /wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT/config/jvm.options] [2023-08-07T13:00:47.167671Z] [BUILD] Creating opensearch keystore with password set to [] [2023-08-07T13:00:47.777952Z] [BUILD] Starting OpenSearch process [2023-08-07T15:00:49,094][INFO ][o.o.n.Node ] [runTask-0] version[2.10.0-SNAPSHOT], pid[63392], build[tar/0fcdd5647c613fd8e653eb68a3086dcaea314e06/2023-08-07T12:56:56.904455Z], OS[Linux/5.19.0-50-generic/amd64], JVM[Ubuntu/OpenJDK 64-Bit Server VM/11.0.20/11.0.20+8-post-Ubuntu-1ubuntu122.04] [2023-08-07T15:00:49,095][INFO ][o.o.n.Node ] [runTask-0] JVM home [/usr/lib/jvm/java-11-openjdk-amd64], using bundled JDK [false] [2023-08-07T15:00:49,095][INFO ][o.o.n.Node ] [runTask-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/wazuh-indexer/build/testclusters/runTask-0/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=logs, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Xms512m, -Xmx512m, -ea, -esa, -XX:TieredStopAtLevel=1, -XX:ReservedCodeCacheSize=64m, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/wazuh-indexer/distribution/archives/linux-tar/build/install/opensearch-2.10.0-SNAPSHOT, -Dopensearch.path.conf=/wazuh-indexer/build/testclusters/runTask-0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true] [2023-08-07T15:00:49,096][WARN ][o.o.n.Node ] [runTask-0] version [2.10.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production [2023-08-07T15:00:49,736][INFO ][o.o.i.r.ReindexPlugin ] [runTask-0] ReindexPlugin reloadSPI called [2023-08-07T15:00:49,737][INFO ][o.o.i.r.ReindexPlugin ] [runTask-0] Unable to find any implementation for RemoteReindexExtension [2023-08-07T15:00:49,743][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [aggs-matrix-stats] [2023-08-07T15:00:49,743][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [analysis-common] [2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [geo] [2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [ingest-common] [2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [ingest-geoip] [2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [ingest-user-agent] [2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [lang-expression] [2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [lang-mustache] [2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [lang-painless] [2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [mapper-extras] [2023-08-07T15:00:49,744][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [opensearch-dashboards] [2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [parent-join] [2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [percolator] [2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [rank-eval] [2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [reindex] [2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [repository-url] [2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [search-pipeline-common] [2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [systemd] [2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [test-delayed-aggs] [2023-08-07T15:00:49,745][INFO ][o.o.p.PluginsService ] [runTask-0] loaded module [transport-netty4] [2023-08-07T15:00:49,746][INFO ][o.o.p.PluginsService ] [runTask-0] no plugins loaded [2023-08-07T15:00:49,755][INFO ][o.o.e.ExtensionsManager ] [runTask-0] ExtensionsManager initialized [2023-08-07T15:00:49,767][INFO ][o.o.e.NodeEnvironment ] [runTask-0] using [1] data paths, mounts [[/ (/dev/nvme0n1p6)]], net usable_space [209.7gb], net total_space [366.3gb], types [ext4] [2023-08-07T15:00:49,768][INFO ][o.o.e.NodeEnvironment ] [runTask-0] heap size [512mb], compressed ordinary object pointers [true] [2023-08-07T15:00:49,784][INFO ][o.o.n.Node ] [runTask-0] node name [runTask-0], node ID [avktelhUT36J9QzyRBZaiA], cluster name [runTask], roles [ingest, remote_cluster_client, data, cluster_manager] [2023-08-07T15:00:51,348][INFO ][o.o.t.NettyAllocator ] [runTask-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}] [2023-08-07T15:00:51,397][INFO ][o.o.d.DiscoveryModule ] [runTask-0] using discovery type [zen] and seed hosts providers [settings, file] [2023-08-07T15:00:51,559][WARN ][o.o.g.DanglingIndicesState] [runTask-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2023-08-07T15:00:51,690][INFO ][o.o.n.Node ] [runTask-0] initialized [2023-08-07T15:00:51,690][INFO ][o.o.n.Node ] [runTask-0] starting ... [2023-08-07T15:00:51,789][INFO ][o.o.t.TransportService ] [runTask-0] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300} [2023-08-07T15:00:51,790][INFO ][o.o.t.TransportService ] [runTask-0] Remote clusters initialized successfully. [2023-08-07T15:00:51,955][DEBUG][o.o.c.c.Coordinator ] [runTask-0] startInitialJoin: coordinator becoming CANDIDATE in term 0 (was null, lastKnownLeader was [Optional.empty]) [2023-08-07T15:00:51,958][INFO ][o.o.c.c.Coordinator ] [runTask-0] setting initial configuration to VotingConfiguration{avktelhUT36J9QzyRBZaiA} [2023-08-07T15:00:51,959][WARN ][o.o.d.FileBasedSeedHostsProvider] [runTask-0] expected, but did not find, a dynamic hosts list at [/wazuh-indexer/build/testclusters/runTask-0/config/unicast_hosts.txt] [2023-08-07T15:00:51,992][DEBUG][o.o.c.c.ElectionSchedulerFactory] [runTask-0] scheduling scheduleNextElection{gracePeriod=0s, thisAttempt=0, maxDelayMillis=100, delayMillis=10, ElectionScheduler{attempt=1, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} [2023-08-07T15:00:52,002][INFO ][o.o.h.AbstractHttpServerTransport] [runTask-0] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200} [2023-08-07T15:00:52,003][INFO ][o.o.n.Node ] [runTask-0] started [2023-08-07T15:00:52,003][DEBUG][o.o.c.c.ElectionSchedulerFactory] [runTask-0] scheduleNextElection{gracePeriod=0s, thisAttempt=0, maxDelayMillis=100, delayMillis=10, ElectionScheduler{attempt=1, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election [2023-08-07T15:00:52,004][DEBUG][o.o.c.c.ElectionSchedulerFactory] [runTask-0] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=1, maxDelayMillis=200, delayMillis=563, ElectionScheduler{attempt=2, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} [2023-08-07T15:00:52,007][DEBUG][o.o.c.c.PreVoteCollector ] [runTask-0] PreVotingRound{preVotesReceived={}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, currentTerm=0}, isClosed=false} requesting pre-votes from [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] [2023-08-07T15:00:52,012][DEBUG][o.o.c.c.PreVoteCollector ] [runTask-0] PreVotingRound{preVotesReceived={{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}=PreVoteResponse{currentTerm=0, lastAcceptedTerm=0, lastAcceptedVersion=0}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, currentTerm=0}, isClosed=false} added PreVoteResponse{currentTerm=0, lastAcceptedTerm=0, lastAcceptedVersion=0} from {runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, starting election [2023-08-07T15:00:52,014][DEBUG][o.o.c.c.Coordinator ] [runTask-0] starting election with StartJoinRequest{term=1,node={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}} [2023-08-07T15:00:52,014][DEBUG][o.o.c.c.Coordinator ] [runTask-0] joinLeaderInTerm: for [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] with term 1 [2023-08-07T15:00:52,014][DEBUG][o.o.c.c.CoordinationState] [runTask-0] handleStartJoin: leaving term [0] due to StartJoinRequest{term=1,node={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}} [2023-08-07T15:00:52,027][DEBUG][o.o.c.c.JoinHelper ] [runTask-0] attempting to join {runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, targetNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}}]} [2023-08-07T15:00:52,027][DEBUG][o.o.c.c.JoinHelper ] [runTask-0] successful response to StartJoinRequest{term=1,node={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}} from {runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true} [2023-08-07T15:00:52,028][DEBUG][o.o.c.c.CoordinationState] [runTask-0] handleJoin: added join Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, targetNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}} from [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] for election, electionWon=true lastAcceptedTerm=0 lastAcceptedVersion=0 [2023-08-07T15:00:52,028][DEBUG][o.o.c.c.CoordinationState] [runTask-0] handleJoin: election won in term [1] with VoteCollection{votes=[avktelhUT36J9QzyRBZaiA], joins=[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, targetNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}}]} [2023-08-07T15:00:52,028][DEBUG][o.o.c.c.Coordinator ] [runTask-0] handleJoinRequest: coordinator becoming LEADER in term 1 (was CANDIDATE, lastKnownLeader was [Optional.empty]) [2023-08-07T15:00:52,039][DEBUG][o.o.c.c.JoinHelper ] [runTask-0] received a join request for an existing node [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] [2023-08-07T15:00:52,041][INFO ][o.o.c.s.MasterService ] [runTask-0] elected-as-cluster-manager ([1] nodes joined)[{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]} [2023-08-07T15:00:52,053][DEBUG][o.o.c.c.PublicationTransportHandler] [runTask-0] received full cluster state version [1] with size [329] [2023-08-07T15:00:52,072][INFO ][o.o.c.c.CoordinationState] [runTask-0] cluster UUID set to [DXmrrJsuQH-F9z4FgDmppg] [2023-08-07T15:00:52,099][INFO ][o.o.c.s.ClusterApplierService] [runTask-0] cluster-manager node changed {previous [], current [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1} [2023-08-07T15:00:52,109][DEBUG][o.o.c.c.C.CoordinatorPublication] [runTask-0] publication ended successfully: Publication{term=1, version=1} [2023-08-07T15:00:52,113][DEBUG][o.o.c.c.JoinHelper ] [runTask-0] successfully joined {runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}, targetNode={runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}}]} [2023-08-07T15:00:52,113][INFO ][o.o.d.PeerFinder ] [runTask-0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{runTask-0}{avktelhUT36J9QzyRBZaiA}{I-ZL_6qoTSGSmdq7f6AoFQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] [2023-08-07T15:00:52,122][DEBUG][o.o.c.c.PublicationTransportHandler] [runTask-0] received full cluster state version [2] with size [282] [2023-08-07T15:00:52,131][DEBUG][o.o.c.c.C.CoordinatorPublication] [runTask-0] publication ended successfully: Publication{term=1, version=2} [2023-08-07T15:00:52,132][INFO ][o.o.g.GatewayService ] [runTask-0] recovered [0] indices into cluster_state [2023-08-07T15:00:52,568][DEBUG][o.o.c.c.ElectionSchedulerFactory] [runTask-0] scheduleNextElection{gracePeriod=500ms, thisAttempt=1, maxDelayMillis=200, delayMillis=563, ElectionScheduler{attempt=2, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} not starting election <============-> 99% EXECUTING [24m 32s] > IDLE > :run > IDLE > IDLE > IDLE > IDLE > IDLE > IDLE ```

API up and running

```json { "name" : "runTask-0", "cluster_name" : "runTask", "cluster_uuid" : "DXmrrJsuQH-F9z4FgDmppg", "version" : { "distribution" : "opensearch", "number" : "2.10.0-SNAPSHOT", "build_type" : "tar", "build_hash" : "0fcdd5647c613fd8e653eb68a3086dcaea314e06", "build_date" : "2023-08-07T12:56:56.904455Z", "build_snapshot" : true, "lucene_version" : "9.7.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } ```

AlexRuiz7 commented 1 year ago

Successfully generated distribution packages from branch 4.8.0.

The following distributions are being built:
* Archives (`*.zip`, `*.tar`): these form the basis for all other OpenSearch distributions
* Packages (`*.deb`, `*.rpm`): specific package formats for some Linux distributions
* Docker images
* Backwards compatibility tests: used internally for version compatibility testing, not for public consumption
Details

```java ./gradlew assemble ======================================= OpenSearch Build Hamster says Hello! Gradle Version : 8.2.1 OS Info : Linux 5.19.0-50-generic (amd64) JDK Version : 11 (Ubuntu JDK) JAVA_HOME : /usr/lib/jvm/java-11-openjdk-amd64 Random Testing Seed : D3FDD69024240051 In FIPS 140 mode : false ======================================= > Task :example-plugins:custom-suggester:compileJava Note: /home/alex/wazuh/wazuh-indexer/plugins/examples/custom-suggester/src/main/java/org/opensearch/example/customsuggester/CustomSuggestion.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :client:rest-high-level:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :benchmarks:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :build-tools:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :plugins:analysis-icu:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :plugins:discovery-ec2:compileJava Note: /home/alex/wazuh/wazuh-indexer/plugins/discovery-ec2/src/main/java/org/opensearch/discovery/ec2/Ec2DiscoveryPlugin.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :plugins:transport-nio:compileJava Note: /home/alex/wazuh/wazuh-indexer/plugins/transport-nio/src/main/java/org/opensearch/http/nio/HttpReadWriteHandler.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :plugins:repository-azure:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :build-tools:compileGroovy Note: /home/alex/wazuh/wazuh-indexer/buildSrc/src/main/groovy/org/opensearch/gradle/test/TestWithSslPlugin.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :test:framework:compileJava Note: Processing Log4j annotations Note: Annotations processed Note: Processing Log4j annotations Note: No elements to process Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :qa:full-cluster-restart:compileTestJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :qa:repository-multi-version:compileTestJava Note: /home/alex/wazuh/wazuh-indexer/qa/repository-multi-version/src/test/java/org/opensearch/upgrades/MultiVersionRepositoryAccessIT.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :plugins:repository-gcs:compileJava Note: /home/alex/wazuh/wazuh-indexer/plugins/repository-gcs/src/main/java/org/opensearch/repositories/gcs/GoogleCloudStorageBlobStore.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. BUILD SUCCESSFUL in 14m 50s 883 actionable tasks: 709 executed, 1 from cache, 173 up-to-date ```

AlexRuiz7 commented 1 year ago

As the generated packages do not include any plugins, I'm looking for the process they call assembly in the https://github.com/opensearch-project/opensearch-build repo.

AlexRuiz7 commented 1 year ago

I've been reading the documentation of the opensearch-build repo, but it's outdated. The mentioned files in the README do not match the existing files.

I've been figuring out which files do correspond to the mentioned ones in the documentation.

AlexRuiz7 commented 1 year ago

I managed to generate a "fat" deb package using OpenSearch's tooling:

There are dedicated runners (docker images) to run these tools:

# Start the runner attaching the tools as a volume
docker run -ti --rm -v .:/wazuh-build opensearchstaging/ci-runner:ci-runner-ubuntu2004-opensearch-build-v2 bash

# Build
./build.sh manifests/2.9.0/opensearch-2.9.0.yml --component OpenSearch security -d deb -p linux -a x64

# Assemble (install plugins)
./assemble.sh deb/builds/opensearch/manifest.yml
AlexRuiz7 commented 1 year ago

Included plugins (v4.6.0)

[root@rhel7 vagrant]# /usr/share/wazuh-indexer/bin/opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-sql
[root@rhel7 vagrant]# /usr/share/wazuh-dashboard/bin/opensearch-dashboards-plugin list --allow-root
alertingDashboards@2.8.0.0
customImportMapDashboards@2.8.0.0
ganttChartDashboards@2.8.0.0
indexManagementDashboards@2.8.0.0
mlCommonsDashboards@2.8.0.0
notificationsDashboards@2.8.0.0
reportsDashboards@2.8.0.0
securityDashboards@2.8.0.0
wazuh@4.6.0-00

Plugins are mostly 1:1, which means each plugin has an indexer and a dashboard component. We have discarded some plugins in dashboard, but haven't done the same with their indexer component.

Some plugins, as opensearch-neural-search, opensearch-knn and opensearch-ml (in this case mlCommonsDashboards too) could be skipped, as we do not have any reason to include them.

AlexRuiz7 commented 12 months ago

Process documented. Also in GDrive

Wazuh indexer packages generation

Compatibility matrix

Pre-requisites

Start the staging docker container:

docker run -ti --rm \
    -e JAVA_HOME=/opt/java/openjdk-17 \
    -v .:/usr/share/opensearch \
    opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3 \
    bash

Build minified packages

By using the build.sh script.

Usage:

usage: run_build.py [-h] [-l] [-s] [-c [COMPONENTS ...]] [--keep] [-p {linux,darwin,windows}] [-a {x64,arm64}] [-v] [-d {tar,zip,rpm,deb}]
                    [--continue-on-error]
                    manifest

Build an OpenSearch Distribution

positional arguments:
  manifest              Manifest file.

optional arguments:
  -h, --help            show this help message and exit
  -l, --lock            Generate a stable reference manifest.
  -s, --snapshot        Build snapshot.
  -c [COMPONENTS ...], --component [COMPONENTS ...]
                        Rebuild one or more components.
  --keep                Do not delete the working temporary directory.
  -p {linux,darwin,windows}, --platform {linux,darwin,windows}
                        Platform to build.
  -a {x64,arm64}, --architecture {x64,arm64}
                        Architecture to build.
  -v, --verbose         Show more verbose output.
  -d {tar,zip,rpm,deb}, --distribution {tar,zip,rpm,deb}
                        Distribution to build.
  --continue-on-error   Do not fail the distribution build on any plugin component failure.

Build RPM and DEB packages for Linux x64:

./build.sh manifests/2.9.0/opensearch-2.9.0.yml -d rpm -p linux -a x64
./build.sh manifests/2.9.0/opensearch-2.9.0.yml -d deb -p linux -a x64

Assemble packages

By using the assemble.sh script.

Usage:

usage: run_assemble.py [-h] [-b BASE_URL] [--keep] [-v] manifest

Assemble an OpenSearch Distribution

positional arguments:
  manifest              Manifest file.

optional arguments:
  -h, --help            show this help message and exit
  -b BASE_URL, --base-url BASE_URL
                        The base url to download the artifacts.
  --keep                Do not delete the working temporary directory.
  -v, --verbose         Show more verbose output.

Assemble RPM and DEB packages from the previous step:

./assemble.sh rpm/builds/opensearch/manifest.yml
./assemble.sh deb/builds/opensearch/manifest.yml
AlexRuiz7 commented 10 months ago

Continued in wazuh/wazuh-packages#54