wazuh / wazuh-packages

Wazuh - Tools for packages creation
https://wazuh.com
GNU General Public License v2.0
105 stars 96 forks source link

PoC - Wazuh Indexer package with Opensearch #1099

Closed okynos closed 2 years ago

okynos commented 2 years ago

Hello team,

We are working on create a new package wazuh-indexer with Opensearch support. We will describe all work and requirements here to keep track on it. Tasks:

RPM

Debian

okynos commented 2 years ago

We are working on this branch poc-indexer We have added an RPM SPEC and builder scripts. We have added migration procedure. We have tested this migration on a single node with unattended installation All In One.

Resources used to achieve this matter:

okynos commented 2 years ago

Hello team,

We have include tons of modifications of wazuh-indexer.spec file:

Command to run wazuh-indexer after install:

sudo -u wazuh-indexer CLK_TK=`/usr/bin/getconf CLK_TCK` OPENSEARCH_PATH_CONF=/etc/wazuh-indexer /usr/share/wazuh-indexer/bin/opensearch

Commands to test that it works:

curl -k -u admin:admin https://localhost:9700
curl -k -u admin:admin https://localhost:9700/_cluster/health?pretty
curl -k -u admin:admin https://localhost:9700/_cat/indices?pretty

Link to the package: https://s3.amazonaws.com/warehouse.wazuh.com/indexer/wazuh-indexer-4.3.0-1.x86_64.rpm

Way to install: yum install -y https://s3.amazonaws.com/warehouse.wazuh.com/indexer/wazuh-indexer-4.3.0-1.x86_64.rpm

rauldpm commented 2 years ago

Testing - CentOS 8


yum install -y https://s3.amazonaws.com/warehouse.wazuh.com/indexer/wazuh-indexer-4.3.0-1.x86_64.rpm


sudo -u wazuh-indexer CLK_TK=`/usr/bin/getconf CLK_TCK` OPENSEARCH_PATH_CONF=/etc/wazuh-indexer /usr/share/wazuh-indexer/bin/opensearch
Warnings and errors ``` [WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] Directory /etc/wazuh-indexer has insecure file permissions (should be 0700) [WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /etc/wazuh-indexer/jvm.options.d/performance-analyzer.options has insecure file permissions (should be 0600) ``` - This is later disabled: `PerformanceAnalyzer Enabled: false` - Needed to increase max_map_count. Failed ``` [2021-12-28T17:10:49,853][INFO ][o.o.b.BootstrapChecks ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks ERROR: [1] bootstrap checks failed [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] ERROR: OpenSearch did not exit normally - check the logs at /var/log/wazuh-indexer/wazuh-cluster.log [2021-12-28T17:10:49,858][INFO ][o.o.s.a.r.AuditMessageRouter] [node-1] Closing AuditMessageRouter [2021-12-28T17:10:49,858][INFO ][o.o.s.a.s.SinkProvider ] [node-1] Closing InternalOpenSearchSink [2021-12-28T17:10:49,858][INFO ][o.o.s.a.s.SinkProvider ] [node-1] Closing DebugSink [2021-12-28T17:10:49,859][INFO ][o.o.n.Node ] [node-1] stopping ... [2021-12-28T17:10:49,867][INFO ][o.o.n.Node ] [node-1] stopped [2021-12-28T17:10:49,867][INFO ][o.o.n.Node ] [node-1] closing ... [2021-12-28T17:10:49,871][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Closing AuditLogImpl [2021-12-28T17:10:49,873][INFO ][o.o.n.Node ] [node-1] closed [root@centos8 vagrant]# ``` - After increase max_map_count to 262144: ``` [2021-12-28T17:14:57,513][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security) org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]; at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:202) ~[opensearch-1.2.3.jar:1.2.3] at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:188) ~[opensearch-1.2.3.jar:1.2.3] at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:76) ~[opensearch-1.2.3.jar:1.2.3] at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:53) ~[opensearch-1.2.3.jar:1.2.3] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:194) [opensearch-1.2.3.jar:1.2.3] at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:141) [opensearch-index-management-1.2.3.0.jar:1.2.3.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.3.jar:1.2.3] at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:234) [opensearch-security-1.2.3.0.jar:1.2.3.0] at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) [opensearch-security-1.2.3.0.jar:1.2.3.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.3.jar:1.2.3] at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.2.3.0.jar:1.2.3.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.2.3.jar:1.2.3] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-1.2.3.jar:1.2.3] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-1.2.3.jar:1.2.3] at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-1.2.3.jar:1.2.3] at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:95) [opensearch-1.2.3.jar:1.2.3] at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433) [opensearch-1.2.3.jar:1.2.3] at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:554) [opensearch-1.2.3.jar:1.2.3] at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:211) [opensearch-security-1.2.3.0.jar:1.2.3.0] at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:102) [opensearch-security-1.2.3.0.jar:1.2.3.0] at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:375) [opensearch-security-1.2.3.0.jar:1.2.3.0] at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:321) [opensearch-security-1.2.3.0.jar:1.2.3.0] at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:306) [opensearch-security-1.2.3.0.jar:1.2.3.0] at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:166) [opensearch-security-1.2.3.0.jar:1.2.3.0] at java.lang.Thread.run(Thread.java:832) [?:?] ``` - Stuck in loop: ``` [2021-12-28T17:16:41,633][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security) ```


curl -k -u admin:admin https://localhost:9700
curl -k -u admin:admin https://localhost:9700/_cluster/health?pretty
curl -k -u admin:admin https://localhost:9700/_cat/indices?pretty

[root@centos8 vagrant]# curl -k -u admin:admin https://localhost:9700 OpenSearch Security not initialized. [root@centos8 vagrant]# curl -k -u admin:admin https://localhost:9700/_cluster/health?pretty OpenSearch Security not initialized. [root@centos8 vagrant]# curl -k -u admin:admin https://localhost:9700/_cat/indices?pretty OpenSearch Security not initialized. [root@centos8 vagrant]#

okynos commented 2 years ago

Hello team,

We have performed new iteration of wazuh-indexer package:

Way to install:

yum install -y https://s3.amazonaws.com/warehouse.wazuh.com/indexer/wazuh-indexer-4.3.0-1.x86_64.rpm

Command to run wazuh-indexer after install:

sudo systemctl enable wazuh-indexer
sudo systemctl start wazuh-indexer

Commands to test that it works:

curl -k -u admin:admin https://localhost:9700
curl -k -u admin:admin https://localhost:9700/_cluster/health?pretty
curl -k -u admin:admin https://localhost:9700/_cat/indices?pretty

Link to the package: https://s3.amazonaws.com/warehouse.wazuh.com/indexer/wazuh-indexer-4.3.0-1.x86_64.rpm

References

dariommr commented 2 years ago

Hello Team, I was testing this in a CentOS 6, since some people is running this OS yet, and found some issues that I was able to solve and get it running:

CentOS 6: Issues found

  1. Errors when installing, solved changing the repos. It seems the official repos for CentOS 6 changed Solution:
    echo "https://vault.centos.org/6.10/" >> /var/cache/yum/x86_64/6/base/mirrorlist.txt

Edit the file /etc/yum.repos.d/Centos-Base.repo changing the [base] block for this:

[base]
name=CentOS-$releasever - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
baseurl=http://vault.centos.org/6.10/centosplus/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
gpgcheck=1

  1. Errors starting the indexer service:
    [root@wzh-indxr ~]# cat /var/log/wazuh-indexer/wazuh-cluster.log | grep "bootstrap checks failed" -A2 -B1
    [2021-12-31T09:49:08,150][ERROR][o.o.b.Bootstrap          ] [node-1] node validation exception
    [2] bootstrap checks failed
    [1]: max number of threads [1024] for user [wazuh-indexer] is too low, increase to at least [4096]
    [2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

Solution: Edit the file /etc/security/limits.d/90-nproc.conf and changed the value for all users to 4096:

*          soft    nproc     4096

And edited the configuration file /etc/wazuh-indexer/opensearch.yml to solve the issues about system call filters:

bootstrap.system_call_filter: false

  1. Errors in API calls:
    # curl -k -u admin:admin https://localhost:9700
    OpenSearch Security not initialized.

Solution: Execute the securityadmin.sh script to re-create the .opendistro index:

export JAVA_HOME=/usr/share/wazuh-indexer/jdk/ &&  bash /usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh -cd /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/ -nhnv -cacert /etc/wazuh-indexer/certs/root-ca.pem -cert /etc/wazuh-indexer/certs/admin.pem -key /etc/wazuh-indexer/certs/admin-key.pem -p 9800 -icl

With that, I was able to run the curl command correctly:

# curl -k -u admin:admin https://localhost:9700
{
  "name" : "node-1",
  "cluster_name" : "wazuh-cluster",
  "cluster_uuid" : "DrfeTYEDSgKaB62PRDUL8A",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.2.3",
    "build_type" : "rpm",
    "build_hash" : "8a529d77c7432bc45b005ac1c4ba3b2741b57d4a",
    "build_date" : "2021-12-21T01:36:21.407473Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

I hope this could be helpful.

okynos commented 2 years ago

We have included the following changes:

okynos commented 2 years ago

Hello team,

We have added some improvements into the Debian package 86931db8:

First package URL: https://s3.amazonaws.com/warehouse.wazuh.com/indexer/wazuh-indexer_4.3.0-1_amd64.deb

okynos commented 2 years ago

Hello team,

We have added the following changes d8b8f3e7:

Package URL today: https://s3.amazonaws.com/warehouse.wazuh.com/indexer/wazuh-indexer_4.3.0-1_amd64.deb

okynos commented 2 years ago

Hello team,

Today we have done tons of improvements with commits: f9aac8ed7a63c8b449df8961926987211d4d82a0 a343063d3475b7c3fbc82db86df8ec769be25ad1 054a1427d5d480ffd68da79e994aec47a64eaf7f bc5e50ecaa44e1c7521fd187b4ca08f5267cc7c2 af189e1ef80fff42aa3b88015faa189d6c57078f 5f1fbab64ee417482deca6090380f91364518b41

Little research links:

Package URL today: https://s3.amazonaws.com/warehouse.wazuh.com/indexer/wazuh-indexer_4.3.0-1_amd64.deb

okynos commented 2 years ago

Hello team,

Today we have made great improvements!


okynos commented 2 years ago

Hello team,

We have finishing the last details over packages and we will detail here all matters

Debian Lintian:

E: wazuh-indexer changes: bad-distribution-in-changes-file stable
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jaotc
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jar
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jarsigner
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/java
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/javac
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/javadoc
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/javap
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jcmd
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jconsole
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jdb
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jdeprscan
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jdeps
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jfr
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jhsdb
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jimage
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jinfo
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jlink
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jmap
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jmod
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jpackage
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jps
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jrunscript
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jshell
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jstack
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jstat
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/jstatd
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/keytool
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/rmid
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/rmiregistry
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/bin/serialver
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/jexec
E: wazuh-indexer: unstripped-binary-or-object usr/share/wazuh-indexer/jdk/lib/jexec
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/jspawnhelper
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libattach.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libawt.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libawt_headless.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libawt_xawt.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libdt_socket.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libextnet.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libfontmanager.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libfreetype.so
E: wazuh-indexer: embedded-library usr/share/wazuh-indexer/jdk/lib/libfreetype.so: freetype
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libinstrument.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libj2gss.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libj2pcsc.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libj2pkcs11.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libjaas.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libjava.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libjavajpeg.so
E: wazuh-indexer: embedded-library usr/share/wazuh-indexer/jdk/lib/libjavajpeg.so: libjpeg
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libjawt.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libjdwp.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libjimage.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libjli.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libjsig.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libjsound.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/liblcms.so
E: wazuh-indexer: embedded-library usr/share/wazuh-indexer/jdk/lib/liblcms.so: lcms2
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libmanagement.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libmanagement_agent.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libmanagement_ext.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libmlib_image.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libnet.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libnio.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libprefs.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/librmi.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libsaproc.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libsctp.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libsplashscreen.so
E: wazuh-indexer: embedded-library usr/share/wazuh-indexer/jdk/lib/libsplashscreen.so: libjpeg
E: wazuh-indexer: embedded-library usr/share/wazuh-indexer/jdk/lib/libsplashscreen.so: libpng
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libsunec.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libverify.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/libzip.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/server/libjsig.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/jdk/lib/server/libjvm.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/plugins/opensearch-knn/knnlib/libgomp.so.1
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/plugins/opensearch-knn/knnlib/libopensearchknn_common.so
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/plugins/opensearch-knn/knnlib/libopensearchknn_faiss.so
E: wazuh-indexer: binary-or-shlib-defines-rpath usr/share/wazuh-indexer/plugins/opensearch-knn/knnlib/libopensearchknn_faiss.so /tmp/tmppye_j7uv/k-NN/jni/release
E: wazuh-indexer: arch-dependent-file-in-usr-share usr/share/wazuh-indexer/plugins/opensearch-knn/knnlib/libopensearchknn_nmslib.so
E: wazuh-indexer: binary-or-shlib-defines-rpath usr/share/wazuh-indexer/plugins/opensearch-knn/knnlib/libopensearchknn_nmslib.so /tmp/tmppye_j7uv/k-NN/jni/release
E: wazuh-indexer: missing-dependency-on-libc needed by usr/share/wazuh-indexer/jdk/bin/jaotc and 73 others
W: wazuh-indexer: new-package-should-close-itp-bug
E: wazuh-indexer: no-copyright-file
W: wazuh-indexer: description-too-long
E: wazuh-indexer: extended-description-is-empty
W: wazuh-indexer: extra-license-file usr/share/wazuh-indexer/jdk/legal/java.base/LICENSE
W: wazuh-indexer: extra-license-file usr/share/wazuh-indexer/plugins/opensearch-sql/LICENSE.txt
E: wazuh-indexer: prerm-calls-updaterc.d wazuh-indexer
W: wazuh-indexer: script-in-etc-init.d-not-registered-via-update-rc.d etc/init.d/wazuh-indexer
E: wazuh-indexer: init.d-script-missing-dependency-on-remote_fs etc/init.d/wazuh-indexer: required-start
E: wazuh-indexer: init.d-script-missing-dependency-on-remote_fs etc/init.d/wazuh-indexer: required-stop
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/jackson-core-2.12.5.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/jackson-dataformat-cbor-2.12.5.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/jackson-dataformat-smile-2.12.5.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/jackson-dataformat-yaml-2.12.5.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/log4j-api-2.17.0.jar (META-INF/versions/9/module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/opensearch-1.2.3.jar (META-INF/versions/11/org/opensearch/monitor/jvm/JvmPid.class -> 55)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/opensearch-core-1.2.3.jar (META-INF/versions/11/org/opensearch/common/collect/List.class -> 55)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/tools/plugin-cli/bc-fips-1.0.2.jar (META-INF/versions/9/module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/tools/plugin-cli/bcpg-fips-1.0.4.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/tools/upgrade-cli/jackson-annotations-2.12.5.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/tools/upgrade-cli/jackson-core-2.12.5.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/lib/tools/upgrade-cli/jackson-databind-2.12.5.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/modules/ingest-common/jcodings-1.0.44.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/modules/ingest-common/joni-2.1.29.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/modules/ingest-geoip/jackson-annotations-2.12.5.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/modules/ingest-geoip/jackson-databind-2.12.5.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/modules/lang-painless/asm-7.2.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/modules/lang-painless/asm-analysis-7.2.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/modules/lang-painless/asm-commons-7.2.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/modules/lang-painless/asm-tree-7.2.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/modules/lang-painless/asm-util-7.2.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/performance-analyzer-rca/lib/bcpkix-jdk15on-1.68.jar (META-INF/versions/9/module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/performance-analyzer-rca/lib/bcprov-jdk15on-1.68.jar (META-INF/versions/11/org/bouncycastle/jcajce/provider/asymmetric/edec/BC11XDHPrivateKey.class -> 55)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/performance-analyzer-rca/lib/gson-2.8.6.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/performance-analyzer-rca/lib/jackson-annotations-2.11.4.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/performance-analyzer-rca/lib/jackson-core-2.11.4.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/performance-analyzer-rca/lib/jackson-databind-2.11.4.jar (module-info.class -> 53)
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/performance-analyzer-rca/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/performance-analyzer-rca/lib/log4j-api-2.17.0.jar (META-INF/versions/9/module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-alerting/google-java-format-1.10.0.jar (com/google/googlejavaformat/CloseOp.class -> 55)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-alerting/ipaddress-5.3.3.jar (module-info.class -> 53)
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/plugins/opensearch-alerting/kotlin-stdlib-common-1.3.72.jar
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/plugins/opensearch-alerting/kotlinx-coroutines-core-common-1.1.1.jar
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/plugins/opensearch-alerting/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-anomaly-detection/gson-2.8.6.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-anomaly-detection/jackson-annotations-2.11.4.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-anomaly-detection/jackson-databind-2.11.4.jar (module-info.class -> 53)
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/plugins/opensearch-anomaly-detection/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-cross-cluster-replication/ipaddress-5.3.3.jar (module-info.class -> 53)
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/plugins/opensearch-cross-cluster-replication/kotlin-stdlib-common-1.3.72.jar
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-index-management/ipaddress-5.3.3.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-index-management/kotlin-stdlib-1.4.0.jar (META-INF/versions/9/module-info.class -> 53)
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/plugins/opensearch-index-management/kotlin-stdlib-common-1.4.0.jar
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-index-management/kotlin-stdlib-jdk7-1.4.0.jar (META-INF/versions/9/module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-index-management/kotlin-stdlib-jdk8-1.4.0.jar (META-INF/versions/9/module-info.class -> 53)
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/plugins/opensearch-knn/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-observability/kotlin-stdlib-1.4.0.jar (META-INF/versions/9/module-info.class -> 53)
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/plugins/opensearch-observability/kotlin-stdlib-common-1.4.0.jar
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/bcpkix-jdk15on-1.68.jar (META-INF/versions/9/module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/bcprov-jdk15on-1.68.jar (META-INF/versions/11/org/bouncycastle/jcajce/provider/asymmetric/edec/BC11XDHPrivateKey.class -> 55)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/gson-2.8.6.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/jackson-annotations-2.11.4.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/jackson-databind-2.11.4.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/jackson-module-paranamer-2.11.4.jar (module-info.class -> 53)
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-reports-scheduler/kotlin-stdlib-1.4.0.jar (META-INF/versions/9/module-info.class -> 53)
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/plugins/opensearch-reports-scheduler/kotlin-stdlib-common-1.4.0.jar
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/asm-9.1.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/bcprov-jdk15on-1.67.jar (META-INF/versions/11/org/bouncycastle/jcajce/provider/asymmetric/edec/BC11XDHPrivateKey.class -> 55)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/istack-commons-runtime-3.0.12.jar (META-INF/versions/9/com/sun/istack/logging/StackHelper.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/jackson-annotations-2.11.2.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/jackson-databind-2.11.2.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/jakarta.activation-1.2.2.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/jakarta.jws-api-2.1.0.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/jakarta.xml.bind-api-2.3.3.jar (META-INF/versions/9/javax/xml/bind/ModuleUtil.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/jakarta.xml.soap-api-1.4.2.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/jakarta.xml.ws-api-2.3.3.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/jaxb-runtime-2.3.4.jar (META-INF/versions/9/com/sun/xml/bind/StackHelper.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/saaj-impl-1.5.3.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/stax-ex-1.8.3.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/stax2-api-4.2.1.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/txw2-2.3.4.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-security/woodstox-core-6.2.6.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-sql/gson-2.8.6.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-sql/jackson-annotations-2.11.4.jar (module-info.class -> 53)
W: wazuh-indexer: unknown-java-class-version usr/share/wazuh-indexer/plugins/opensearch-sql/jackson-databind-2.11.4.jar (module-info.class -> 53)
W: wazuh-indexer: codeless-jar usr/share/wazuh-indexer/plugins/opensearch-sql/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
W: wazuh-indexer: script-not-executable usr/share/wazuh-indexer/bin/systemd-entrypoint
W: wazuh-indexer: script-not-executable usr/share/wazuh-indexer/performance-analyzer-rca/pa_bin/performance-analyzer-agent
W: wazuh-indexer: script-not-executable usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/extensions/performance-analyzer-agent
W: wazuh-indexer: script-not-executable usr/share/wazuh-indexer/plugins/opensearch-performance-analyzer/pa_bin/performance-analyzer-agent
W: wazuh-indexer: script-not-executable usr/share/wazuh-indexer/plugins/opensearch-security/tools/audit_config_migrater.sh
W: wazuh-indexer: script-not-executable usr/share/wazuh-indexer/plugins/opensearch-security/tools/hash.sh
W: wazuh-indexer: script-not-executable usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh
W: wazuh-indexer: script-not-executable usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-cert-tool.sh
W: wazuh-indexer: script-not-executable usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh
W: wazuh-indexer: executable-not-elf-or-script usr/share/wazuh-indexer/NOTICE.txt
W: wazuh-indexer: executable-not-elf-or-script usr/share/wazuh-indexer/LICENSE.txt
W: wazuh-indexer: command-with-path-in-maintainer-script postinst:68 /usr/bin/getconf
W: wazuh-indexer: possible-bashism-in-maintainer-script postinst:77 '?ulimit '
W: wazuh-indexer: maintainer-script-ignores-errors postinst
W: wazuh-indexer: maintainer-script-ignores-errors postrm
W: wazuh-indexer: maintainer-script-ignores-errors preinst
W: wazuh-indexer: maintainer-script-ignores-errors prerm
W: wazuh-indexer: postinst-has-useless-call-to-ldconfig
E: wazuh-indexer: systemd-service-file-outside-lib usr/lib/systemd/system/wazuh-indexer-performance-analyzer.service
E: wazuh-indexer: systemd-service-file-outside-lib usr/lib/systemd/system/wazuh-indexer.service
W: wazuh-indexer: init.d-script-does-not-source-init-functions etc/init.d/wazuh-indexer
W: wazuh-indexer: maintainer-script-calls-systemctl postinst:44
W: wazuh-indexer: maintainer-script-calls-systemctl prerm:48
W: wazuh-indexer: maintainer-script-calls-systemctl prerm:75

We have to solve as many as possible Lintian problems.

Main problems:

Problems solved:

Generated stable packages: DEB: https://s3.amazonaws.com/warehouse.wazuh.com/indexer/stable/wazuh-indexer_4.3.0-1_amd64.deb RPM: https://s3.amazonaws.com/warehouse.wazuh.com/indexer/stable/wazuh-indexer-4.3.0-1.x86_64.rpm

okynos commented 2 years ago

13/01/2022

Hello team,

Today we have included:

14/01/2022

We have included:

Changes:

References of research

okynos commented 2 years ago

Hello team,

Today we have added some modifications plus tests:

We have made changes on preun and postun on RPM SPECs Also we have moved such changes to Debian packages to keep both equally. We have removed names from copyright files.

Test: We have tested the full removal on RPM We want to test the upgrade part of SPECs so we are generating an OpenSearch 1.2.2 base to build 4.3.0 packages then, upgrade to 9.99 packages.

Regards.

okynos commented 2 years ago

Hello team,

Today we have done UX tests with @alberpilot. We have produced some preliminary questions about UX that probably we have to discuss or improve.

Commits:

Fixed problem into ulimit configuration when start the indexer. Added restart of the service on upgrade package part/SPECs Reviewed wazuh-dashboard package SPECs suggest some improvements.

okynos commented 2 years ago

Hello team,

We have added the following change:

We have changed the order to set files permissions. We have added improvements to the installation time (postinstall) to decrease the installation time by 15 seconds on both systems, Debian and RPM. Finally we have changed securityadmin.sh command launch to use runuser instead of sudo, this fix several problems in docker environment.

okynos commented 2 years ago

Hello team,

We are working on docker installation, we want to ensure that the package works in the worst conditions, to achieve that we have added some fixes:

We have moved permissions from postinstall to rules section on dh_fixperms Added -h option to the security admin launch. Fixed sysctl command call into debian dockers. Added purge to prerm maintainer script. Fixed service file inside base package to support docker installation and service use. Research about new changes made on 1.2.4 Prepared new base for 1.2.4, need work. We want to keep this package ready for prod!!

okynos commented 2 years ago

Hello team,

We have made changes to indexer SPECs:

31/01/2022

01/02/2022

We could announce now the V1 release of indexer package, we close this issue now in favour of https://github.com/wazuh/wazuh-packages/issues/1225

v1 indexer packages: DEB: https://s3.amazonaws.com/warehouse.wazuh.com/stack/indexer/stable/v1/wazuh-indexer_4.3.0-1_amd64.deb RPM: https://s3.amazonaws.com/warehouse.wazuh.com/stack/indexer/stable/v1/wazuh-indexer-4.3.0-1.x86_64.rpm