search

wazuh / wazuh-packages

Wazuh - Tools for packages creation
https://wazuh.com
GNU General Public License v2.0
105 stars 95 forks source link

Include ARM support to indexer and dashboard #1637

Closed okynos closed 5 months ago

okynos commented 2 years ago

Hello all,

Description

We need to include a procedure and scripts to build ARM packages from indexer and dashboard base package. Some users are requesting ARM support for our Wazuh indexer and dashboard products. We want to include a way to manage the ARM package build and include an automatic procedure to build and release them.

Tasks

Validation

c-bordon commented 2 years ago

I found a bug in the Debian 8 repositories for arm, the build worked with Debian 9, I have to confirm that this is valid

Sending build context to Docker daemon   5.12kB
Step 1/6 : FROM arm64v8/debian:8
8: Pulling from arm64v8/debian
559afc254be4: Pull complete 
Digest: sha256:410257f511e4fe5cf234148cff2fddf5fc8586f4bb8a9a2d4c56e98190928e7c
Status: Downloaded newer image for arm64v8/debian:8
 ---> 89af0bd7c556
Step 2/6 : ENV DEBIAN_FRONTEND noninteractive
 ---> Running in 86595267fd11
Removing intermediate container 86595267fd11
 ---> d4f781a9fac4
Step 3/6 : RUN apt-get update && apt-get install -y apt-utils &&     apt-get install -y     curl sudo wget expect gnupg build-essential     devscripts equivs selinux-basics procps gawk
 ---> Running in ea3213046e05
Get:1 http://security.debian.org jessie/updates InRelease [44.9 kB]
Ign http://deb.debian.org jessie InRelease
Get:2 http://deb.debian.org jessie-updates InRelease [16.3 kB]
Get:3 http://deb.debian.org jessie Release.gpg [1652 B]
Get:4 http://deb.debian.org jessie Release [77.3 kB]
Fetched 140 kB in 0s (1134 kB/s)
W: Failed to fetch http://security.debian.org/debian-security/dists/jessie/updates/InRelease  Unable to find expected entry 'main/binary-arm64/Packages' in Release file (Wrong sources.list entry or malformed file)

W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/InRelease  Unable to find expected entry 'main/binary-arm64/Packages' in Release file (Wrong sources.list entry or malformed file)                                      

W: Failed to fetch http://deb.debian.org/debian/dists/jessie/Release  Unable to find expected entry 'main/binary-arm64/Packages' in Release file (Wrong sources.list entry or malformed file)                                                

E: Some index files failed to download. They have been ignored, or old ones used instead.                                                                                                                                                    
The command '/bin/sh -c apt-get update && apt-get install -y apt-utils &&     apt-get install -y     curl sudo wget expect gnupg build-essential     devscripts equivs selinux-basics procps gawk' returned a non-zero code: 100
c-bordon commented 2 years ago

I managed to create the first Wazuh dashboard and indexer 4.3.5 packages for ARM64, I only managed to successfully test the installation of dashboard on both Ubuntu 20.04 and Amazon Linux 2.

It remains to test the installation of indexer and then the operation of the complete stack on these systems.

Another thing that I have to solve is the creation of these packages through the automatic process of Jenkins, I still need to create the necessary images for these processes.

Packages:

Dashboard:

c-bordon commented 2 years ago

We do not have filebeat 7.10.2 packages for arm64 architecture, so this requires further investigation, the options are:

In the branch https://github.com/wazuh/wazuh-packages/tree/1637-RPMSuportfoStack-to4.3 are the changes made for the creation of the arm64 packages

thiscantbeserious commented 2 years ago

@c-bordon I'm currently trying to test-compile your branch-changes on an arm64 bullseye node (debian 11) ...

How exactly do the commands look like to build the dashboard and indexer packages?

Following https://documentation.wazuh.com/current/development/packaging/generate-deb-package.html there's only the

manager and client

options for the shellscript in debs subdir

(https://github.com/wazuh/wazuh-packages/blob/1637-RPMSuportfoStack-to4.3/debs/generate_debian_package.sh)

Edit I recon these are in here?

https://github.com/wazuh/wazuh-packages/tree/master/stack

Do they get used by the above shell script or are they unrelated?

c-bordon commented 2 years ago

Hi @thiscantbeserious, I share with you some steps that you should execute to build the packages.

  1. You must work on the https://github.com/wazuh/wazuh-packages/tree/1637-RPMSuportfoStack-to4.3 branch since this development is not completed

  2. You must run the scripts on an arm64 host

  3. You must create the bases for both indexer and dashboard by executing the script as follows:

    ./generate_base.sh -a arm64 -s /tmp

    This script is located at:

    wazuh-packages/stack/dashboard/base
wazuh-packages/stack/indexer/base

    The -s is to put the base file in /tmp, you can change the path or omit this parameter and an output directory will be created in the directory where the script is located.

  4. You must create the packages by executing the following script:

    ./build_package.sh -a arm64 -s /tmp --base local --base-path /tmp

    This script is located at:

    wazuh-packages/stack/dashboard/rpm
wazuh-packages/stack/dashboard/deb
wazuh-packages/stack/indexer/rpm
wazuh-packages/stack/indexer/deb

    the -s works the same as for step 3 --base-path has to contain the directory where you put the base in step 3

With this, you should be able to build your Wazuh indexer and Wazuh dashboard arm64 packages

RedstoneWizard08 commented 1 year ago

By the way, if it helps, I think I got filebeat 7.10.2 to package for ARM64 with this patch (its long lol):

Show patch content ```patch diff --git a/dev-tools/mage/build.go b/dev-tools/mage/build.go index 2efe61502a..1b616f8db2 100644 --- a/dev-tools/mage/build.go +++ b/dev-tools/mage/build.go @@ -83,6 +83,24 @@ func DefaultGolangCrossBuildArgs() BuildArgs { return args } +// DefaultGolangNativeBuildArgs returns the default BuildArgs for use in +// native-builds. +func DefaultGolangNativeBuildArgs() BuildArgs { + args := DefaultBuildArgs() + args.Name += "-" + Platform.GOOS + "-" + Platform.Arch + args.OutputDir = filepath.Join("build", "golang-nativebuild") + if bp, found := BuildPlatforms.Get(Platform.Name); found { + args.CGO = bp.Flags.SupportsCGO() + } + + // Enable DEP (data execution protection) for Windows binaries. + if Platform.GOOS == "windows" { + args.LDFlags = append(args.LDFlags, "-extldflags=-Wl,--nxcompat") + } + + return args +} + // GolangCrossBuild invokes "go build" inside of the golang-crossbuild Docker // environment. func GolangCrossBuild(params BuildArgs) error { @@ -95,6 +113,11 @@ func GolangCrossBuild(params BuildArgs) error { return Build(params) } +// GolangNativeBuild invokes "go build" natively. +func GolangNativeBuild(params BuildArgs) error { + return Build(params) +} + // Build invokes "go build" to produce a binary. func Build(params BuildArgs) error { fmt.Println(">> build: Building", params.Name) diff --git a/dev-tools/mage/godaemon.go b/dev-tools/mage/godaemon.go index b580bc3ed5..6e5227d215 100644 --- a/dev-tools/mage/godaemon.go +++ b/dev-tools/mage/godaemon.go @@ -79,6 +79,46 @@ func BuildGoDaemon() error { return RunCmds(compileCmd) } +// BuildGoDaemon builds the go-deamon binary. +func NativeBuildGoDaemon() error { + if GOOS != "linux" { + return errors.New("go-daemon only builds for linux") + } + + // Test if binaries are up-to-date. + godaemonDir, err := gotool.ListModuleCacheDir("github.com/tsg/go-daemon") + if err != nil { + return err + } + input := filepath.Join(godaemonDir, "src", "god.c") + output := MustExpand("build/golang-nativebuild/god-{{.Platform.GOOS}}-{{.Platform.Arch}}") + if IsUpToDate(output, input) { + log.Println(">>> nativeBuildGoDaemon is up-to-date for", Platform.Name) + return nil + } + + // Determine what compiler to use based on CC that is set by golang-crossbuild. + cc := os.Getenv("CC") + if cc == "" { + cc = "cc" + } + + compileCmd := []string{ + cc, + input, + "-o", createDir(output), + "-lpthread", "-static", + } + switch Platform.Name { + case "linux/amd64": + compileCmd = append(compileCmd, "-m64") + case "linux/386": + compileCmd = append(compileCmd, "-m32") + } + + return RunCmds(compileCmd) +} + // CrossBuildGoDaemon cross-build the go-daemon binary using the // golang-crossbuild environment. func CrossBuildGoDaemon(options ...CrossBuildOption) error { diff --git a/dev-tools/mage/pkg.go b/dev-tools/mage/pkg.go index 4ecdec89d3..5a7366e85a 100644 --- a/dev-tools/mage/pkg.go +++ b/dev-tools/mage/pkg.go @@ -106,6 +106,87 @@ func Package() error { return nil } +// Package native packages the Beat for distribution. It generates packages based on +// the current platform and registered packaging specifications. +func NativePackage() error { + if len(Packages) == 0 { + return errors.New("no package specs are registered. Call " + + "UseCommunityBeatPackaging, UseElasticBeatPackaging or USeElasticBeatWithoutXPackPackaging first.") + } + + var tasks []interface{} + + currentPlatform := BuildPlatform{ + Name: Platform.GOOS + "/" + Platform.GOARCH, + } + + os.Setenv("PLATFORMS", currentPlatform.Name) + + currentPlatform.Flags = BuildPlatforms.filter(func(p BuildPlatform) bool { + return p.Name == currentPlatform.Name + })[0].Flags + + for _, pkg := range Packages { + if pkg.OS != currentPlatform.GOOS() { + continue + } + + for _, pkgType := range pkg.Types { + if pkgType == DMG && runtime.GOOS != "darwin" { + log.Printf("Skipping DMG package type because build host isn't darwin") + continue + } + + packageArch, err := getOSArchName(currentPlatform, pkgType) + if err != nil { + log.Printf("Skipping arch %v for package type %v: %v", currentPlatform.Arch(), pkgType, err) + continue + } + + agentPackageType := TarGz + if pkg.OS == "windows" { + agentPackageType = Zip + } + + agentPackageArch, err := getOSArchName(currentPlatform, agentPackageType) + if err != nil { + log.Printf("Skipping arch %v for package type %v: %v", currentPlatform.Arch(), pkgType, err) + continue + } + + agentPackageDrop, _ := os.LookupEnv("AGENT_DROP_PATH") + + spec := pkg.Spec.Clone() + spec.OS = currentPlatform.GOOS() + spec.Arch = packageArch + spec.Snapshot = Snapshot + spec.evalContext = map[string]interface{}{ + "GOOS": currentPlatform.GOOS(), + "GOARCH": currentPlatform.GOARCH(), + "GOARM": currentPlatform.GOARM(), + "Platform": currentPlatform, + "AgentArchName": agentPackageArch, + "PackageType": pkgType.String(), + "BinaryExt": binaryExtension(currentPlatform.GOOS()), + "AgentDropPath": agentPackageDrop, + } + + spec.packageDir, err = pkgType.PackagingDir(packageStagingDir, currentPlatform, spec) + if err != nil { + log.Printf("Skipping arch %v for package type %v: %v", currentPlatform.Arch(), pkgType, err) + continue + } + + spec = spec.Evaluate() + + tasks = append(tasks, packageBuilder{currentPlatform, spec, pkgType}.Build) + } + } + + Parallel(tasks...) + return nil +} + type packageBuilder struct { Platform BuildPlatform Spec PackageSpec diff --git a/dev-tools/mage/pkgspecs.go b/dev-tools/mage/pkgspecs.go index 558b8e37ae..b656c7cb52 100644 --- a/dev-tools/mage/pkgspecs.go +++ b/dev-tools/mage/pkgspecs.go @@ -68,6 +68,12 @@ func UseElasticBeatOSSPackaging() { MustUsePackaging("elastic_beat_oss", packageSpecFile) } +// UseElasticBeatOSSPackaging configures the package target to build OSS +// packages. +func UseElasticNativeBeatOSSPackaging() { + MustUsePackaging("elastic_native_beat_oss", packageSpecFile) +} + // UseElasticBeatXPackPackaging configures the package target to build Elastic // licensed (X-Pack) packages. func UseElasticBeatXPackPackaging() { diff --git a/dev-tools/packaging/packages.yml b/dev-tools/packaging/packages.yml index ce48eb2209..7172b9af01 100644 --- a/dev-tools/packaging/packages.yml +++ b/dev-tools/packaging/packages.yml @@ -352,6 +352,53 @@ shared: template: '{{ elastic_beats_dir }}/dev-tools/packaging/templates/{{.PackageType}}/init.sh.tmpl' mode: 0755 + # Deb/RPM native spec for community beats. + - &deb_rpm_native_spec + <<: *common + post_install_script: '{{ elastic_beats_dir }}/dev-tools/packaging/files/linux/systemd-daemon-reload.sh' + files: + /usr/share/{{.BeatName}}/bin/{{.BeatName}}{{.BinaryExt}}: + source: build/golang-nativebuild/{{.BeatName}}-{{.GOOS}}-{{.Platform.Arch}}{{.BinaryExt}} + mode: 0755 + /etc/{{.BeatName}}/fields.yml: + source: fields.yml + mode: 0644 + /usr/share/{{.BeatName}}/LICENSE.txt: + source: '{{ repo.RootDir }}/LICENSE.txt' + mode: 0644 + /usr/share/{{.BeatName}}/NOTICE.txt: + source: '{{ repo.RootDir }}/NOTICE.txt' + mode: 0644 + /usr/share/{{.BeatName}}/README.md: + template: '{{ elastic_beats_dir }}/dev-tools/packaging/templates/common/README.md.tmpl' + mode: 0644 + /usr/share/{{.BeatName}}/.build_hash.txt: + content: > + {{ commit }} + mode: 0644 + /etc/{{.BeatName}}/{{.BeatName}}.reference.yml: + source: '{{.BeatName}}.reference.yml' + mode: 0644 + /etc/{{.BeatName}}/{{.BeatName}}.yml: + source: '{{.BeatName}}.yml' + mode: 0600 + config: true + /usr/share/{{.BeatName}}/kibana: + source: _meta/kibana.generated + mode: 0644 + /usr/share/{{.BeatName}}/bin/{{.BeatName}}-god: + source: build/golang-nativebuild/god-{{.GOOS}}-{{.Platform.Arch}} + mode: 0755 + /usr/bin/{{.BeatName}}: + template: '{{ elastic_beats_dir }}/dev-tools/packaging/templates/linux/beatname.sh.tmpl' + mode: 0755 + /lib/systemd/system/{{.BeatServiceName}}.service: + template: '{{ elastic_beats_dir }}/dev-tools/packaging/templates/linux/systemd.unit.tmpl' + mode: 0644 + /etc/init.d/{{.BeatServiceName}}: + template: '{{ elastic_beats_dir }}/dev-tools/packaging/templates/{{.PackageType}}/init.sh.tmpl' + mode: 0755 + # MacOS pkg spec for community beats. - &macos_beat_pkg_spec <<: *common @@ -428,12 +475,49 @@ shared: source: _meta/kibana.generated mode: 0644 + - &binary_native_files + '{{.BeatName}}{{.BinaryExt}}': + source: build/golang-nativebuild/{{.BeatName}}-{{.GOOS}}-{{.Platform.Arch}}{{.BinaryExt}} + mode: 0755 + fields.yml: + source: fields.yml + mode: 0644 + LICENSE.txt: + source: '{{ repo.RootDir }}/LICENSE.txt' + mode: 0644 + NOTICE.txt: + source: '{{ repo.RootDir }}/NOTICE.txt' + mode: 0644 + README.md: + template: '{{ elastic_beats_dir }}/dev-tools/packaging/templates/common/README.md.tmpl' + mode: 0644 + .build_hash.txt: + content: > + {{ commit }} + mode: 0644 + '{{.BeatName}}.reference.yml': + source: '{{.BeatName}}.reference.yml' + mode: 0644 + '{{.BeatName}}.yml': + source: '{{.BeatName}}.yml' + mode: 0600 + config: true + kibana: + source: _meta/kibana.generated + mode: 0644 + # Binary package spec (tar.gz for linux/darwin) for community beats. - &binary_spec <<: *common files: <<: *binary_files + # Binary package spec (tar.gz for linux/darwin) for community beats. + - &binary_native_spec + <<: *common + files: + <<: *binary_native_files + # Binary package spec (zip for windows) for community beats. - &windows_binary_spec <<: *common @@ -459,6 +543,19 @@ shared: mode: 0600 config: true + - &docker_native_spec + <<: *binary_native_spec + extra_vars: + from: 'centos:7' + buildFrom: 'centos:7' + user: '{{ .BeatName }}' + linux_capabilities: '' + files: + '{{.BeatName}}.yml': + source: '{{.BeatName}}.docker.yml' + mode: 0600 + config: true + - &docker_ubi_spec extra_vars: image_name: '{{.BeatName}}-ubi8' @@ -591,6 +688,53 @@ specs: <<: *apache_license_for_binaries name: '{{.BeatName}}-oss' + + # Elastic Native Beat with Apache License (OSS) and binary taken the current + # directory. + elastic_native_beat_oss: + - os: windows + types: [zip] + spec: + <<: *windows_binary_spec + <<: *apache_license_for_binaries + name: '{{.BeatName}}-oss' + + - os: darwin + types: [tgz] + spec: + <<: *binary_spec + <<: *apache_license_for_binaries + name: '{{.BeatName}}-oss' + + - os: darwin + types: [dmg] + spec: + <<: *macos_beat_pkg_spec + <<: *apache_license_for_macos_pkg + name: '{{.BeatName}}-oss' + + - os: linux + types: [tgz] + spec: + <<: *binary_native_spec + <<: *apache_license_for_binaries + name: '{{.BeatName}}-oss' + + - os: linux + types: [deb, rpm] + spec: + <<: *deb_rpm_native_spec + <<: *apache_license_for_deb_rpm + name: '{{.BeatName}}-oss' + + - os: linux + types: [docker] + spec: + <<: *docker_native_spec + <<: *elastic_docker_spec + <<: *apache_license_for_binaries + name: '{{.BeatName}}-oss' + # Elastic Beat with Elastic License and binary taken the current directory. elastic_beat_xpack: ### diff --git a/filebeat/magefile.go b/filebeat/magefile.go index 9b90cc68d9..1ce395eeec 100644 --- a/filebeat/magefile.go +++ b/filebeat/magefile.go @@ -15,6 +15,7 @@ // specific language governing permissions and limitations // under the License. +//go:build mage // +build mage package main @@ -75,6 +76,11 @@ func GolangCrossBuild() error { return devtools.GolangCrossBuild(devtools.DefaultGolangCrossBuildArgs()) } +// GolangNativeBuild build the Beat binary natively; +func NativeBuild() error { + return devtools.GolangNativeBuild(devtools.DefaultGolangNativeBuildArgs()) +} + // BuildGoDaemon builds the go-daemon binary (use crossBuildGoDaemon). func BuildGoDaemon() error { return devtools.BuildGoDaemon() @@ -90,6 +96,11 @@ func CrossBuildGoDaemon() error { return devtools.CrossBuildGoDaemon() } +// NativeBuildGoDaemon builds the go-daemon binary. +func NativeBuildGoDaemon() error { + return devtools.NativeBuildGoDaemon() +} + // Package packages the Beat for distribution. // Use SNAPSHOT=true to build snapshots. // Use PLATFORMS to control the target platforms. @@ -107,6 +118,20 @@ func Package() { mg.SerialDeps(devtools.Package, TestPackages) } +// Use VERSION_QUALIFIER to control the version qualifier. +func NativePackage() { + start := time.Now() + defer func() { fmt.Println("package ran for", time.Since(start)) }() + + devtools.UseElasticNativeBeatOSSPackaging() + devtools.PackageKibanaDashboardsFromBuildDir() + filebeat.CustomizePackaging() + + mg.Deps(Update) + mg.Deps(NativeBuild, NativeBuildGoDaemon) + mg.SerialDeps(devtools.NativePackage, TestPackages) +} + // TestPackages tests the generated packages (i.e. file modes, owners, groups). func TestPackages() error { return devtools.TestPackages(devtools.WithModules(), devtools.WithModulesD()) ```

Edit: It packaged! The package test also worked!

RedstoneWizard08 commented 1 year ago

Also, while untested, the Wazuh indexer and dashboard package as DEBs successfully with this patch:

Show patch content ```patch diff --git a/.gitignore b/.gitignore index 822b7b01..6bbbbe55 100644 --- a/.gitignore +++ b/.gitignore @@ -21,6 +21,8 @@ ova/Config_files/filebeat.yml .java stack/dashboard/base/output stack/indexer/base/output +stack/dashboard/tmp +stack/indexer/tmp .cache wpk/versions solaris/packer/builds/ diff --git a/VERSION b/VERSION index 6016e8ad..cbe06cdb 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -4.6.0 +4.4.4 diff --git a/stack/dashboard/base/docker/Dockerfile b/stack/dashboard/base/docker/Dockerfile index 184d894e..90601fa7 100644 --- a/stack/dashboard/base/docker/Dockerfile +++ b/stack/dashboard/base/docker/Dockerfile @@ -23,7 +23,7 @@ RUN yum install -y \ RUN git clone https://github.com/google/brotli.git -RUN cd brotli && chmod +x ./bootstrap && ./bootstrap && ./configure --prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib64/brotli --libdir=/usr/lib64/brotli --datarootdir=/usr/share --mandir=/usr/share/man/man1 --docdir=/usr/share/doc \ +RUN cd brotli && chmod +x ./bootstrap && ./bootstrap && ./configure --prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib/brotli --libdir=/usr/lib/brotli --datarootdir=/usr/share --mandir=/usr/share/man/man1 --docdir=/usr/share/doc \ && make && make install # Add the scripts to build the RPM package diff --git a/stack/dashboard/base/generate_base.sh b/stack/dashboard/base/generate_base.sh index e1a665e8..f81bd8e0 100755 --- a/stack/dashboard/base/generate_base.sh +++ b/stack/dashboard/base/generate_base.sh @@ -73,6 +73,7 @@ help() { echo "Usage: $0 [OPTIONS]" echo echo " --app-url [Optional] Set the repository from where the Wazuh plugin should be downloaded. By default, will be used pre-release." + echo " -a, --arch [Optional] Set the architecture to build for. By default ${architecture}" echo " -s, --store [Optional] Set the destination path of package. By default, an output folder will be created." echo " --reference [Optional] wazuh-packages branch or tag" echo " --future [Optional] Build test future package 99.99.0 Used for development purposes." @@ -107,6 +108,14 @@ main() { help 1 fi ;; + "-a"|"--arch") + if [ -n "${2}" ]; then + architecture="${2}" + shift 2 + else + help 1 + fi + ;; "--reference") if [ -n "${2}" ]; then reference="${2}" diff --git a/stack/dashboard/deb/build_package.sh b/stack/dashboard/deb/build_package.sh index e75e9aa1..63d36a3a 100755 --- a/stack/dashboard/deb/build_package.sh +++ b/stack/dashboard/deb/build_package.sh @@ -14,11 +14,12 @@ outdir="${current_path}/output" revision="1" build_docker="yes" deb_amd64_builder="deb_dashboard_builder_amd64" +deb_arm64_builder="deb_dashboard_builder_arm64" deb_builder_dockerfile="${current_path}/docker" future="no" base_cmd="" -url="" -build_base="yes" +base="s3" +base_path="${current_path}/../base/output" trap ctrl_c INT @@ -46,20 +47,6 @@ build_deb() { # Copy the necessary files cp ${current_path}/builder.sh ${dockerfile_path} - if [ "${build_base}" == "yes" ];then - # Base generation - if [ "${future}" == "yes" ];then - base_cmd+="--future " - fi - if [ "${reference}" ];then - base_cmd+="--reference ${reference}" - fi - if [ "${url}" ];then - base_cmd+="--app-url ${url}" - fi - ../base/generate_base.sh -s ${outdir} -r ${revision} ${base_cmd} - fi - # Build the Docker image if [[ ${build_docker} == "yes" ]]; then docker build -t ${container_name} ${dockerfile_path} || return 1 @@ -67,15 +54,19 @@ build_deb() { # Build the Debian package with a Docker container volumes="-v ${outdir}/:/tmp:Z" - if [ "${reference}" ];then + if [ "${reference}" ]; then docker run -t --rm ${volumes} \ ${container_name} ${architecture} ${revision} \ - ${future} ${url} ${reference} || return 1 + ${future} ${base} ${architecture_base} ${reference} || return 1 else + if [ "${base}" = "local" ];then + volumes="${volumes} -v ${base_path}:/root/output:Z" + fi + docker run -t --rm ${volumes} \ -v ${current_path}/../../..:/root:Z \ ${container_name} ${architecture} ${revision} \ - ${future} ${url} || return 1 + ${future} ${base} ${architecture_base} || return 1 fi echo "Package $(ls -Art ${outdir} | tail -n 1) added to ${outdir}." @@ -86,14 +77,22 @@ build_deb() { build() { build_name="" file_path="" + if [ "${architecture}" = "x86_64" ] || [ "${architecture}" = "amd64" ]; then architecture="amd64" + architecture_base="x64" build_name="${deb_amd64_builder}" file_path="${deb_builder_dockerfile}/${architecture}" + elif [ "${architecture}" = "arm64" ] || [ "${architecture}" = "aarch64" ]; then + architecture="arm64" + architecture_base="arm64" + build_name="${deb_arm64_builder}" + file_path="${deb_builder_dockerfile}/${architecture}" else - echo "Invalid architecture. Choose: amd64 (x86_64 is accepted too)" + echo "Invalid architecture. Choose: amd64 | arm64 | x86_64 | aarch64" return 1 fi + build_deb ${build_name} ${file_path} || return 1 return 0 @@ -104,13 +103,13 @@ help() { echo "Usage: $0 [OPTIONS]" echo echo " -a, --architecture [Optional] Target architecture of the package [amd64]." - echo " --app-url [Optional] Set the repository from where the Wazuh plugin should be downloaded. By default, will be used pre-release." - echo " -b, --build-base [Optional] Build a new base or use a existing one. By default, yes." echo " -r, --revision [Optional] Package revision. By default: 1." echo " -s, --store [Optional] Set the destination path of package. By default, an output folder will be created." echo " --reference [Optional] wazuh-packages branch to download SPECs, not used by default." echo " --dont-build-docker [Optional] Locally built docker image will be used instead of generating a new one." echo " --future [Optional] Build test future package 99.99.0 Used for development purposes." + echo " --base [Optional] Base file location, can be \"local\" or \"s3\". Default: s3" + echo " --base-path [Optional] If base is local, you can indicate the full path where the base is located. Default: stack/dashboard/base/output" echo " -h, --help Show this help." echo exit $1 @@ -132,22 +131,6 @@ main() { help 1 fi ;; - "--app-url") - if [ -n "$2" ]; then - repository="$2" - shift 2 - else - help 1 - fi - ;; - "-b"|"--build-base") - if [ -n "${2}" ]; then - build_base="${2}" - shift 2 - else - help 1 - fi - ;; "-r"|"--revision") if [ -n "${2}" ]; then revision="${2}" @@ -172,6 +155,22 @@ main() { future="yes" shift 1 ;; + "--base") + if [ -n "${2}" ]; then + base="${2}" + shift 2 + else + help 1 + fi + ;; + "--base-path") + if [ -n "${2}" ]; then + base_path="${2}" + shift 2 + else + help 1 + fi + ;; "-s"|"--store") if [ -n "${2}" ]; then outdir="${2}" diff --git a/stack/dashboard/deb/builder.sh b/stack/dashboard/deb/builder.sh index c3fd6107..6c095518 100755 --- a/stack/dashboard/deb/builder.sh +++ b/stack/dashboard/deb/builder.sh @@ -14,8 +14,9 @@ target="wazuh-dashboard" architecture=$1 revision=$2 future=$3 -repository=$4 -reference=$5 +base_location=$4 +architecture_base=$5 +reference=$6 directory_base="/usr/share/wazuh-dashboard" if [ -z "${revision}" ]; then @@ -32,21 +33,6 @@ else fi fi -if [ "${repository}" ];then - valid_url='(https?|ftp|file)://[-[:alnum:]\+&@#/%?=~_|!:,.;]*[-[:alnum:]\+&@#/%=~_|]' - if [[ $repository =~ $valid_url ]];then - url="${repository}" - if ! curl --output /dev/null --silent --head --fail "${url}"; then - echo "The given URL to download the Wazuh plugin zip does not exist: ${url}" - exit 1 - fi - else - url="https://packages-dev.wazuh.com/${repository}/ui/dashboard/wazuh-${version}-${revision}.zip" - fi -else - url="https://packages-dev.wazuh.com/pre-release/ui/dashboard/wazuh-${version}-${revision}.zip" -fi - # Build directories build_dir=/build pkg_name="${target}-${version}" @@ -64,13 +50,13 @@ else cp -r /root/stack/dashboard/deb/debian/* ${source_dir}/debian/ fi - # Generating directory structure to build the .deb package cd ${build_dir}/${target} && tar -czf ${pkg_name}.orig.tar.gz "${pkg_name}" # Configure the package with the different parameters sed -i "s:VERSION:${version}:g" ${source_dir}/debian/changelog sed -i "s:RELEASE:${revision}:g" ${source_dir}/debian/changelog +sed -i "s:ARCHITECTURE:${architecture_base}:g" ${source_dir}/debian/rules sed -i "s:export INSTALLATION_DIR=.*:export INSTALLATION_DIR=${directory_base}:g" ${source_dir}/debian/rules # Installing build dependencies @@ -78,7 +64,7 @@ cd ${source_dir} mk-build-deps -ir -t "apt-get -o Debug::pkgProblemResolver=yes -y" # Build package -debuild --no-lintian -eINSTALLATION_DIR="${directory_base}" -eVERSION="${version}" -eREVISION="${revision}" -eURL="${url}" -b -uc -us +debuild --no-lintian -eINSTALLATION_DIR="${directory_base}" -eBASE="${base_location}" -eBASE_VERSION="${version}" -eBASE_REVISION="${revision}" -b -uc -us deb_file="${target}_${version}-${revision}_${architecture}.deb" diff --git a/stack/dashboard/deb/debian/rules b/stack/dashboard/deb/debian/rules index 22f2c14c..3e923eab 100644 --- a/stack/dashboard/deb/debian/rules +++ b/stack/dashboard/deb/debian/rules @@ -28,9 +28,10 @@ export TARGET_DIR=${CURDIR}/debian/wazuh-dashboard export NAME=wazuh-dashboard export CONFIG_DIR=/etc/$(NAME) export INSTALLATION_DIR=$(INSTALLATION_DIR) +export ARCH=ARCHITECTURE export USER=$(NAME) export GROUP=$(NAME) -export DASHBOARD_FILE=wazuh-dashboard-base-$(VERSION)-$(REVISION)-linux-x64.tar.xz +export DASHBOARD_FILE=wazuh-dashboard-base-$(BASE_VERSION)-$(BASE_REVISION)-linux-$(ARCH).tar.xz # ----------------------------------------------------------------------------- @@ -52,7 +53,11 @@ override_dh_auto_install: # ----------------------------------------------------------------------------- override_dh_install: - cp /tmp/$(DASHBOARD_FILE) ./ + if [ "$(BASE)" = "s3" ]; then \ + curl -kOL https://packages-dev.wazuh.com/stack/dashboard/base/$(DASHBOARD_FILE) ;\ + else \ + cp /root/output/$(DASHBOARD_FILE) ./ ;\ + fi groupadd $(GROUP) useradd -g $(GROUP) $(USER) @@ -79,11 +84,7 @@ override_dh_install: chown -R $(USER):$(GROUP) $(TARGET_DIR)$(INSTALLATION_DIR) chown -R $(USER):$(GROUP) $(TARGET_DIR)$(CONFIG_DIR) - if [ "$(VERSION)" = "99.99.0" ]; then \ - runuser $(USER) --shell="/bin/bash" --command="$(TARGET_DIR)$(INSTALLATION_DIR)/bin/opensearch-dashboards-plugin install https://packages-dev.wazuh.com/futures/ui/dashboard/wazuh-99.99.0-$(REVISION).zip" ;\ - else \ - runuser $(USER) --shell="/bin/bash" --command="$(TARGET_DIR)$(INSTALLATION_DIR)/bin/opensearch-dashboards-plugin install $(URL)" ;\ - fi + runuser $(USER) --shell="/bin/bash" --command="$(TARGET_DIR)$(INSTALLATION_DIR)/bin/opensearch-dashboards-plugin install https://packages-dev.wazuh.com/pre-release/ui/dashboard/wazuh-$(BASE_VERSION)-${BASE_REVISION}.zip" ;\ find $(TARGET_DIR)$(INSTALLATION_DIR)/plugins/wazuh/ -exec chown $(USER):$(GROUP) {} \; diff --git a/stack/indexer/base/generate_base.sh b/stack/indexer/base/generate_base.sh index 20a224e9..914a2955 100755 --- a/stack/indexer/base/generate_base.sh +++ b/stack/indexer/base/generate_base.sh @@ -66,7 +66,8 @@ help() { echo echo "Usage: $0 [OPTIONS]" echo - echo " -s, --store [Optional] Set the destination path of package. By default, an output folder will be created." + echo " -a, --arch [Optional] Set the architecture to build for. By default ${architecture}" + echo " -s, --store [Optional] Set the destination path of package. By default, an output folder will be created." echo " --reference [Optional] wazuh-packages branch or tag" echo " --future [Optional] Build test future package 99.99.0 Used for development purposes." echo " -r, --revision [Optional] Package revision. By default ${revision}" @@ -84,6 +85,14 @@ main() { "-h"|"--help") help 0 ;; + "-a"|"--arch") + if [ -n "${2}" ]; then + architecture="${2}" + shift 2 + else + help 1 + fi + ;; "-s"|"--store") if [ -n "${2}" ]; then outdir="${2}" diff --git a/stack/indexer/deb/build_package.sh b/stack/indexer/deb/build_package.sh index b5032d07..ef3e6760 100755 --- a/stack/indexer/deb/build_package.sh +++ b/stack/indexer/deb/build_package.sh @@ -14,10 +14,11 @@ outdir="${current_path}/output" revision="1" build_docker="yes" deb_amd64_builder="deb_indexer_builder_amd64" +deb_arm64_builder="deb_indexer_builder_arm64" deb_builder_dockerfile="${current_path}/docker" future="no" -base_cmd="" -build_base="yes" +base="s3" +base_path="${current_path}/../base/output" trap ctrl_c INT @@ -41,34 +42,26 @@ build_deb() { # Copy the necessary files cp ${current_path}/builder.sh ${dockerfile_path} - if [ "${build_base}" == "yes" ];then - # Base generation - if [ "${future}" == "yes" ];then - base_cmd+="--future " - fi - if [ "${reference}" ];then - base_cmd+="--reference ${reference}" - fi - ../base/generate_base.sh -s ${outdir} -r ${revision} ${base_cmd} - fi - # Build the Docker image if [[ ${build_docker} == "yes" ]]; then docker build -t ${container_name} ${dockerfile_path} || return 1 fi - # Build the Debian package with a Docker container volumes="-v ${outdir}/:/tmp:Z" if [ "${reference}" ];then docker run -t --rm ${volumes} \ ${container_name} ${architecture} ${revision} \ - ${future} ${reference} || return 1 + ${future} ${base} ${architecture_base} ${reference} || return 1 else + if [ "${base}" = "local" ];then + volumes="${volumes} -v ${base_path}:/root/output:Z" + fi + docker run -t --rm ${volumes} \ -v ${current_path}/../../..:/root:Z \ ${container_name} ${architecture} \ - ${revision} ${future} || return 1 + ${revision} ${future} ${base} ${architecture_base} || return 1 fi echo "Package $(ls -Art ${outdir} | tail -n 1) added to ${outdir}." @@ -81,12 +74,19 @@ build() { file_path="" if [ "${architecture}" = "x86_64" ] || [ "${architecture}" = "amd64" ]; then architecture="amd64" + architecture_base="x64" build_name="${deb_amd64_builder}" file_path="${deb_builder_dockerfile}/${architecture}" + elif [ "${architecture}" = "arm64" ] || [ "${architecture}" = "aarch64" ]; then + architecture="arm64" + architecture_base="arm64" + build_name="${deb_arm64_builder}" + file_path="${deb_builder_dockerfile}/${architecture}" else - echo "Invalid architecture. Choose: amd64 (x86_64 is accepted too)" + echo "Invalid architecture. Choose: amd64 | arm64 | x86_64 | aarch64" return 1 fi + build_deb ${build_name} ${file_path} || return 1 return 0 @@ -96,13 +96,14 @@ help() { echo echo "Usage: $0 [OPTIONS]" echo - echo " -a, --architecture [Optional] Target architecture of the package [amd64]." - echo " -b, --build-base [Optional] Build a new base or use a existing one. By default, yes." + echo " -a, --architecture [Optional] Target architecture of the package [amd64, arm64]." echo " -r, --revision [Optional] Package revision. By default: 1." echo " -s, --store [Optional] Set the destination path of package. By default, an output folder will be created." echo " --reference [Optional] wazuh-packages branch to download SPECs, not used by default." echo " --dont-build-docker [Optional] Locally built docker image will be used instead of generating a new one." echo " --future [Optional] Build test future package 99.99.0 Used for development purposes." + echo " --base [Optional] Base file location, can be \"local\" or \"s3\". Default: s3" + echo " --base-path [Optional] If base is local, you can indicate the full path where the base is located. Default: stack/indexer/base/output" echo " -h, --help Show this help." echo exit $1 @@ -124,14 +125,6 @@ main() { help 1 fi ;; - "-b"|"--build-base") - if [ -n "${2}" ]; then - build_base="${2}" - shift 2 - else - help 1 - fi - ;; "-r"|"--revision") if [ -n "${2}" ]; then revision="${2}" @@ -156,6 +149,22 @@ main() { future="yes" shift 1 ;; + "--base") + if [ -n "${2}" ]; then + base="${2}" + shift 2 + else + help 1 + fi + ;; + "--base-path") + if [ -n "${2}" ]; then + base_path="${2}" + shift 2 + else + help 1 + fi + ;; "-s"|"--store") if [ -n "${2}" ]; then outdir="${2}" diff --git a/stack/indexer/deb/builder.sh b/stack/indexer/deb/builder.sh index fa349531..3fd5cb36 100755 --- a/stack/indexer/deb/builder.sh +++ b/stack/indexer/deb/builder.sh @@ -15,7 +15,9 @@ target="wazuh-indexer" architecture=$1 revision=$2 future=$3 -reference=$4 +base_location=$4 +architecture_base=$5 +reference=$6 directory_base="/usr/share/wazuh-indexer" if [ -z "${revision}" ]; then @@ -55,13 +57,14 @@ cd ${build_dir}/${target} && tar -czf ${pkg_name}.orig.tar.gz "${pkg_name}" # Configure the package with the different parameters sed -i "s:VERSION:${version}:g" ${source_dir}/debian/changelog sed -i "s:RELEASE:${revision}:g" ${source_dir}/debian/changelog +sed -i "s:ARCHITECTURE:${architecture_base}:g" ${source_dir}/debian/rules # Installing build dependencies cd ${source_dir} mk-build-deps -ir -t "apt-get -o Debug::pkgProblemResolver=yes -y" # Build package -debuild --no-lintian -eINSTALLATION_DIR="${directory_base}" -eVERSION="${version}" -eREVISION="${revision}" -b -uc -us +debuild --no-lintian -eINSTALLATION_DIR="${directory_base}" -eBASE="${base_location}" -eBASE_VERSION="${version}" -eBASE_REVISION="${revision}" -b -uc -us deb_file="${target}_${version}-${revision}_${architecture}.deb" diff --git a/stack/indexer/deb/debian/rules b/stack/indexer/deb/debian/rules index 9759cd19..71787f4b 100644 --- a/stack/indexer/deb/debian/rules +++ b/stack/indexer/deb/debian/rules @@ -36,7 +36,8 @@ export LIB_DIR=/var/lib/${NAME} export PID_DIR=/run/${NAME} export SYS_DIR=/usr/lib export BASE_DIR=${NAME}-* -export INDEXER_FILE=wazuh-indexer-base-$(VERSION)-$(REVISION)-linux-x64.tar.xz +export ARCH=ARCHITECTURE +export INDEXER_FILE=wazuh-indexer-base-$(BASE_VERSION)-$(BASE_REVISION)-linux-$(ARCH).tar.xz export REPO_DIR=/root/unattended_installer # ----------------------------------------------------------------------------- @@ -60,7 +61,12 @@ override_dh_auto_install: override_dh_install: rm -rf $(INSTALLATION_DIR)/ - cp /tmp/$(INDEXER_FILE) ./ + + if [ "$(BASE)" = "s3" ]; then \ + curl -kOL https://packages-dev.wazuh.com/stack/indexer/base/$(INDEXER_FILE) ;\ + else \ + cp /root/output/$(INDEXER_FILE) ./ ;\ + fi tar -xf $(INDEXER_FILE) @@ -1208,10 +1214,13 @@ override_dh_fixperms: chmod 640 $(TARGET_DIR)$(INSTALLATION_DIR)/jdk/jmods/jdk.incubator.vector.jmod chmod 640 $(TARGET_DIR)$(INSTALLATION_DIR)/jdk/jmods/jdk.jpackage.jmod chmod 640 $(TARGET_DIR)$(INSTALLATION_DIR)/jdk/jmods/jdk.random.jmod - chmod 640 $(TARGET_DIR)$(INSTALLATION_DIR)/jdk/lib/libjsvml.so chmod 640 $(TARGET_DIR)$(INSTALLATION_DIR)/jdk/lib/libsyslookup.so chmod 640 $(TARGET_DIR)$(INSTALLATION_DIR)/jdk/lib/security/blocked.certs + if [ -f "$(TARGET_DIR)$(INSTALLATION_DIR)/jdk/lib/libjsvml.so" ]; then \ + chmod 640 $(TARGET_DIR)$(INSTALLATION_DIR)/jdk/lib/libjsvml.so; \ + fi + # ----------------------------------------------------------------------------- override_dh_strip: ```

Note that RPMs do not package with this, as I have not modified the build scripts for them since I don't know how RPM packaging works xD

Update: Just a note that this was done on the master branch, but I had to change the contents of the VERSION file since the version listed does not exist on the packages server, causing the builds to throw an error.

Yoanndp commented 11 months ago

Hello, any update on that? Do you know when Wazuh will officially be supported for ARM devices?

fuzzysb commented 9 months ago

I am waiting for this, the indexer and dashboard is not working on ARM. i have a five node ARM cluster in my lab. I really do not want to introduce x64 nodes for Wazuh

RedstoneWizard08 commented 9 months ago

I might try to work on this again - I'll have to see if I have the time. I also wanted to say that the patches I made above are pretty outdated, likely won't work with current Wazuh code, and, well, are pretty garbage to be honest. I did all of this remotely with a web-based VS Code instance on an arm-based server, so I had many limitations. I'll try to make some better patches for the future, and even open a PR, but as far as filebeat may go, that might have to be an extra build step on Wazuh's part. I don't think that the version of filebeat Wazuh uses is still maintained. (I'm going off of memory, sorry if some of the details are wrong)

AlexRuiz7 commented 5 months ago

Currently, we have no plans on supporting ARM architectures on wazuh-dashboard and wazuh-indexer

We might revisit this in the future.