Open rvalente opened 8 years ago
I would like to have this, but I think we need to have proper FreeBSD pkg files. I have already done the work so that the tests run on FreeBSD (see the beats-freebsd job http://build-eu-00.elastic.co/view/Beats/).
It looks FreeBSD support is coming soon to FPM which will make building FreeBSD pkg files easier because we won't need FreeBSD to build them. This will fit nicely into the existing package build process which is based entirely on FPM.
If you would like to contribute, all of the packaging code is in https://github.com/elastic/beats/tree/master/dev-tools/packer. Basically, it cross-compiles the binaries inside docker then runs FPM to create packages. Once we have this we can start publishing FreeBSD pkg files for download. :smile:
Just out of curiosity, couldn't it be built on one of the Ubuntu build servers using GOOS=freebsd GOARCH=amd64
and then provide the tarball for a port maintainer to take and create a FreeBSD pkg for? Providing a native FreeBSD pkgng pkg wouldn't be required per se.
I think providing a 32 and 64-bit tarball would be a great start. pkg files could be added later.
... build servers using
GOOS=freebsd GOARCH=amd64
Yes, that is how we cross-compile. That alone works fine for Filebeat, but Packetbeat uses cgo so it will require a C cross-compiler to build from Linux. We do this for windows and mac now, for example, build.sh.
It looks like FPM now has FreeBSD packaging support. https://github.com/jordansissel/fpm/pull/1073
@andrewkroh FreeBSD doesn't require binary packages. Instead we have a "ports" system which builds from source. All that is needed is the correct Makefile which is checked into the FreeBSD svn tree.
FreeBSD infrastructure will create binary pkgs for those who want them.
Let me know if you need help. I know the FreeBSD ports system well enough, but I've never tried to build beats. I'm still using the logstash system for which there is a FreeBSD port.
Yes, if we can just integrate with ports we will be all set. Leave the pkg building to the superb FreeBSD infrastructure.
Might be relevant: https://github.com/elastic/beats/issues/974#issuecomment-197949686
beats are in the FreeBSD port collection (stuff that the system would download and compile) and also in the packages (prebuilt binary packages), even if the latter lags a little behind.
While I'm writing this, the port is at version beats-5.5.2
while the package is beats-5.4.3
.
Still, it would be nice to have "official" binaries just like they are available for linux.
FreeBSD ports are definitely lagging (6.4.2 at the moment, jumped from 6.2.4 previously). I'm seeing some errors and it would be nice to know if they were fixed in the latest upstream version.
Any update on this? 7.x is out, and ports are still stuck on 6.4.
Edit: Er...I meant 6.7
I am +1 this issue, i'd really like to see a Beats for 7.4 on BSD, would be great to run on a PFsense with suricata, and have that as integration to the SIEM App!
I am +1 this issue, i'd really like to see a Beats for 7.4 on BSD, would be great to run on a PFsense with suricata, and have that as integration to the SIEM App!
Agreed. I would also like to see Beats available on FreeBSD/PFsense.
Meanwhile, there is anyone was able to compile filebeats 7.4.x on freebsd ?
Is there already someone working on the compile for filebeat 7.4.* for Freebsd? i would like to see the futures of SIEM in pfsense
I am +1 this issue, i'd really like to see a Beats for 7.4 on BSD, would be great to run on a PFsense with suricata, and have that as integration to the SIEM App!
You don't necessarily need to run a beat on pfsense to get some of the data in the SIEM app. I'm using softflowd on my pfsense to send the netflow data to a filebeat with netflow module, running in a docker container on a Linux host and this netflow data is used in the SIEM app.
But yeah, for suricana it look like you should read the local file and for that it would be better to have filebeat run on pfsense.
I do run filebeat and metricbeat on my pfsense in version 7.3 (not the suricana module though) and it was pretty easy to compile.
Used a FreeBSD 11.2 kvm image from freebsd.org for that:
root@freebsd:~ # mkdir go
root@freebsd:~ # cd go
root@freebsd:~/go # go get github.com/elastic/beats
package github.com/elastic/beats: build constraints exclude all Go files in /root/go/src/github.com/elastic/beats
root@freebsd:~/go # cd src/github.com/elastic/beats/
root@freebsd:~/go/src/github.com/elastic/beats # git checkout v7.3.2
Checking out files: 100% (6951/6951), done.
Note: checking out 'v7.3.2'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 5b046c5a9 [7.3] Fix typos + add CHANGELOG (#13481) (#13510)
root@freebsd:~/go/src/github.com/elastic/beats # cd metricbeat/
root@freebsd:~/go/src/github.com/elastic/beats/metricbeat # gmake
go build -i -ldflags "-X github.com/elastic/beats/libbeat/version.buildTime=2019-11-21T21:04:50Z -X github.com/elastic/beats/libbeat/version.commit=5b046c5a97fe1e312f22d40a1f05365621aad621"
root@freebsd:~/go/src/github.com/elastic/beats/metricbeat # file metricbeat
metricbeat: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.2, FreeBSD-style, with debug_info, not stripped
root@freebsd:~/go/src/github.com/elastic/beats/metricbeat # ./metricbeat version
metricbeat version 7.3.2 (amd64), libbeat 7.3.2 [5b046c5a97fe1e312f22d40a1f05365621aad621 built 2019-11-21 21:04:50 +0000 UTC]
root@freebsd:~/go/src/github.com/elastic/beats/metricbeat # cd ../filebeat/
root@freebsd:~/go/src/github.com/elastic/beats/filebeat # gmake
go build -i -ldflags "-X github.com/elastic/beats/libbeat/version.buildTime=2019-11-21T21:05:58Z -X github.com/elastic/beats/libbeat/version.commit=5b046c5a97fe1e312f22d40a1f05365621aad621"
root@freebsd:~/go/src/github.com/elastic/beats/filebeat # file filebeat
filebeat: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.2, FreeBSD-style, with debug_info, not stripped
root@freebsd:~/go/src/github.com/elastic/beats/filebeat # ./filebeat version
filebeat version 7.3.2 (amd64), libbeat 7.3.2 [5b046c5a97fe1e312f22d40a1f05365621aad621 built 2019-11-21 21:05:58 +0000 UTC]
root@freebsd:~/go/src/github.com/elastic/beats/filebeat #
Then I just copied over the binaries to my pfsense box and replaced the 6.x binaries that I had installed from pgk.
For the suricana module, it might be ok to run 7.3 with that module and just ship it to a 7.4 Kibana/Elasticsearch. I haven't tested it, but I think the document structure should be the same and so it might just work with the SIEM app in 7.4.
7.4 fail to compile and errors out with:
root@freebsd:~/go/src/github.com/elastic/beats # git checkout v7.4.2
Checking out files: 100% (4035/4035), done.
Previous HEAD position was 5b046c5a9 [7.3] Fix typos + add CHANGELOG (#13481) (#13510)
HEAD is now at 150751563 [DOCS] Fixes link to role mapping (#13927) (#14117)
root@freebsd:~/go/src/github.com/elastic/beats # cd filebeat/
root@freebsd:~/go/src/github.com/elastic/beats/filebeat # gmake
go build -i -ldflags "-X github.com/elastic/beats/libbeat/version.buildTime=2019-11-21T21:08:43Z -X github.com/elastic/beats/libbeat/version.commit=15075156388b44390301f070960fd8aeac1c9712"
# github.com/elastic/beats/vendor/github.com/docker/docker/pkg/system
../vendor/github.com/docker/docker/pkg/system/mknod.go:12:19: cannot use dev (type int) as type uint64 in argument to unix.Mknod
gmake: *** [../libbeat/scripts/Makefile:107: filebeat] Error 2
root@freebsd:~/go/src/github.com/elastic/beats/filebeat # cd ../metricbeat/
root@freebsd:~/go/src/github.com/elastic/beats/metricbeat # gmake
go build -i -ldflags "-X github.com/elastic/beats/libbeat/version.buildTime=2019-11-21T21:11:59Z -X github.com/elastic/beats/libbeat/version.commit=15075156388b44390301f070960fd8aeac1c9712"
# github.com/elastic/beats/vendor/github.com/docker/docker/pkg/system
../vendor/github.com/docker/docker/pkg/system/mknod.go:12:19: cannot use dev (type int) as type uint64 in argument to unix.Mknod
gmake: *** [../libbeat/scripts/Makefile:107: metricbeat] Error 2
root@freebsd:~/go/src/github.com/elastic/beats/metricbeat #
But looks like this has been addressed already in 7.5, see https://github.com/elastic/beats/issues/13400 and https://github.com/elastic/beats/pull/13509
On a related note: I just had a Twitter discussion with some FreeBSD folks and they would be keen to help out with build and test infrastructures if needed (and our builds work I assume). Best contat is elastic at FreeBSD.org
I would like to suggest that once we have the development built fine one FreeBSD, is it possible to integrate CI services that supports FreeBSD in this repository? Here are some more information and examples: https://wiki.freebsd.org/HostedCI
Please also let me know anything I can help with this.
Maintain local patches to make something work on FreeBSD is a pain. I really would like to see all of ELK stack tested and built by upstream and not just in FreeBSD ports tree.
FWIW I successfully built filebeat and metricbeat 7.5.0
with the instructions from https://github.com/elastic/beats/issues/1034#issuecomment-557276051 yesterday.
i did too, i also wrote a small guide based on @jakommo instructions in #1034 https://blog.securitybits.io/2019/12/beats-7.5.0-on-pfsense-2.4.4/
edit: updated my link due to blog migration
To add to @jakommo 's install instructions, if you get build errors, it may be because you have a version of go from ports that's out-of-date. I removed the go
and go14
packages, then rm -rf /usr/local/go
, then reinstalled the packages.
Hi, I follow the @jakommo 's install instructions and got the error:
# cd filebeat/
# gmake
go build -ldflags "-X github.com/elastic/beats/libbeat/version.buildTime=2020-05-18T08:05:30Z -X github.com/elastic/beats/libbeat/version.commit=6a23e8f8f30f5001ba344e 4e54d8d9cb82cb107c"
# github.com/elastic/beats/vendor/github.com/DataDog/zstd
exec: "clang": executable file not found in $PATH
# github.com/elastic/beats/vendor/github.com/elastic/gosigar
exec: "clang": executable file not found in $PATH
gmake: *** [../libbeat/scripts/Makefile:121: filebeat] Error 2
This is v7.6.0 and v7.5.0 too. I very new here and don't know how to fix that. Anyone know solution please share with me. Thanks a lots
Same error as @tientmse62290 I went further and tried to fix the problems, and it is a nightmare of dependencies
pkg install clang
pkg install llvm60
the above doesn't work
Then I tried to compile llvm from source
git clone https://github.com/llvm/llvm-project.git
# About 1GB of source-code !!
# cmake not found !!
pkg install cmake
/tmp/llvm-project/build: cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm
# other set of problems
cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
-- The ASM compiler identification is unknown
-- Didn't find assembler
CMake Error at CMakeLists.txt:57 (project):
No CMAKE_C_COMPILER could be found.
I was fedup and thought, might go with Syslog !!
Hi @tientmse62290 and @getkub
I wrote a blogpost about compiling beats for FreeBSD a while back: https://blog.securitybits.io/2019/12/beats-7.5.0-on-pfsense-2.4.4/ I just compiled 7.8 with these instructions. based from @Jakommo
Just change:
git checkout 7.5.0
To:
git checkout 7.8
To get the most recent builds.
Hi @tientmse62290 and @getkub
I wrote a blogpost about compiling beats for FreeBSD a while back: https://blog.securitybits.io/2019/12/beats-7.5.0-on-pfsense-2.4.4/ I just compiled 7.8 with these instructions. based from @jakommo
Just change:
git checkout 7.5.0
To:git checkout 7.8
To get the most recent builds.
hi mate Yes I tried it and without checkout (as the recent one was 7.8) and hit the above problem while doing gmake may be you have an old copy of clang or c++ compilers within your system?
Cool, did you follow all the instructions, including the vagrant part? I usually run a debian based system, so i utilized vagrant to get me a clean FreeBSD to compile the beats on? By using vagrant, i try to remove the "it works on my computer" problem! :)
Cool, did you follow all the instructions, including the vagrant part? I usually run a debian based system, so i utilized vagrant to get me a clean FreeBSD to compile the beats on? By using vagrant, i try to remove the "it works on my computer" problem! :)
That's my mistake. I tried to do directly on the freeBSD server (in pfsense). seems they have customised it. (there are few issues with the Vagrant file as it needed a filesystem, but sorted out)
Vagrant.configure("2") do |config|
config.vm.box = "freebsd/FreeBSD-11.2-RELEASE"
config.vm.guest = :freebsd
config.ssh.shell = "sh"
config.vm.network "private_network", type: "dhcp"
config.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true
config.vm.box_version = "2018.06.22"
end
So it all worked perfectly with vagrant . cheers mate
No problem! Glad it worked! Happy logging! As soon as Github enables Github-actions FreeBSD Containers, that would be an alternative as well with a form and automatic build.
@Securitybits-io thanks for you blog article, found it while having this problem myself. So really appreciate. Compilation was no problem, but what came a little short was the installation/configuration part: "configure it according to the beats documentation." I am getting problems with the modules and stuff like:
ERROR fileset/modules.go:127 Not loading modules. Module directory not found: /usr/local/sbin/module INFO instance/beat.go:647 Home path: [/usr/local/sbin] Config path: [/usr/local/sbin] Data path: [/usr/local/sbin/data] Logs path: [/usr/local/sbin/logs]
Although I am handling it in the config file what I got from this article here: https://villekaaria.eu/2019/03/24/suricata-logs-to-logstash-with-filebeat-on-pfsense-2-4/
What else did you do on your pfsense system to have this filebeat running as a service WORKING ;) ?
I automated the build of FreeBSD AMD 64bit filebeat binaries for my usecase at work. I automated the build of the filebeat binaries for FreeBSD using GitHub actions. In theory, it should automatically build a freebsd/amd64 release for every version that is tagged upstream.
See https://github.com/omniitgmbh/beats/releases for the output.
I automated the build of FreeBSD AMD 64bit filebeat binaries for my usecase at work. I automated the build of the filebeat binaries for FreeBSD using GitHub actions. In theory, it should automatically build a freebsd/amd64 release for every version that is tagged upstream.
See omniitgmbh/beats/releases for the output.
@fadenb you mind sharing that GH Action?
@fadenb you mind sharing that GH Action?
@slmingol see https://github.com/omniitgmbh/beats/blob/oi-github-actions/.github/workflows/repo-sync.sh and https://github.com/omniitgmbh/beats/blob/oi-github-actions/.github/workflows/go.yml
The workflow fails if there is nothing to build. An additional condition should fix that (otherwise you will get a failure notification if there is nothing to build). Also note that it is no longer able to build older releases. I have not bothered to check out why as I only care about the new stuff.
Feel free to ping me if you have any questions.
I am +1 this issue, i'd really like to see a Beats for 7.4 on BSD, would be great to run on a PFsense with suricata, and have that as integration to the SIEM App!
We just shipped a new pfSense integration with Elastic Agent as part of the 7.15 release. It enables easy ingestion of pfSense logs and includes ECS mappings and dashboards. It may be of interest to some folks who have commented in this issue.
Hello @jamiehynds , thx for pointing out to the new pfSense integration.
Although, how pfSense will send event data to Elastic Cloud without an Elastic Agent available for FreeBSD OS? https://github.com/elastic/integrations/tree/master/packages/pfsense tnx🙏
For anyone looking to ship pfSense firewall logs to Elastic Cloud, our solution was:
Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)
Greetings Elastic Team-
Curious if Elastic Agent for FreeBSD is on the roadmap? Thanks for any rough ETA. We love the product!
For anyone looking to ship pfSense firewall logs to Elastic Cloud, our solution was:
- pfSense UI - send logs to intermediate Linux host via UDP port 9100
- Linux host - install Elastic Agent
- Elastic Cloud - Add pfSense integration to Agent policy followed this guide: https://www.elastic.co/guide/en/fleet/current/fleet-quick-start.html
are you using syslog-ng on the linux server, do you have any examples of your setup?
For anyone looking to ship pfSense firewall logs to Elastic Cloud, our solution was:
- pfSense UI - send logs to intermediate Linux host via UDP port 9100
- Linux host - install Elastic Agent
- Elastic Cloud - Add pfSense integration to Agent policy followed this guide: https://www.elastic.co/guide/en/fleet/current/fleet-quick-start.html
also curious
You do not need to run syslog-ng on a linux server, you run Elastic Agent with the PFsense integration (which will listen on a port) and then send the PFsense logs there.
For reference, these are the high level steps:
If you can't get it working, please post on the forums https://discuss.elastic.co/c/elastic-stack/elastic-agent/91 and provide details what failed
Hi! We just realized that we haven't looked into this issue in a while. We're sorry!
We're labeling this issue as Stale
to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1
.
Thank you for your contribution!
Stalebot...by any other name....
Yes, still an issue.
Yep, still an issue, native agent support for FreeBSD would be helpful
would be highly useful, i.e. for larger fileservers. we can ship audit logs from there but there would be many more interesting things that aren't possible now.
current state;
===> License APACHE20 accepted by the user
===> beats8-8.8.2_2 depends on file: /usr/local/sbin/pkg - found
===> beats8-8.8.2_2 depends on file: /usr/local/bin/go120 - found
===> beats8-8.8.2_2 depends on package: ca_root_nss>0 - found
=> go.mod doesn't seem to exist in /usr/ports/distfiles/go/sysutils_beats8/elastic-beats-v8.8.2_GH0.
=> Attempting to fetch https://raw.githubusercontent.com/elastic/beats/v8.8.2/go.mod
go.mod 18 kB 6795 kBps 00s
=> auditbeat-8.8.2-darwin-x86_64.tar.gz doesn't seem to exist in /usr/ports/distfiles/go/sysutils_beats8/elastic-beats-v8.8.2_GH0.
=> Attempting to fetch https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-8.8.2-darwin-x86_64.tar.gz
auditbeat-8.8.2-darwin-x86_64.tar.gz 27 MB 5477 kBps 05s
also keen for this - I have some zfs servers I'd like to keep an eye on..
Would it be possible to compile Filebeat for FreeBSD 32-bit and 64-bit along with the rest of the platforms? I know it is easy to compile myself, but adding that to my CI environment and running it on every new release seams redundant. Thank you.