search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
9.03k stars 467 forks source link

FreeBSD support #651

Closed sbz closed 3 years ago

sbz commented 3 years ago

We want to provide crowdsecurity/crowdsec to FreeBSD users.

This issue will help tracking the on-going FreeBSD effort needed.

TODO:

zorglube commented 3 years ago

Following the same intention, is this possible to provide an FreeNAS/TrueNAS plugin deployment ?

Obviously not the bouncer deployment buy the CrowdSec "server".

zorglube commented 3 years ago

Seems that replacing make by gmake allow tu build CS on BSD, also, since Docker is brocken on FreeBSD the cscli dashboard wont work.

sbz commented 3 years ago

Following the same intention, is this possible to provide an FreeNAS/TrueNAS plugin deployment ?

Obviously not the bouncer deployment buy the CrowdSec "server".

Hi @zorglube , thanks, it could be indeed a nice addition to provide, as well for the others FreeBSD derivatives such as pfSense but it's secondary for now. I like the overall idea 🤝

sbz commented 3 years ago

Seems that replacing make by gmake allow tu build CS on BSD, also, since Docker is brocken on FreeBSD the cscli dashboard wont work.

Yes, it does but there are other things to take care such as wizard, init services, prefix, etc. So far, it builds fine and it is on its way to be added upstream. You're correct for the dashboard, as it use a metabase docker image.

zorglube commented 3 years ago

Following the same intention, is this possible to provide an FreeNAS/TrueNAS plugin deployment ?

Obviously not the bouncer deployment buy the CrowdSec "server".

Hi @zorglube , thanks, it could be indeed a nice addition to provide, as well for the others FreeBSD derivatives such as pfSense but it's secondary for now. I like the overall idea 🤝

I'm currently working on making an TrueNAS plugin.

zorglube commented 3 years ago

Seems that replacing make by gmake allow tu build CS on BSD, also, since Docker is brocken on FreeBSD the cscli dashboard wont work.

Yes, it does but there are other things to take care such as wizard, init services, prefix, etc. So far, it builds fine and it is on its way to be added upstream. You're correct for the dashboard, as it use a metabase docker image.

I found the CrowdSec documentation on how to use the dashboard without Docker, that imply installing Java.

farfaaa commented 3 years ago

Hello,

I installed the package successfully But I encountered some problems launching the agent To make it work, I had to:

and now it works! So I guess it's planned to have postinstall and rc scripts :)

zorglube commented 3 years ago

I'm working on that, however any help is appreciate.

sbz commented 3 years ago

Hello,

I installed the package successfully But I encountered some problems launching the agent To make it work, I had to:

  • import the conf files from another linux server to '/usr/local/etc/crowdsec/'
  • update hub (with update hub)
  • register to central api (with capi register)
  • register to local api (with add machines --auto)

and now it works! So I guess it's planned to have postinstall and rc scripts :)

Hi @farfaaa, thanks for the feedbacks.

Currently, the "import of configuration" step is wanted, ultimately such configurations could either live inside the package or be deployed using configuration management systems. As it needs to have multiple files in /usr/local/etc/crowdsec/ directory, it's debatable if we should include them in the package or not.

It seems the agent update and register steps are done through the wizard currently, hub update here and register here.

Nothing prevent us from adding these steps into a rc script. I'll add a rc script for handling the service anyway.

sbz commented 3 years ago

In order to summarize for users, after installing the latest package, you will need the following commands and steps to bootstrap the agent on FreeBSD:

sudo crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml hub update
sudo crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml machines add --auto
sudo crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml machines list
sudo crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml capi register
sudo service crowdsec start
sudo service crowdsec status
zorglube commented 3 years ago

@sbz do you have any clue about the Metabase deployment ?

sbz commented 3 years ago

@zorglube Nope, I didn't look at it yet

zathras777 commented 3 years ago

I recently managed to use a full container image as a simpler install on FreeBSD for the HomeAutomation project using vm-bhyve Not sure if something similar could be done for the Metabase deployment but if so it might make it simpler. Quite how it could be incorporated into the install process I'm not sure.

https://blog.david-reid.com/home-assistant-on-freebsd/

zorglube commented 3 years ago

I was coming for update about BSD deployment and I just see @zathras777 post. Thank man !

sbz commented 3 years ago

@zorglube we made good progress and updates, we were focused on having the crowdsec firewall bouncer ready and working with FreeBSD packet filter pf.

The new package for that should be available soon 🤝 .

karolyi commented 3 years ago

@sbz just out of curiosity, how do you ban IPs using pf? do you add/remove IPs from pf tables? I'm looking forward to replacing fail2ban with this project on my FreeBSD server, if it becomes mature enough for FreeBSD.

sbz commented 3 years ago

Hi @alelaba, thanks for you comment.

Would you please share your /etc/pf.conf configuration and the output of the bouncer logs? The bouncer role does not block anything, it just updates the pf(4) tables IPs, then pf(4) should act according the decision and the config. Would you mind to run the following command?

sudo pfctl -sr
sbz commented 3 years ago

@sbz just out of curiosity, how do you ban IPs using pf? do you add/remove IPs from pf tables? I'm looking forward to replacing fail2ban with this project on my FreeBSD server, if it becomes mature enough for FreeBSD.

Hi @karolyi, exactly this is what it should do, dynamically add/remove IPs from pf tables. You can verify the current IPs on a given table using the following command:

sudo pfctl -t <table> -T show
sbz commented 3 years ago

FreeBSD Support is completed. I'm closing the issue.

You could refer the following to get started and setup Crowdsec on FreeBSD

Feel free to report any issues you encountered 🤝

odhiambo commented 3 years ago

In order to summarize for users, after installing the latest package, you will need the following commands and steps to bootstrap the agent on FreeBSD:

  • import the configuration into config directory /usr/local/etc/crowdsec
  • update hub
sudo crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml hub update
  • add your machine
sudo crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml machines add --auto
sudo crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml machines list
  • register to central API
sudo crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml capi register
  • Validate configuration
sudo service crowdsec configtest
  • Add the following /etc/rc.conf entries (only mandatory is really required)
crowdsec_enable="YES" # mandatory
crowdsec_config="/usr/local/etc/crowdsec/config.yaml" # optional
crowdsec_flags="-info" # optional
  • Start the agent
sudo service crowdsec start
sudo service crowdsec status

I found that importing the files from a Linux machine into FreeBSD and putting them in /usr/local/etc/crowdsec breaks the symlinks created in the parsers so the better way was:

  1. Install
  2. Blow away the installed /usr/local/etc/crowdsec
  3. Import the configs from a Linux machine and put them in /etc/crowdsec
  4. Create a symlink in /usr/local/etc (ln -s /etc/crowdsec /usr/local/etc/crowdsec
  5. I also had to import /var/lib/crowdsec to FreeBSD otherwise the agent wouldn't start

Below is what happened in my case:

root@gw:/usr/local/etc/crowdsec # crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml hub update INFO[15-08-2021 02:03:57 PM] Wrote new 126215 bytes index to /etc/crowdsec/hub/.index.json root@gw:/usr/local/etc/crowdsec # crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml machines add --auto WARN[15-08-2021 02:04:25 PM] can't load CAPI credentials from '/etc/crowdsec/online_api_credentials.yaml' (missing field) INFO[15-08-2021 02:04:25 PM] push and pull to crowdsec API disabled FATA[15-08-2021 02:04:25 PM] unable to create new database client: failed to create SQLite database file "/var/lib/crowdsec/data/crowdsec.db": open /var/lib/crowdsec/data/crowdsec.db: no such file or directory root@gw:/usr/local/etc/crowdsec # mkdir /var/lib/crowdsec root@gw:/usr/local/etc/crowdsec # crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml machines add --auto WARN[15-08-2021 02:04:52 PM] can't load CAPI credentials from '/etc/crowdsec/online_api_credentials.yaml' (missing field) INFO[15-08-2021 02:04:52 PM] push and pull to crowdsec API disabled FATA[15-08-2021 02:04:52 PM] unable to create new database client: failed to create SQLite database file "/var/lib/crowdsec/data/crowdsec.db": open /var/lib/crowdsec/data/crowdsec.db: no such file or directory root@gw:/usr/local/etc/crowdsec # rm -rf /var/lib//crowdsec root@gw:/usr/local/etc/crowdsec # cp -Rp /var/lib/crowdsec-/ /var/lib/crowdsec #Imported the directory from Linux root@gw:/usr/local/etc/crowdsec # crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml machines add --auto WARN[15-08-2021 02:05:36 PM] can't load CAPI credentials from '/etc/crowdsec/online_api_credentials.yaml' (missing field) INFO[15-08-2021 02:05:36 PM] push and pull to crowdsec API disabled INFO[15-08-2021 02:05:36 PM] Machine '9d6e6680ee6211e28c707446a094189b696pgxVDfKGdekVy' successfully added to the local API INFO[15-08-2021 02:05:36 PM] API credentials dumped to '/etc/crowdsec/local_api_credentials.yaml' root@gw:/usr/local/etc/crowdsec # crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml machines list WARN[15-08-2021 02:06:27 PM] can't load CAPI credentials from '/etc/crowdsec/online_api_credentials.yaml' (missing field) INFO[15-08-2021 02:06:27 PM] push and pull to crowdsec API disabled NAME IP ADDRESS LAST UPDATE STATUS VERSION 9d6e6680ee6211e28c707446a094189bghne9qdXgcLj1neG 2021-08-14T17:35:11+03:00 ✔️ 9d6e6680ee6211e28c707446a094189bq9GWKdqTGKu1BO1P 127.0.0.1 2021-08-14T17:53:44+03:00 ✔️ v1.1.1-freebsd- 9d6e6680ee6211e28c707446a094189b696pgxVDfKGdekVy 2021-08-15T14:05:36+03:00 ✔️

root@gw:/usr/local/etc/crowdsec # crowdsec-cli -c /usr/local/etc/crowdsec/config.yaml capi register WARN[15-08-2021 02:07:02 PM] can't load CAPI credentials from '/etc/crowdsec/online_api_credentials.yaml' (missing field) INFO[15-08-2021 02:07:02 PM] push and pull to crowdsec API disabled INFO[15-08-2021 02:07:04 PM] Successfully registered to Central API (CAPI) INFO[15-08-2021 02:07:04 PM] Central API credentials dumped to '/etc/crowdsec/online_api_credentials.yaml' WARN[15-08-2021 02:07:04 PM] Run 'sudo service crowdsec reload' for the new configuration to be effective. root@gw:/usr/local/etc/crowdsec # service crowdsec configtest Performing sanity check on crowdsec configuration root@gw:/usr/local/etc/crowdsec # service crowdsec status crowdsec is running as pid 65539.

Now that the agent is running, the next question is how to start the bouncer.

root@gw:/usr/local/etc/crowdsec # /usr/local/etc/rc.d/crowdsec_firewall start Starting crowdsec_firewall. root@gw:/usr/local/etc/crowdsec # service crowdsec_firewall status crowdsec_firewall is not running. root@gw:/usr/local/etc/crowdsec # less /var/log/crowdsec/crowdsec-firewall-bouncer.log time="15-08-2021 15:09:41" level=info msg="backend type : pf" time="15-08-2021 15:09:41" level=info msg="pf table clean-up : /sbin/pfctl -t crowdsec-blacklists -T flush" time="15-08-2021 15:09:41" level=info msg="Checking pf table: crowdsec-blacklists" time="15-08-2021 15:09:41" level=info msg="pf initiated for ipv4" time="15-08-2021 15:09:41" level=info msg="pf table clean-up : /sbin/pfctl -t crowdsec6-blacklists -T flush" time="15-08-2021 15:09:41" level=info msg="Checking pf table: crowdsec6-blacklists" time="15-08-2021 15:09:41" level=info msg="pf initiated for ipv6" time="15-08-2021 15:09:41" level=info msg="Processing new and deleted decisions . . ." time="15-08-2021 15:09:41" level=fatal msg="API error: access forbidden" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I went to https://doc.crowdsec.net/Crowdsec/v1/user_tutorial/crowdsec_firewall_on_freebsd/ and I found that I need to add a new bouncer instance. The command to do this is explained as: crowdsec-cli bouncers add --name freebsd-pf-bouncer

It seems that things have changed, such that the argument --name is no longer valid. So I did it with: (so some doco edit required, or?)

crowdsec-cli bouncers add freebsd-pf-bouncer

That gave me the API key and I was able to start the bouncer.

Now that I have everything running, could someone please help me with parsers for Exim and Dovecot?? Or the general integration procedure for the two with crowdesc??

odhiambo commented 3 years ago

I recently managed to use a full container image as a simpler install on FreeBSD for the HomeAutomation project using vm-bhyve Not sure if something similar could be done for the Metabase deployment but if so it might make it simpler. Quite how it could be incorporated into the install process I'm not sure.

https://blog.david-reid.com/home-assistant-on-freebsd/

@zathras777 - could you please share the bits you made for handling dovecot?

zathras777 commented 3 years ago

On Sun, 15 Aug 2021, 14:29 Odhiambo WASHINGTON, @.***> wrote:

I recently managed to use a full container image as a simpler install on FreeBSD for the HomeAutomation project using vm-bhyve Not sure if something similar could be done for the Metabase deployment but if so it might make it simpler. Quite how it could be incorporated into the install process I'm not sure.

https://blog.david-reid.com/home-assistant-on-freebsd/

@zathras777 https://github.com/zathras777 - could you please share the bits you made for handling dovecot?

—

I can do, but there is a plugin I. The Hub that may work for you? In my case I primarily catch auth failures from the auth service so it may not really help you.

Adding a service is pretty straightforward and I have a small app on GitHub that may help with checking things.

If you're still interested in my files let me know .

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/crowdsecurity/crowdsec/issues/651#issuecomment-899050641, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFD6NIFN5XGKY3GYMSM6PTT466KDANCNFSM4YGR7PNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

odhiambo commented 3 years ago

Hi,

I'm also just interested in the auth failures. Please share your files.

I'd love to know how to add my own too. I'd like to add Exim.

Thanks in advance

On Sun, Aug 15, 2021, 23:24 david reid @.***> wrote:

On Sun, 15 Aug 2021, 14:29 Odhiambo WASHINGTON, @.***> wrote:

I recently managed to use a full container image as a simpler install on FreeBSD for the HomeAutomation project using vm-bhyve Not sure if something similar could be done for the Metabase deployment but if so it might make it simpler. Quite how it could be incorporated into the install process I'm not sure.

https://blog.david-reid.com/home-assistant-on-freebsd/

@zathras777 https://github.com/zathras777 - could you please share the bits you made for handling dovecot?

—

I can do, but there is a plugin I. The Hub that may work for you? In my case I primarily catch auth failures from the auth service so it may not really help you.

Adding a service is pretty straightforward and I have a small app on GitHub that may help with checking things.

If you're still interested in my files let me know .

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/crowdsecurity/crowdsec/issues/651#issuecomment-899050641 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAFD6NIFN5XGKY3GYMSM6PTT466KDANCNFSM4YGR7PNQ

. Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675

or Android < https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/crowdsecurity/crowdsec/issues/651#issuecomment-899105247, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGK4XQZUBSBNQWNUF2T7UTT5APBRANCNFSM4YGR7PNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .