Support for *BSD - Githubissues

dguido commented 8 years ago

Particularly interested in FreeBSD, HardenedBSD, and OpenBSD.

jackivanov commented 8 years ago

Note about FreeBSD: We need to recompile the kernel with options:

options         IPSEC
options         IPSEC_NAT_T
device          crypto

dguido commented 7 years ago

$500 bounty! Submit a pull request and email dan@trailofbits.com to claim it. Partial solutions may be rewarded.

PolymathMonkey commented 7 years ago

So just to get this right. If i get this server running on OpenBSD I can make 500 bugs? Just want to know, so I will give it a try.

defunctio commented 7 years ago

@PolymathMonkey Just getting Algo to deploy properly on OpenBSD would probably be considered a partial solution but may still be awarded. I would think that it would be considered a full solution when you are able to deploy *BSD to at least one cloud provider just as you currently can with Ubuntu 16.04.

This means, ensuring available images on at least one provider with appropriate kernel options for deployment, etc. I'd be willing to throw some of the bounty rewards associated with my contributions towards a fully working solution on AWS if @dguido has no objections to that.

dguido commented 7 years ago

Yep! At minimum, it needs to be a port of the essential features: the common and VPN roles. You need to make a PR that integrates these features into the codebase. Either @gunph1ld or @defunctio need to review the PR to accept it. I'll consider rewarding partial solutions with partial credit.

Does DigitalOcean have hosted OpenBSD images? I know they have FreeBSD.

PolymathMonkey commented 7 years ago

Don't know if they have OpenBSD. I know at least AWS has obsd images (5.9) , but still have to take a look to get a overview of the scope of the Project. But in general if there are no short time frames to complete the bounty. I would give it a try, I love bsd and the idea of the challenge of implementing a cloud image plus get the algo code running..

defunctio commented 7 years ago

@PolymathMonkey I don't believe there is a time-frame for completion. As far as I know, these are just like any other bounties to incentivize advancing an OSS project but I don't work for ToB so don't hold me to that ;)

+1 for HardenedBSD

dguido commented 7 years ago

No timeframe! I think @gunph1ld had his eye on this one too though :-P

PolymathMonkey commented 7 years ago

I would only try to do the obsd stuff ^^. But I am always in for cooperating on projects depends on @gunph1ld . If he wants to to this on his own I would step to the aside, but I would enjoy working on some nice OSS Project with some other folks. But one way or another I first have to setup some dev/test environment to start with the bounty :D .

dguido commented 7 years ago

Yeah try it! Jack had his hands full with a dozen other issues. Give it a shot.

sean9999 commented 7 years ago

I'll give it a go on FreeNAS (FreeBSD)

lattera commented 7 years ago

I'm a bit busy with a few things at the moment, but I could set up a little test cluster on HardenedBSD. I've got 2-3 servers sitting around that I could deploy this on to. I'll keep you updated as to my progress. Might take a couple weeks, though.

defunctio commented 7 years ago

@lattera Much appreciated!

PolymathMonkey commented 7 years ago

:\ sorry guys have to cancel my participation in this project (@openbsd implementation). Because some other topic is consuming all my time right now

lattera commented 7 years ago

Update: I haven't forgotten about this. I should be able to start work on testing in early 2017. Got really sick twice in a row, preventing me from working on this.

jackivanov commented 7 years ago

FreeBSD:

[x] common role and pre tasks
[x] vpn role
[x] ssh_tunneling role
[x] dns_adblocking role
[x] update-users
[x] recompile the kernel while deploying
[ ] security role

~~- [ ] ? logging role ?~~ ~~- [ ] ? Algo prompts ?~~ ~~- [ ] proxy role~~

jackivanov commented 7 years ago

HardenedBSD:

[x] common role and pre tasks
[x] vpn role
[x] ssh_tunneling role
[x] dns_adblocking role
[x] update-users
[x] recompile the kernel while deploying
[ ] security role

~~- [ ] ? logging role ?~~ ~~- [ ] ? Algo prompts ?~~ ~~- [ ] proxy role~~

jackivanov commented 7 years ago

in addition:

[ ] firewall configuration
[ ] resource management (alternative for cgroups)

Hultner commented 7 years ago

I tried installing under FreeBSD 11.1-RELEASE-p1 however were unable to complete the installation, it seems to fail rebuilding the kernel.

When researching the problem I found a reference from strongSwan saying that recompilation for NAT_T isn't needed anymore in 11.1.

However IPSEC_NAT_T is not preset in the kern conftxt

# sysctl kern.conftxt | grep -iE "IPSEC|crypto"
options IPSEC
device  crypto

I also tried recompiling the 11.0 kernel manually with source from https://svn.freebsd.org/base/releng/11.0 using the following KERNCONF

include GENERIC
ident           GENERIC_IPsec

# Options for an IPsec enabled kernel
#options         IPSEC       #already included with GENERIC on FreeBSD11
#device          crypto       #already included with GENERIC on FreeBSD11
options         IPSEC_NAT_T

But neither this granted me any acces as it fails when compling nvme

In file included from /usr/src/sys/cam/nvme/nvme_all.h:32:
/usr/src/sys/dev/nvme/nvme.h:922:16: error: taking address of packed member 'cdw10' of class or structure 'nvme_command' may result in an
      unaligned pointer value [-Werror,-Waddress-of-packed-member]
        *(uint64_t *)&cmd->cdw10 = lba;
                      ^~~~~~~~~~
1 error generated.
*** Error code 1

Stop.
make[2]: stopped in /usr/obj/usr/src/sys/HULTBSD
*** Error code 1

Stop.
make[1]: stopped in /usr/src
*** Error code 1

Stop.
make: stopped in /usr/src

I have no previous experience with writing ansible scripts so I'm not sure how to modify the script to continue the installation without IPSEC_NAT_T, I suppose this could still work according to strongswan documentation.

I've attached the install log from ansible when trying to install below. This is a fresh install and I've only installed some basic tools such as tmux before trying the ansible deploy of algo.

PLAY [Configure the server] *********************************************************************************************************************************************

TASK [Gathering Facts] **************************************************************************************************************************************************
ok: [localhost]

TASK [Generate the SSH private key] *************************************************************************************************************************************
ok: [localhost]

TASK [Generate the SSH public key] **************************************************************************************************************************************
ok: [localhost]

TASK [Change mode for the SSH private key] ******************************************************************************************************************************
ok: [localhost]

TASK [Ensure the dynamic inventory exists] ******************************************************************************************************************************
ok: [localhost]

TASK [Ensure the local ssh directory is exist] **************************************************************************************************************************
skipping: [localhost]

TASK [Copy the algo ssh key to the local ssh directory] *****************************************************************************************************************
skipping: [localhost]

TASK [local : Add the instance to an inventory group] *******************************************************************************************************************
changed: [localhost]

TASK [local : Add the instance to an inventory group] *******************************************************************************************************************
skipping: [localhost]

TASK [local : set_fact] *************************************************************************************************************************************************
ok: [localhost]

TASK [local : Ensure the group local exists in the dynamic inventory file] **********************************************************************************************
ok: [localhost]

TASK [local : Populate the dynamic inventory] ***************************************************************************************************************************
ok: [localhost]

PLAY [Configure the server and install required software] ***************************************************************************************************************

TASK [Check the system] *************************************************************************************************************************************************
changed: [172.27.0.40]

TASK [Ubuntu | Install prerequisites] ***********************************************************************************************************************************
skipping: [172.27.0.40]

TASK [Ubuntu | Configure defaults] **************************************************************************************************************************************
skipping: [172.27.0.40]

TASK [FreeBSD / HardenedBSD | Install prerequisites] ********************************************************************************************************************
changed: [172.27.0.40]

TASK [FreeBSD / HardenedBSD | Configure defaults] ***********************************************************************************************************************
changed: [172.27.0.40]

TASK [set_fact] *********************************************************************************************************************************************************
ok: [172.27.0.40]

TASK [Gather Facts] *****************************************************************************************************************************************************
ok: [172.27.0.40]

TASK [Enable IPv6] ******************************************************************************************************************************************************
skipping: [172.27.0.40]

TASK [Generate password for the CA key] *********************************************************************************************************************************
changed: [172.27.0.40 -> localhost]

TASK [Generate p12 export password] *************************************************************************************************************************************
changed: [172.27.0.40 -> localhost]

TASK [Define password facts] ********************************************************************************************************************************************
ok: [172.27.0.40]

TASK [Define the commonName] ********************************************************************************************************************************************
ok: [172.27.0.40]

TASK [common : Loopback for services configured] ************************************************************************************************************************
skipping: [172.27.0.40]

TASK [common : Loopback included into the network config] ***************************************************************************************************************
skipping: [172.27.0.40]

TASK [common : set_fact] ************************************************************************************************************************************************
skipping: [172.27.0.40]

TASK [common : set_fact] ************************************************************************************************************************************************
ok: [172.27.0.40]

TASK [common : Loopback included into the rc config] ********************************************************************************************************************
ok: [172.27.0.40]

TASK [common : Enable the gateway features] *****************************************************************************************************************************
ok: [172.27.0.40] => (item={u'value': u'"YES"', u'param': u'firewall_enable'})
ok: [172.27.0.40] => (item={u'value': u'"open"', u'param': u'firewall_type'})
ok: [172.27.0.40] => (item={u'value': u'"YES"', u'param': u'gateway_enable'})
ok: [172.27.0.40] => (item={u'value': u'"YES"', u'param': u'natd_enable'})
ok: [172.27.0.40] => (item={u'value': u'"em0"', u'param': u'natd_interface'})
ok: [172.27.0.40] => (item={u'value': u'"-dynamic -m"', u'param': u'natd_flags'})

TASK [common : Install tools] *******************************************************************************************************************************************
ok: [172.27.0.40] => (item=git)
ok: [172.27.0.40] => (item=subversion)
ok: [172.27.0.40] => (item=screen)
ok: [172.27.0.40] => (item=coreutils)
ok: [172.27.0.40] => (item=openssl)
ok: [172.27.0.40] => (item=bash)
ok: [172.27.0.40] => (item=wget)

TASK [common : Sysctl tuning] *******************************************************************************************************************************************
ok: [172.27.0.40] => (item={u'item': u'net.inet.ip.forwarding', u'value': 1})
ok: [172.27.0.40] => (item={u'item': u'net.inet6.ip6.forwarding', u'value': 1})

TASK [vpn : Ensure that the strongswan group exist] *********************************************************************************************************************
ok: [172.27.0.40]

TASK [vpn : Ensure that the strongswan user exist] **********************************************************************************************************************
ok: [172.27.0.40]

TASK [vpn : set_fact] ***************************************************************************************************************************************************
skipping: [172.27.0.40]

TASK [vpn : Ubuntu | Install strongSwan] ********************************************************************************************************************************
skipping: [172.27.0.40]

TASK [vpn : Ubuntu | Enforcing ipsec with apparmor] *********************************************************************************************************************
skipping: [172.27.0.40] => (item=/usr/lib/ipsec/charon)
skipping: [172.27.0.40] => (item=/usr/lib/ipsec/lookip)
skipping: [172.27.0.40] => (item=/usr/lib/ipsec/stroke)

TASK [vpn : Ubuntu | Enable services] ***********************************************************************************************************************************
skipping: [172.27.0.40] => (item=apparmor)
skipping: [172.27.0.40] => (item=strongswan)
skipping: [172.27.0.40] => (item=netfilter-persistent)

TASK [vpn : Ubuntu | Ensure that the strongswan service directory exist] ************************************************************************************************
skipping: [172.27.0.40]

TASK [vpn : Ubuntu | Setup the cgroup limitations for the ipsec daemon] *************************************************************************************************
skipping: [172.27.0.40]

TASK [vpn : Iptables configured] ****************************************************************************************************************************************
skipping: [172.27.0.40] => (item={u'dest': u'/etc/iptables/rules.v4', u'src': u'rules.v4.j2'})                                                   

TASK [vpn : Iptables configured] ****************************************************************************************************************************************
skipping: [172.27.0.40] => (item={u'dest': u'/etc/iptables/rules.v6', u'src': u'rules.v6.j2'})                                                   

TASK [vpn : FreeBSD / HardenedBSD | Get the existing kernel parameters] *************************************************************************************************
changed: [172.27.0.40]

TASK [vpn : FreeBSD / HardenedBSD | Set the rebuild_needed fact] ********************************************************************************************************
skipping: [172.27.0.40] => (item=IPSEC)
ok: [172.27.0.40] => (item=IPSEC_NAT_T)
skipping: [172.27.0.40] => (item=crypto)

TASK [vpn : FreeBSD / HardenedBSD | Make the kernel config] *************************************************************************************************************
changed: [172.27.0.40]

TASK [vpn : FreeBSD / HardenedBSD | Ensure the all options are enabled] *************************************************************************************************
ok: [172.27.0.40] => (item=options      IPSEC)
changed: [172.27.0.40] => (item=options IPSEC_NAT_T)
ok: [172.27.0.40] => (item=device       crypto)

TASK [vpn : HardenedBSD | Determine the sources] ************************************************************************************************************************
skipping: [172.27.0.40]

TASK [vpn : FreeBSD | Determine the sources] ****************************************************************************************************************************
ok: [172.27.0.40]

TASK [vpn : FreeBSD / HardenedBSD | Increase the git postBuffer size] ***************************************************************************************************
ok: [172.27.0.40]

TASK [vpn : FreeBSD / HardenedBSD | Fetching the sources...] ************************************************************************************************************
changed: [172.27.0.40]

TASK [vpn : FreeBSD / HardenedBSD | Fetching the sources...] ************************************************************************************************************
FAILED - RETRYING: FreeBSD / HardenedBSD | Fetching the sources... (600 retries left).                                                           
ok: [172.27.0.40]

TASK [vpn : FreeBSD / HardenedBSD | The kernel is being built...] *******************************************************************************************************
changed: [172.27.0.40]

TASK [vpn : FreeBSD / HardenedBSD | The kernel is being built...] *******************************************************************************************************
fatal: [172.27.0.40]: FAILED! => {"ansible_job_id": "381194201046.27205", "attempts": 1, "changed": true, "cmd": "mv /tmp/IPSEC /usr/krnl_src/sys/amd64/conf && make buildkernel KERNCONF=IPSEC && make installkernel KERNCONF=IPSEC", "delta": "0:00:00.307364", "end": "2017-09-11 14:56:33.401883", "failed": true, "finished": 1, "rc": 1, "start": "2017-09-11 14:56:33.094519", "stderr": "make[1]: \"/usr/krnl_src/Makefile.inc1\" line 158: SYSTEM_COMPILER: Determined that CC=cc matches the source tree.  Not bootstrapping a cross-compiler.\nWARNING: duplicate option `NEW_PCIB' encountered.\nWARNING: duplicate option `GEOM_PART_MBR' encountered.\nWARNING: duplicate option `GEOM_PART_EBR_COMPAT' encountered.\nWARNING: duplicate option `GEOM_PART_EBR' encountered.\nWARNING: duplicate option `GEOM_PART_BSD' encountered.\nWARNING: duplicate option `DEV_ISA' encountered.\nWARNING: duplicate device `isa' encountered.\nWARNING: duplicate option `DEV_MEM' encountered.\nWARNING: duplicate device `mem' encountered.\nWARNING: duplicate option `DEV_IO' encountered.\nWARNING: duplicate device `io' encountered.\nWARNING: duplicate option `DEV_UART_NS8250' encountered.\nWARNING: duplicate device `uart_ns8250' encountered.\n/usr/krnl_src/sys/amd64/conf/IPSEC: unknown option \"IPSEC_NAT_T\"", "stderr_lines": ["make[1]: \"/usr/krnl_src/Makefile.inc1\" line 158: SYSTEM_COMPILER: Determined that CC=cc matches the source tree.  Not bootstrapping a cross-compiler.", "WARNING: duplicate option `NEW_PCIB' encountered.", "WARNING: duplicate option `GEOM_PART_MBR' encountered.", "WARNING: duplicate option `GEOM_PART_EBR_COMPAT' encountered.", "WARNING: duplicate option `GEOM_PART_EBR' encountered.", "WARNING: duplicate option `GEOM_PART_BSD' encountered.", "WARNING: duplicate option `DEV_ISA' encountered.", "WARNING: duplicate device `isa' encountered.", "WARNING: duplicate option `DEV_MEM' encountered.", "WARNING: duplicate device `mem' encountered.", "WARNING: duplicate option `DEV_IO' encountered.", "WARNING: duplicate device `io' encountered.", "WARNING: duplicate option `DEV_UART_NS8250' encountered.", "WARNING: duplicate device `uart_ns8250' encountered.", "/usr/krnl_src/sys/amd64/conf/IPSEC: unknown option \"IPSEC_NAT_T\""], "stdout": "\n--------------------------------------------------------------\n>>> Kernel build for IPSEC started on Mon Sep 11 14:56:33 UTC 2017\n--------------------------------------------------------------\n===> IPSEC\nmkdir -p /usr/obj/usr/krnl_src/sys\n\n--------------------------------------------------------------\n>>> stage 1: configuring the kernel\n--------------------------------------------------------------\ncd /usr/krnl_src/sys/amd64/conf;  PATH=/usr/obj/usr/krnl_src/tmp/legacy/usr/sbin:/usr/obj/usr/krnl_src/tmp/legacy/usr/bin:/usr/obj/usr/krnl_src/tmp/legacy/bin:/usr/obj/usr/krnl_src/tmp/usr/sbin:/usr/obj/usr/krnl_src/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin  config  -d /usr/obj/usr/krnl_src/sys/IPSEC  -I '/usr/krnl_src/sys/amd64/conf' '/usr/krnl_src/sys/amd64/conf/IPSEC'\n*** Error code 1\n\nStop.\nmake[1]: stopped in /usr/krnl_src\n*** Error code 1\n\nStop.\nmake: stopped in /usr/krnl_src", "stdout_lines": ["", "--------------------------------------------------------------", ">>> Kernel build for IPSEC started on Mon Sep 11 14:56:33 UTC 2017", "--------------------------------------------------------------", "===> IPSEC", "mkdir -p /usr/obj/usr/krnl_src/sys", "", "--------------------------------------------------------------", ">>> stage 1: configuring the kernel", "--------------------------------------------------------------", "cd /usr/krnl_src/sys/amd64/conf;  PATH=/usr/obj/usr/krnl_src/tmp/legacy/usr/sbin:/usr/obj/usr/krnl_src/tmp/legacy/usr/bin:/usr/obj/usr/krnl_src/tmp/legacy/bin:/usr/obj/usr/krnl_src/tmp/usr/sbin:/usr/obj/usr/krnl_src/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin  config  -d /usr/obj/usr/krnl_src/sys/IPSEC  -I '/usr/krnl_src/sys/amd64/conf' '/usr/krnl_src/sys/amd64/conf/IPSEC'", "*** Error code 1", "", "Stop.", "make[1]: stopped in /usr/krnl_src", "*** Error code 1", "", "Stop.", "make: stopped in /usr/krnl_src"]}

TASK [vpn : debug] ******************************************************************************************************************************************************
ok: [172.27.0.40] => {
    "building_kernel": {
        "ansible_job_id": "381194201046.27205",
        "changed": true,
        "finished": 0,
        "results_file": "/root/.ansible_async/381194201046.27205",
        "started": 1
    }
}

TASK [vpn : fail] *******************************************************************************************************************************************************
fatal: [172.27.0.40]: FAILED! => {"changed": false, "failed": true, "msg": "Something went wrong. Check the debug output above."}

TASK [vpn : debug] ******************************************************************************************************************************************************
ok: [172.27.0.40] => {
    "fail_hint": [
        "Sorry, but something went wrong!",
        "Please check the troubleshooting guide.",
        "https://trailofbits.github.io/algo/troubleshooting.html"
    ]
}

TASK [vpn : fail] *******************************************************************************************************************************************************
fatal: [172.27.0.40]: FAILED! => {"changed": false, "failed": true, "msg": "Failed as requested from task"}

PLAY RECAP **************************************************************************************************************************************************************
172.27.0.40                : ok=27   changed=10   unreachable=0    failed=3

Hultner commented 7 years ago

@dguido FreeBSD 10.4 is released now, maybe someone should consider updating the scripts to work with newer kernels.

I've never written ansible scripts previously so I don't think I could provide code of good quality but can atleast add that I managed to hack the current scripts to ignore the NAT-T kernel extension which is now built in since 11.1 but I don't think so is the case for 10.4 so someone with a little more experience should probably write a conditional based on output of freebsd-version in a way so that old behaviour is preserved at older versions kernels but if the returned version is >=11.1 we can skip it.

Pseudo code

kernel_version = /usr/bin/env freebsd-version

if kernel_version < 11.1
  old_behaviour_rebuild_kernel_w_nat_t()
else
  continue without rebuilding kernel

debdrup commented 7 years ago

Please note that as of revision 315514, IPSec support has been substancially changed, so the setup for FreeBSD might have to be changed a bit.

nooneischgl commented 6 years ago

I have gotten this to work with some workarounds. I am deploying directly on a Digital Ocean a FreeBSD11.1 droplet. The issue I have found are

Python being python2.7
Base64 not being installed (this is used later for exporting p12 certs)
Source env/bin/active needs active.csh
The algo certs are not placed as in in .ssh

This installation is assumed to be done as root (not best practice)

pkg install -y wget
pkg install -y python27
pkg install -y base64 
wget https://github.com/trailofbits/algo/archive/master.zip
tar xvfz master.zip
cd algo-master
cp /usr/local/bin/python2.7 /usr/local/bin/python
cp /usr/local/bin/python2.7-config  /usr/local/bin/python-config
python -m ensurepip --user
python -m pip install --user --upgrade virtualenv
python -m virtualenv env && source env/bin/activate.csh && python -m pip install -U pip && python -m pip install -r requirements.txt
#First run to generate Certs 
ansible-playbook deploy.yml -t local,vpn -e "server_ip=$server_ip server_user=root IP_subject_alt_name=$server_ip Store_CAKEY=N" --skip-tags cloud
# This will fail $server_ip unreachable=1
cat configs/algo.pem.pub >> ~/.ssh/authorized_keys
cp configs/algo.pem ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
cat configs/algo.pem.pub >> ~/.ssh/known_hosts
ansible-playbook deploy.yml -t local,vpn -e "server_ip=$server_ip server_user=root IP_subject_alt_name=$server_ip Store_CAKEY=N" --skip-tags cloud

Hultner commented 6 years ago

@jackivanov Why was this closed? Is it working in new FreeBSD-releases now? Or are you dropping support for FreeBSD? Or is it replaced by another issue?

jackivanov commented 6 years ago

Oh sorry, it was accidentally. An update is coming in this PR which closes the issue

Hultner commented 6 years ago

Ah great to hear, I was afraid that you were silently dropping support for *BSD.

trailofbits / algo

Support for *BSD #35