search

unofficial-unifi / unifi-pfsense

A script that installs the UniFi Controller software on pfSense and other FreeBSD systems
BSD 2-Clause "Simplified" License
752 stars 244 forks source link

mongodb removed #128

Closed ghost closed 3 years ago

ghost commented 6 years ago

I noticed when I installed the pfBlockerNG package it removed the mongodb package. Has anyone else seen this? Doesn't the UniFi software rely on mongodb?

ghost commented 6 years ago

This also happens with the Avahi package. I have to re-run the script and re-install.

toredash commented 6 years ago

I noticed when I installed the pfBlockerNG package it removed the mongodb package. Has anyone else seen this? Doesn't the UniFi software rely on mongodb?

Yes - https://github.com/gozoinks/unifi-pfsense/blob/master/install-unifi/install-unifi.sh#L99

vilhelmen commented 5 years ago

I tried installing unifi after installing the Avahi and pfBlockerNG packages and DNS was completely wrecked, I assume that is why. Can these packages coexist with the unifi software, or should I just figure out how to get docker running on my minnowboard?

vilhelmen commented 5 years ago

After looking more closely, it appears that Avahi and pfBlockerNG are triggering a downgrade of icu from 63.1 to 62.1_1, which triggers a removal of boost-libs 1.68.0_3 (which requires icu 63), which triggers a removal of mongo34 (which requires boost 1.68).

If there's an older version of boost/mongo that uses icu 62, that would fix it, but I'm very new to BSD and VERY bad with pkg.

Solution (well, kinda): Don't trust pfsense's package installer, it WILL NOT tell you when it's going to downgrade icu until it's too late. To install a package, go to the command line and run pkg add XXX (for example pfSense-pkg-pfBlockerNG) to get the list of packages needed. If it's not going to downgrade and uninstall mongodb, then it's fine and you can install it normally. Otherwise, run pkg fetch on each of those dependencies and go to /var/cache/pkg and run pkg add -f XXXX.txz to force install each of the dependencies and the main package. After that, it should hopefully just work unless it really needed icu 62. You generally need to logout and back in to see the new packages on your path.

cypherstream commented 5 years ago

Im having this same issue almost a year later wtih pfsense 2.4.4-RELEASEp3. Installing ntopng from web gui or installing nano text editor from ssh command line.

ntopng install log `>>> Installing pfSense-pkg-ntopng... Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. Checking integrity... done (1 conflicting)

Installed packages to be REMOVED: boost-libs-1.71.0_2 mongodb34-3.4.23

New packages to be INSTALLED: ntopng: 3.6.d201800910,1 [pfSense]

Installed packages to be DOWNGRADED: icu: 65.1,1 -> 62.1_1,1 [pfSense]

Installed packages to be REINSTALLED: pkg-1.10.5_6 [pfSense]

Number of packages to be removed: 2 Number of packages to be installed: 1 Number of packages to be reinstalled: 1 Number of packages to be downgraded: 1

The operation will free 282 MiB. [1/5] Deinstalling mongodb34-3.4.23... [1/5] Deleting files for mongodb34-3.4.23: .......... done ==> You should manually remove the "mongodb" user. ==> You should manually remove the "mongodb" group [2/5] Deinstalling boost-libs-1.71.0_2... [2/5] Deleting files for boost-libs-1.71.0_2: .......... done [3/5] Reinstalling pkg-1.10.5_6... [3/5] Extracting pkg-1.10.5_6: .......... done You may need to manually remove /usr/local/etc/pkg.conf if it is no longer needed. [4/5] Downgrading icu from 65.1,1 to 62.1_1,1... [4/5] Extracting icu-62.1_1,1: .......... done [5/5] Installing ntopng-3.6.d201800910,1... ===> Creating groups. Creating group 'ntopng' with gid '288'. ===> Creating users Creating user 'ntopng' with uid '288'. [5/5] Extracting ntopng-3.6.d201800910,1: .......... done Message from ntopng-3.6.d201800910,1:

WARNING:

ntopng runs a web interface service by default, it is suggested to protect such network accessible services with packet filters or TCP wrappers.

ntopng requires to connect to a redis server to work. Please install redis server from databases/redis or use -r option via ntopng_flags to specify a remote one.

If you enabled GeoIP support(the default), please use ntopng-geoipupdate.sh to update GeoIP database to the latest available data.

To pass a configuration file to ntopng, which overrides any command line arguments, add something like the following to rc.conf:

ntopng_flags="/path/to/file.conf"

Cleaning up cache... done. Success`

Installing nano text editor in SSH (Really why does something as brain dead simple as nano want to even touch mongodb?) `pkg install nano Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... Fetching meta.txz: 100% 944 B 0.9kB/s 00:01 Fetching packagesite.txz: 100% 138 KiB 141.4kB/s 00:01 Processing entries: 100% pfSense repository update completed. 522 packages processed. All repositories are up to date. The following 4 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED: boost-libs-1.71.0_2 mongodb34-3.4.23

New packages to be INSTALLED: nano: 2.9.8 [pfSense]

Installed packages to be DOWNGRADED: icu: 65.1,1 -> 62.1_1,1 [pfSense]

Number of packages to be removed: 2 Number of packages to be installed: 1 Number of packages to be downgraded: 1

The operation will free 295 MiB. 463 KiB to be downloaded.

Proceed with this action? [y/N]: y [1/1] Fetching nano-2.9.8.txz: 100% 463 KiB 473.9kB/s 00:01 Checking integrity... done (0 conflicting) [1/4] Deinstalling mongodb34-3.4.23... [1/4] Deleting files for mongodb34-3.4.23: 100% ==> You should manually remove the "mongodb" user. ==> You should manually remove the "mongodb" group [2/4] Deinstalling boost-libs-1.71.0_2... [2/4] Deleting files for boost-libs-1.71.0_2: 100% [3/4] Installing nano-2.9.8... [3/4] Extracting nano-2.9.8: 100% [4/4] Downgrading icu from 65.1,1 to 62.1_1,1... [4/4] Extracting icu-62.1_1,1: 100%`

vilhelmen commented 5 years ago

It's most likely caused by the icu downgrade. I'd suggest manually force installing the individual packages and dependencies and cross your fingers. But I can't promise an update or installation of something completely unrelated won't cause pkg to realize something doesn't add up and trash your install later :(

I gave up on running the unifi software after it eventually garbled itself and refused to run anymore, I'm running everything in standalone mode now. I just don't have the energy to find a fix since jails seem explicitly disabled on pfsense and I'm not aware of any container support.

cypherstream commented 5 years ago

That’s interesting since this is the first time I’m experiencing it. I upgraded from a Dell Optiplex 390 core i3 to a Dell Optiplex 3020 core i5. The new pfsense is EFI booted into a ZFS formatted SSD, but other than that I restored my old pfsense config to it. My old box did not have this issue.

Really I just get it working and I rarely touch the packages, but what could be concerning are package updates (which worked without issue on the Optiplex 390, bios + ufs install). Right now it’s all working but god forbid I want to install or update an existing package on the box. I guess one alternative would be to load a hypervisor on first and install two VM’s, keeping UniFi on its own Ubuntu VM (or similar). However I’m looking to keep the power usage down and with the pfsense on the bare metal I’m down to 22 watts at idle when the processor is clocked down. That’s better than the older optiplex 390 which averaged around 38-40 watts at idle.

vilhelmen commented 5 years ago

Everyone keeps telling me to just put a pi on my network for unifi, but I really don't want any more hardware floating around to maintain.

This is my first pfsense machine, so I got one of negate's dual core minnowboards. The thing idles at like 65C for some reason, so I ordered a cheap fan for RPIs off amazon and jammed it in and now it's down to ~34C which is an improvement, at least. Not sure about its power consumption, though.

I'm tempted to figure out how to put a basic hypervisor on it but I just cannot be bothered; I'm not a networking guy.

cypherstream commented 5 years ago

Does anyone know what the package icu does or why pkg wants to downgrade it? Is icu a dependency of mongodb and boost-libs? So is the thought if you manually install the older icu, pkg won’t want to remove the other two packages? To me it seems pkg was broken in pfsense 2.4.4 release 3. I was in 2.4.4 release 2 on my old box, no issues there, though it began its life on 2.3.4 and has been upgraded over time.

vilhelmen commented 5 years ago

ICU is some sort of unicode library. I don't have the repo plugged into pkg so I'm digging through the package listing on my laptop. Looks like boost is the one that wants icu, not mongo. I might have found something if you know how to work pkg (I do not).

The options block for boost-libs seems to suggest that ICU is optional in some way: "options":{"DEBUG":"off","OPTIMIZED_CFLAGS":"off","ICONV":"on","ICU":"on"},, so maybe ICU=off is something you can do? It sounds like something you'd see in a gentoo ebuild, but idk if pkg generally compiles things. It may just be reporting what their copy of boost was compiled with. I couldn't say what the ramifications could be other than potentially broken unicode support somewhere.

Installing both works in theory, but idk if pkg would throw a fit once it notices. I'm sure you can do it with force installations, but symlinks will clobber trying to point to the actual so file. assuming 65 and 62 are similar enough, it shouldn't matter. I'd suggest installing 62 and then 65 on top.

cypherstream commented 5 years ago

Just to experiment I took another SSD and this time I loaded VMware ESXi 6.7u3. Ran into an issue and update 3 has a bug where I had to change the UEFI boot to legacy. Anyway to make a long story short I was able to build 2 VM’s, one PFsense set to start first, the second Ubuntu 18.04 LTS and I installed UniFi in there. I restored my configs to both. They seem to operate at slightly higher wattage. Pfsense on bare metal is better on power, plus it boots faster and it’s much more simpler. The VMware approach uses more power at first, when it boots and then the VMs boot, up to 50 watts, and it hovers around 45 watts if doing a long iperf test between WAN and LAN around at around 984 mbps. At idle it’s about 25-26 watts. On bare metal idle is about 22 watts, and a fully loaded WAN to LAN gigabit iperf test maxes out around 38 watts. It’s not terribly off, and VMware gives you some flexibility to run other things in the same box. However I still lean towards simplicity and speed of boot of pfsense natively. I know this test goes a little beyond the scope of this issue, but my long winded point is that unifi on pfsense IS simple, preferred and when it works it’s really a great solution.

cypherstream commented 5 years ago

Lets say I want to install the pfsense package ntopng. Here is what it prompts me. notice again boost libs and mongodb removed, and icu downgrade again.

Why didnt 2.4.4 p2 exhibit this behavior?

# pkg install ntopng
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
The following 8 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        boost-libs-1.71.0_2
        mongodb34-3.4.23

New packages to be INSTALLED:
        ntopng: 3.6.d201800910,1 [pfSense]
        libsodium: 1.0.16 [pfSense]
        ndpi: 2.4.d20180830,1 [pfSense]
        GeoIP: 1.6.12 [pfSense]
        mysql56-client: 5.6.41 [pfSense]

Installed packages to be DOWNGRADED:
        icu: 65.1,1 -> 62.1_1,1 [pfSense]

Number of packages to be removed: 2
Number of packages to be installed: 5
Number of packages to be downgraded: 1

The operation will free 233 MiB.

Proceed with this action? [y/N]: n
cypherstream commented 5 years ago

I found a command called pkg lock that appear to fix this issue. I'm not sure if I should also lock icu or not, so I'll have to test all that. More on pkg lock: https://www.freebsd.org/cgi/man.cgi?query=pkg-lock&sektion=8&manpath=freebsd-release-ports

pkg lock mongodb34-3.4.23
mongodb34-3.4.23: lock this package? [y/N]: y
Locking mongodb34-3.4.23

pkg lock boost-libs-1.71.0_2
boost-libs-1.71.0_2: lock this package? [y/N]: y
Locking boost-libs-1.71.0_2

pkg install ntopng
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 6 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        ntopng: 3.6.d201800910,1 [pfSense]
        libsodium: 1.0.16 [pfSense]
        ndpi: 2.4.d20180830,1 [pfSense]
        GeoIP: 1.6.12 [pfSense]
        mysql56-client: 5.6.41 [pfSense]

Installed packages to be DOWNGRADED:
        icu: 65.1,1 -> 62.1_1,1 [pfSense]

Number of packages to be installed: 5
Number of packages to be downgraded: 1

The process will require 62 MiB more space.
vilhelmen commented 5 years ago

I couldn't tell you what changed, unfortunately. You'd probably have to dig through package listings for that version. I can't believe we did't think of version locking before now.

I would probably lock icu to the newer version. I'd be worried about breaking boost by downgrading. Ideally, icu is relatively backwards compatible.

I'm looking forward to getting the controller software back up and running, good find!

cypherstream commented 5 years ago

I wiped my pfsense box and started over. When I restored my working config from the old machine, the WAN and LAN interfaces were different (re0 and igb0). So when it rebooted it "didn't have internet", so it never reinstalled the packages properly. So next I edited my config backup XML in notepad++ and fixed my WAN and LAN interfaces (igb0 and igb1). Wiped pfsense again, started over and imported the config. When it rebooted it had internet and it correctly downloaded all of my previous packages (like pfBlockerNG, ntopng, nut, bandwidthd, iperf, etc..). After that settled I installed the Unifi controller via script. I then installed nano and lsof via command line. Nothing complained about needing to remove packages. Mongodb remained intact. Also pkg info no longer said "database version 35 is newer than libpkg(3) version 34". The output of the command looks clean.

So I have to wonder if initially restoring the old config resulted in some broken or mishandled packages since it did not initially have internet access until I resolved the interfaces.

Anyway just to make sure I pkg locked mongodb, boost and icu (at version 65) to ensure they won't be uninstalled - just in case.

vilhelmen commented 5 years ago

Rad! I'll replicate sometime this weekendish and update my PR. Maybe I'll add locking to the installer, but I'm worried about not receiving updated for those libraries. But, well, we can't trust the system to not uninstall it so maybe a lock is best.

vilhelmen commented 5 years ago

I got halfway through running the installer script when I noticed that the mongodb installation goes EOL in less than two months and that both the latest and lts versions of the unifi controller software are available through the package list we're pulling from. I, too, have wiped my pfsense box and started over, haha. I really appreciate the ability to extract the config when reinstalling, and now that I'm on ZFS I should never have to do this again.

I'm going to fiddle with the installer and see if I can temporarily load the repo in pkg or parse the dependency graph to do things a bit cleaner.

vilhelmen commented 4 years ago

Scott pilgrim and the infinite edits: I've gotten it working!

You can pretty easily enable the standard FreeBSD repo by modifying /usr/local/etc/pkg/repos/pfSense.conf. Strip out the disabled blob (FreeBSD: { enabled: no }) and add the following with the appropriate ABI:

FreeBSD: {
   url: "pkg+https://pkg.freebsd.org/${ABI}/latest",
   mirror_type: "srv",
   signature_type: "fingerprints",
   fingerprints: "/usr/share/keys/pkg/",
   enabled: yes
}

From there you need to lock pkg (it will want to update) and you can install the unifi5 package. There's a minor update to php72-intl most likely related to icu and not installing the php update makes pfsense freak out. You should then disable the FreeBSD repo to make sure pkg doesn't start going haywire later. You should then lock the icu package.

Adding the unifi_enable=YES to the /etc/rc.local.conf is needed to start it, and swapping com.ubnt.ace.Launcher for -jar /usr/local/share/java/unifi/lib/ace.jar in the rc script gets it starting correctly.

Here's a rough installer! I think I got all the steps in here. It may be worthwhile to add echo "" | nc -l 127.0.0.1 8080 >/dev/null & to start_precmd as noted in the existing rc script in the repo.

I'd love to get this integrated with the existing installer script, but it would take a lot of work to migrate existing installs. Maybe a 2.0 sort of thing?

#!/bin/sh

ABI=`/usr/sbin/pkg config abi`

# Modify /usr/local/etc/pkg/repos/pfSense.conf to include FreeBSD data:
# FreeBSD: {
#    url: "pkg+https://pkg.freebsd.org/${ABI}/latest",
#    mirror_type: "srv",
#    signature_type: "fingerprints",
#    fingerprints: "/usr/share/keys/pkg/",
#    enabled: no
# }
# For best safety, extract current config, delete, add, work, restore?
# Ugh just copy it over to /tmp and assume the default config
cp /usr/local/etc/pkg/repos/pfSense.conf /tmp/pfSense.conf
fgrep -v 'FreeBSD: { enabled: no }' /tmp/pfSense.conf > /usr/local/etc/pkg/repos/pfSense.conf
cat << EOF >> /usr/local/etc/pkg/repos/pfSense.conf

FreeBSD: {
   url: "pkg+https://pkg.freebsd.org/${ABI}/latest",
   mirror_type: "srv",
   signature_type: "fingerprints",
   fingerprints: "/usr/share/keys/pkg/",
   enabled: yes
}
EOF

# pkg wants to update, please don't
pkg lock pkg
# maybe just yes | pkg lock -a

pkg install unifi5

# RESTORE PACKAGE LOCK
# yes | pkg unlock -a
# Unecessary icu lock if unifi locked?
# boost/mongo lock does NOT lock icu, so yes.
pkg unlock pkg
pkg lock icu

# Restore package conf
cp /tmp/pfSense.conf /usr/local/etc/pkg/repos/pfSense.conf

# fix java launch
# look into integrating boot speedup fix in the custom launcher
sed -i.bak -e 's|com.ubnt.ace.Launcher|-jar /usr/local/share/java/unifi/lib/ace.jar|g' /usr/local/etc/rc.d/unifi
rm /usr/local/etc/rc.d/unifi.bak
mv /usr/local/etc/rc.d/mv /usr/local/etc/rc.d/unifi /usr/local/etc/rc.d/unifi.sh 

if [ ! -f /etc/rc.conf.local ] || [ $(grep -c unifi_enable /etc/rc.conf.local) -eq 0 ]; then
  echo -n "Enabling the unifi service..."
  echo "unifi_enable=YES" >> /etc/rc.conf.local
  echo " done."
fi 

service unifi.sh start
AliceGrey commented 4 years ago

Just a quick tip for anyone looking to recover from mongodb getting removed after having a working install but no backup.

Stop unifi service unifi.sh stop Rename the current install mv /usr/local/Unifi /usr/local/Unifi.old Then re-install unifi-pfsense fetch -o - https://git.io/j7Jy | sh -s once completed stop unifi service unifi.sh stop Then rename the new install mv /usr/local/Unifi /usr/local/Unifi.new Move the old install back mv /usr/local/Unifi.old /usr/local/Unifi Start unifi again service unifi.sh start

Then login to your unifi and make a backup! This is a bit hacky but saved my butt today.

I'd recommend locking icu, mongodb34 and boost-libs afterwards. Hopefully @vilhelmen 's improved installer will iron out this issue soon!

vilhelmen commented 4 years ago

An in place upgrade to 2.4.5 seems to have wrecked java/snappy/etc /usr/local/share/java, which contained the unify jar, is missing, so that's not good.

Following along with my install script seems to mostly work, but the dependencies for the unifi software, mongoldb, and java have updates that don't get automagically detected.

Running a generic upgrade, with pkg locked, seemed to only pickup unifi-related packages:

    snappy: 1.1.7 -> 1.1.8 [FreeBSD]

    xorgproto: 2019.2 -> 2020.1 [FreeBSD]
    openjdk8: 8.232.09.1_1 -> 8.252.09.1 [FreeBSD]
    mongodb36: 3.6.14_1 -> 3.6.17 [FreeBSD]
    libinotify: 20180201_1 -> 20180201_2 [FreeBSD]
    libX11: 1.6.9,1 -> 1.6.9_1,1 [FreeBSD]
    javavmwrapper: 2.7.4 -> 2.7.5 [FreeBSD]
    java-zoneinfo: 2019.b -> 2020.a [FreeBSD]
    boost-libs: 1.72.0 -> 1.72.0_2 [FreeBSD]

but it's still a no-go with Exception in thread "main" java.lang.NoClassDefFoundError: org/xerial/snappy/OSInfo when launching the jar.

A force reinstall of snappy and snappyjava pulls in a new dependency despite no version changes, which suggests that the dependency tree has gone haywire.

The force reinstall (pkg install -f snappy snappy java)has fixed that issue, but now it's something about tomcat: Exception in thread "launcher" java.lang.IllegalStateException: Tomcat failed to start up

But it's now 5am and I'd like to go to bed, so I've done a snapshot rollback.

dvdp commented 4 years ago

Sounds like the classical 'wrong snappy version' issue. Please check the threads, it should be mentioned somewhere which version is required (you need to download the right version and copy it over the version in the UniFi tree and changing the name to whatever name is listed in that tree)

cypherstream commented 4 years ago

I upgraded my unifi controller to 5.12.66 using the same install script and no issues. My pfsense is already on 2.4.5.

The same packages I had locked before are still locked to my knowledge. I did this as a part of re-iping my home network - going from 192.168.1.x to 192.168.5.x so I could still print to my printer when VPN'd in and working from home (192.168.1.x overlapped with a work network).

gnkidwell commented 3 years ago

The newest version of the script has a pkg lock on it, also a mongodb repair script added. Closing this for now.