search

safing / portmaster

🏔 Love Freedom - ❌ Block Mass Surveillance
https://safing.io
GNU General Public License v3.0
9.2k stars 292 forks source link

Packaging for NixOS #306

Open 06kellyjac opened 3 years ago

06kellyjac commented 3 years ago

Pre-Submit Checklist:

 - system: `"x86_64-linux"`
 - host os: `Linux 5.11.16, NixOS, 21.05.20210514.65a9923 (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20210503_6d2553a`
 - channels(root): `"home-manager, nixos-21.05pre282015.a73020b2a15, nixos-hardware"`
 - channels(user): `""`
 - nixpkgs: `/nix/store/8xvwfm02dicgbpkyyaqi0961nd01r8g0-source`

What worked?

What did not work?

portmaster-start fails to run on a NixOS system due to the way it's linked

λ wget -O /tmp/portmaster-start https://updates.safing.io/latest/linux_amd64/start/portmaster-start
--2021-05-18 10:43:49--  https://updates.safing.io/latest/linux_amd64/start/portmaster-start
Resolving updates.safing.io (updates.safing.io)... 116.203.130.137
Connecting to updates.safing.io (updates.safing.io)|116.203.130.137|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13186257 (13M) [application/octet-stream]
Saving to: ‘/tmp/portmaster-start’

/tmp/portmaster-start             100%[============================================================>]  12.58M  8.25MB/s    in 1.5s

2021-05-18 10:43:51 (8.25 MB/s) - ‘/tmp/portmaster-start’ saved [13186257/13186257]

λ chmod +x /tmp/portmaster-start
λ cd /tmp
λ ./portmaster-start --help
zsh: no such file or directory: ./portmaster-start

I've built portmaster-start on NixOS and it runs fine but then the resources it fetches such as the UI and Core etc have the same linking issue.

λ ./result/bin/portmaster-start update --data=/tmp/portmaster
210518 10:57:19.116 ▶ BOF
210518 10:57:19.310 r/updating:096 ▶ INFO 001 updates: updated index stable.json
210518 10:57:19.310 r/updating:138 ▶ INFO 002 updates: starting to download 5 updates
210518 10:57:21.971 ater/fetch:074 ▶ INFO 003 updates: fetched https://updates.safing.io/linux_amd64/core/portmaster-core_v0-6-13 (stored to /tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-13)
210518 10:57:22.085 ater/fetch:074 ▶ INFO 004 updates: fetched https://updates.safing.io/all/ui/modules/portmaster_v0-1-12.zip (stored to /tmp/portmaster/updates/all/ui/modules/portmaster_v0-1-12.zip)
210518 10:57:23.610 ater/fetch:074 ▶ INFO 005 updates: fetched https://updates.safing.io/linux_amd64/notifier/portmaster-notifier_v0-2-2 (stored to /tmp/portmaster/updates/linux_amd64/notifier/portmaster-notifier_v0-2-2)
210518 10:57:38.964 ater/fetch:074 ▶ INFO 006 updates: fetched https://updates.safing.io/linux_amd64/app/portmaster-app_v0-2-1.zip (stored to /tmp/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-1.zip)
210518 10:57:40.778 ater/fetch:074 ▶ INFO 007 updates: fetched https://updates.safing.io/linux_amd64/start/portmaster-start_v0-5-3 (stored to /tmp/portmaster/updates/linux_amd64/start/portmaster-start_v0-5-3)
210518 10:57:40.778 r/updating:152 ▶ INFO 008 updates: finished downloading updates
210518 10:57:40.778 r/resource:262 ▶ TRAC 009 updater: selected version 0.0.5 for resource all/spn/bootstrap.dsd
210518 10:57:40.778 r/resource:262 ▶ TRAC 010 updater: selected version 0.1.8 for resource all/ui/modules/settings.zip
210518 10:57:40.778 r/resource:262 ▶ TRAC 011 updater: selected version 0.2.11 for resource all/ui/modules/base.zip
210518 10:57:40.778 r/resource:262 ▶ TRAC 012 updater: selected version 0.6.13 for resource windows_amd64/core/portmaster-core.exe
210518 10:57:40.778 r/resource:262 ▶ TRAC 013 updater: selected version 0.1.11 for resource all/ui/modules/console.zip
210518 10:57:40.778 r/resource:262 ▶ TRAC 014 updater: selected version 0.2.5 for resource windows_amd64/hub/spn-hub.exe
210518 10:57:40.778 r/resource:262 ▶ TRAC 015 updater: selected version 0.5.3 for resource linux_amd64/start/portmaster-start
210518 10:57:40.778 r/resource:262 ▶ TRAC 016 updater: selected version 1.0.10 for resource windows_amd64/kext/portmaster-kext.dll
210518 10:57:40.778 r/resource:262 ▶ TRAC 017 updater: selected version 0.5.2 for resource windows_amd64/packages/portmaster-installer.exe
210518 10:57:40.778 r/resource:262 ▶ TRAC 018 updater: selected version 1.0.10 for resource windows_amd64/kext/portmaster-kext.sys
210518 10:57:40.778 r/resource:262 ▶ TRAC 019 updater: selected version 0.6.13 for resource darwin_amd64/core/portmaster-core
210518 10:57:40.778 r/resource:262 ▶ TRAC 020 updater: selected version 0.5.2 for resource linux_amd64/packages/portmaster-installer.deb
210518 10:57:40.778 r/resource:262 ▶ TRAC 021 updater: selected version 0.2.1 for resource windows_amd64/app/portmaster-app.zip
210518 10:57:40.778 r/resource:262 ▶ TRAC 022 updater: selected version 0.2.2 for resource all/ui/modules/assets.zip
210518 10:57:40.778 r/resource:262 ▶ TRAC 023 updater: selected version 0.6.13 for resource linux_amd64/core/portmaster-core
210518 10:57:40.778 r/resource:262 ▶ TRAC 024 updater: selected version 0.6.0 for resource windows_amd64/notifier/portmaster-snoretoast.exe
210518 10:57:40.778 r/resource:262 ▶ TRAC 025 updater: selected version 0.1.12 for resource all/ui/modules/portmaster.zip
210518 10:57:40.778 r/resource:262 ▶ TRAC 026 updater: selected version 0.2.4 for resource all/ui/modules/monitor.zip
210518 10:57:40.778 r/resource:262 ▶ TRAC 027 updater: selected version 0.5.3 for resource darwin_amd64/start/portmaster-start
210518 10:57:40.778 r/resource:262 ▶ TRAC 028 updater: selected version 0.5.2 for resource linux_amd64/packages/portmaster-installer.pkg.tar.xz
210518 10:57:40.778 r/resource:262 ▶ TRAC 029 updater: selected version 0.2.5 for resource linux_amd64/hub/spn-hub
210518 10:57:40.778 r/resource:262 ▶ TRAC 030 updater: selected version 0.2.2 for resource linux_amd64/notifier/portmaster-notifier
210518 10:57:40.778 r/resource:262 ▶ TRAC 031 updater: selected version 0.2.2 for resource windows_amd64/notifier/portmaster-notifier.exe
210518 10:57:40.778 r/resource:262 ▶ TRAC 032 updater: selected version 0.2.5 for resource darwin_amd64/hub/spn-hub
210518 10:57:40.778 r/resource:262 ▶ TRAC 033 updater: selected version 0.2.1 for resource linux_amd64/app/portmaster-app.zip
210518 10:57:40.778 r/resource:262 ▶ TRAC 034 updater: selected version 0.1.7 for resource all/ui/modules/profilemgr.zip
210518 10:57:40.778 r/resource:262 ▶ TRAC 035 updater: selected version 0.5.3 for resource windows_amd64/start/portmaster-start.exe
210518 10:57:42.385 /unpacking:142 ▶ INFO 036 updates: unpacked linux_amd64/app/portmaster-app_v0-2-1.zip
210518 10:57:42.395 ◀ EOF
λ ./result/bin/portmaster-start app --data=/tmp/portmaster
[control] 2021/05/18 09:57:54 starting /tmp/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-1/portmaster-app_v0-2-1 --data /tmp/portmaster
[control] 2021/05/18 09:57:54 app/portmaster-app.zip failed with: failed to start app/portmaster-app.zip: fork/exec /tmp/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-1/portmaster-app_v0-2-1: no such file or directory
[control] 2021/05/18 09:57:54 /tmp/portmaster/logs/app/2021-05-18-09-57-54.error.log: writing logs failed after 0 bytes: read |0: file already closed
[control] 2021/05/18 09:57:54 /tmp/portmaster/logs/app/2021-05-18-09-57-54.log: writing logs failed after 0 bytes: read |0: file already closed
[control] 2021/05/18 09:57:56 starting /tmp/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-1/portmaster-app_v0-2-1 --data /tmp/portmaster
[control] 2021/05/18 09:57:56 /tmp/portmaster/logs/app/2021-05-18-09-57-56.log: writing logs failed after 0 bytes: read |0: file already closed
[control] 2021/05/18 09:57:56 app/portmaster-app.zip failed with: failed to start app/portmaster-app.zip: fork/exec /tmp/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-1/portmaster-app_v0-2-1: no such file or directory
[control] 2021/05/18 09:58:00 updating registry index
[control] 2021/05/18 09:58:00 starting /tmp/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-1/portmaster-app_v0-2-1 --data /tmp/portmaster
[control] 2021/05/18 09:58:00 /tmp/portmaster/logs/app/2021-05-18-09-58-00.error.log: writing logs failed after 0 bytes: read |0: file already closed
[control] 2021/05/18 09:58:00 app/portmaster-app.zip failed with: failed to start app/portmaster-app.zip: fork/exec /tmp/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-1/portmaster-app_v0-2-1: no such file or directory
^C[control] 2021/05/18 09:58:04 got interrupt signal, exiting... (not executing anything

Also due to the nature of package managers and especially nixpkgs it's strongly prefered to have all the updating done by the package manager. I can understand the DB being updated by the program like trivy, clamAV etc, but we'd want the program itself updated through nix

I've had a look at building everything using nix and considered writing a utility or patching the portmaster-start update command to just update the db and copy over all the prebuilt files but I thought I'd post here first.

Building static go binaries might also make the downloaded files work on NixOS (and other systems really) but as I said we prefer building everything from source

Debug Information:

dhaavi commented 3 years ago

Hey @06kellyjac, thanks for reporting this. NixOS hasn't been on our radar yet, but let's see how hard it would be get the Portmaster running there!

portmaster-start fails to run on a NixOS system due to the way it's linked

I had some fixing to do and enabled static linking for portmaster-start and portmaster-core in the process. The newest release of these two components is now statically linked.

Unfortunately, statically linking the UI components is not feasible, as they bring loads of dependencies with them. I'm not sure how these would be best built for NixOS.

Also due to the nature of package managers and especially nixpkgs it's strongly prefered to have all the updating done by the package manager.

We built the auto-updating system for operating systems with no included package management. The auto-updater also helps us move fast, as we are still in alpha and we do push a lot of changes. Looking into how to distribute the Portmaster with all the Linux distributions is not something we have the resources to do now. We will however look into this and re-evaluate all options once we are in late beta or stable and we have enough resources to dedicate to this.

I've had a look at building everything using nix and considered writing a utility or patching the portmaster-start update command to just update the db and copy over all the prebuilt files but I thought I'd post here first.

From where would you copy the pre-built files?

Building static go binaries might also make the downloaded files work on NixOS (and other systems really) but as I said we prefer building everything from source

I'd be interested how you would build our UI components in NixOS:

06kellyjac commented 3 years ago

The newest release of these two components is now statically linked.

Pulled the latest portmaster-start ran out of the box :+1:

core looks pretty good

λ sudo /tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14 --data /tmp/portmaster
[sudo] password for user:
210518 16:51:44.215 ▶ BOF
210518 16:51:44.216 ules/start:099 ▶ INFO 001 modules: initiating...
210518 16:51:44.216 ules/start:203 ▶ INFO 002 modules: started database
210518 16:51:44.216 ules/start:203 ▶ INFO 003 modules: started config
210518 16:51:44.216 ules/start:203 ▶ INFO 004 modules: started runtime
210518 16:51:44.216 ules/start:203 ▶ INFO 005 modules: started api
210518 16:51:44.216 api/router:050 ▶ INFO 006 api: starting to listen on 127.0.0.1:817
210518 16:51:44.216 ules/start:203 ▶ INFO 007 modules: started metrics
210518 16:51:44.222 ules/start:203 ▶ INFO 008 modules: started rng
210518 16:51:44.222 ules/start:203 ▶ INFO 009 modules: started base
210518 16:51:44.223 ules/start:203 ▶ INFO 010 modules: started subsystems
210518 16:51:44.223 ules/start:108 ▶ INFO 011 modules: initiated subsystems manager
210518 16:51:44.323 dules/mgmt:094 ▶ INFO 012 modules: managing changes
210518 16:51:44.323 ules/start:203 ▶ INFO 013 modules: started status
210518 16:51:44.323 ules/start:203 ▶ INFO 014 modules: started notifications
210518 16:51:44.323 ules/start:203 ▶ INFO 015 modules: started netenv
210518 16:51:44.329 s/upgrader:232 ▶ WARN 016 updates: parent process does not seem to be portmaster-start, name is sudo
210518 16:51:44.329 ules/start:203 ▶ INFO 018 modules: started updates
210518 16:51:44.329 ules/start:203 ▶ INFO 019 modules: started geoip
210518 16:51:44.329 ules/start:203 ▶ INFO 020 modules: started ui
210518 16:51:44.336 ules/start:203 ▶ INFO 021 modules: started profiles
210518 16:51:44.336 ules/start:203 ▶ INFO 022 modules: started processes
210518 16:51:44.339 ules/start:203 ▶ INFO 023 modules: started network
210518 16:51:44.368 /resolvers:282 ▶ INFO 024 resolver: no local resolvers loaded
210518 16:51:44.368 /resolvers:306 ▶ INFO 025 resolver: no scopes loaded
210518 16:51:44.368 ules/start:203 ▶ INFO 026 modules: started resolver
210518 16:51:44.393 ules/start:203 ▶ INFO 027 modules: started filterlists
210518 16:51:44.393 ules/start:203 ▶ INFO 028 modules: started intel
210518 16:51:44.448 ules/start:203 ▶ INFO 029 modules: started interception
210518 16:51:44.448 ules/start:203 ▶ INFO 030 modules: started core
210518 16:51:44.448 ules/start:203 ▶ INFO 031 modules: started filter
210518 16:51:44.448 ver/module:105 ▶ INFO 032 nameserver: starting to listen on 127.0.0.17:53
210518 16:51:44.448 ver/module:105 ▶ INFO 033 nameserver: starting to listen on [::1]:53
210518 16:51:44.448 ules/start:203 ▶ INFO 034 modules: started nameserver
210518 16:51:44.448 dules/mgmt:112 ▶ INFO 035 modules: finished managing
210518 16:51:44.456 all/master:079 ▶ INFO 036 filter: re-evaluating verdict on Unknown::-1 <- XX.XXX.XXX.129
210518 16:51:44.461 all/master:147 ▶ INFO 037 filter: granting own connection root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> XXX.XXX.X.100
210518 16:51:44.461 connection:615 ▶ INFO 038 filter: connection root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> XXX.XXX.X.100 accepted: connection by Portmaster
210518 16:51:44.461 all/master:079 ▶ INFO 039 filter: re-evaluating verdict on root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> XXX.XXX.X.100
210518 16:51:44.461 all/master:147 ▶ INFO 040 filter: granting own connection root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> XXX.XXX.X.100
210518 16:51:44.461 connection:615 ▶ INFO 041 filter: connection root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> XXX.XXX.X.100 accepted: connection by Portmaster
210518 16:51:44.479 all/master:147 ▶ INFO 042 filter: granting own connection root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> 116.203.130.137
210518 16:51:44.479 connection:615 ▶ INFO 043 filter: connection root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> 116.203.130.137 accepted: connection by Portmaster
210518 16:51:45.169 all/master:079 ▶ INFO 044 filter: re-evaluating verdict on user:/nix/store/g0sgj34x9wds4vyvsxlay16vs58m8m19-firefox-88.0.1/lib/firefox/firefox:3295 -> XXX.XXX.XXX.234
210518 16:51:45.328 all/master:147 ▶ INFO 045 filter: granting own connection root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> XXX.XXX.X.100
210518 16:51:45.328 connection:615 ▶ INFO 046 filter: connection root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> XXX.XXX.X.100 accepted: connection by Portmaster
210518 16:51:45.328 all/master:147 ▶ INFO 047 filter: granting own connection root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> XXX.XXX.X.100
210518 16:51:45.328 connection:615 ▶ INFO 048 filter: connection root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> XXX.XXX.X.100 accepted: connection by Portmaster
210518 16:51:45.349 ine-status:207 ▶ INFO 049 netenv: setting online status to Online (all checks passed)
210518 16:51:47.381 all/master:079 ▶ INFO 050 filter: re-evaluating verdict on user:/nix/store/1rh9fqgj12p52fd0mnqb8dng8zilmzmv-keybase-5.6.1/bin/keybase:3340686 -> XX.XX.XXX.64
210518 16:51:48.120 ater/fetch:073 ▶ INFO 051 updates: fetched https://updates.safing.io/all/intel/geoip/geoipv4_v20200514-6-49.mmdb.gz (stored to /tmp/portmaster/updates/all/intel/geoip/geoipv4_v20200514-6-49.mmdb.gz)
210518 16:51:51.152 connection:615 ▶ INFO 052 filter: connection Unknown::-1 -> XXX.XXX.X.100 to nameserver: redirecting rogue dns query
210518 16:51:51.161 connection:615 ▶ INFO 053 filter: connection root:/tmp/portmaster/updates/linux_amd64/core/portmaster-core_v0-6-14:1062514 -> X.X.X.2 accepted: connection by Portmaster

The UI at localhost:817 just spins though:

image

hub just sits there quietly which is a good sign, no errors, not sure what hub looks like when it's working:

λ ./portmaster-start --data /tmp/portmaster hub
^C[control] 2021/05/18 15:54:49 got interrupt signal, exiting... (not executing anything)

recover-iptables ran and exited fine but I have no idea if it did what it should :laughing: 

λ sudo ./portmaster-start --data /tmp/portmaster recover-iptables

From where would you copy the pre-built files?

I imagined I would package the prebuilt files within the nix store and the wrapper would just run portmaster-start update and overwrite the downloaded files with the nix prebuilt copies.

I'd be interested how you would build our UI components in NixOS

go binaries using GTK3 should be pretty easy

Building electron apps is a bit of a pain with nix but it is possible: https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/applications/networking/instant-messengers/element/element-desktop.nix If building from source isn't feasable we usually unpack from an appimage or dmg and patch it to work, e.g: https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/applications/networking/cluster/octant/desktop.nix

The app doesn't run as expected,

here's an ldd output (any links that are found aren't guarenteed to work and probably need patchelf-ing)

ldd /tmp/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-1/portmaster-app_v0-2-1
    linux-vdso.so.1 (0x00007fff69c6d000)
    libffmpeg.so => /tmp/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-1/libffmpeg.so (0x00007f5fb8867000)
    libdl.so.2 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libdl.so.2 (0x00007f5fb8862000)
    libpthread.so.0 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libpthread.so.0 (0x00007f5fb8841000)
    librt.so.1 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/librt.so.1 (0x00007f5fb8836000)
    libgobject-2.0.so.0 => not found
    libglib-2.0.so.0 => not found
    libxshmfence.so.1 => not found
    libgio-2.0.so.0 => not found
    libnss3.so => not found
    libnssutil3.so => not found
    libsmime3.so => not found
    libnspr4.so => not found
    libatk-1.0.so.0 => not found
    libatk-bridge-2.0.so.0 => not found
    libdbus-1.so.3 => not found
    libdrm.so.2 => not found
    libgdk_pixbuf-2.0.so.0 => not found
    libgtk-3.so.0 => not found
    libgdk-3.so.0 => not found
    libpango-1.0.so.0 => not found
    libcairo.so.2 => not found
    libm.so.6 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libm.so.6 (0x00007f5fb86ed000)
    libX11.so.6 => not found
    libXcomposite.so.1 => not found
    libXdamage.so.1 => not found
    libXext.so.6 => not found
    libXfixes.so.3 => not found
    libXrandr.so.2 => not found
    libexpat.so.1 => not found
    libxcb.so.1 => not found
    libxkbcommon.so.0 => not found
    libgbm.so.1 => not found
    libasound.so.2 => not found
    libcups.so.2 => not found
    libatspi.so.0 => not found
    libgcc_s.so.1 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libgcc_s.so.1 (0x00007f5fb86cd000)
    libc.so.6 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libc.so.6 (0x00007f5fb850c000)
    /lib64/ld-linux-x86-64.so.2 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib64/ld-linux-x86-64.so.2 (0x00007f5fc10e8000)

similar story with the notifier

ldd /tmp/portmaster/updates/linux_amd64/notifier/portmaster-notifier_v0-2-3
    linux-vdso.so.1 (0x00007fff4fd05000)
    libappindicator3.so.1 => not found
    libgtk-3.so.0 => not found
    libgdk_pixbuf-2.0.so.0 => not found
    libgio-2.0.so.0 => not found
    libgobject-2.0.so.0 => not found
    libglib-2.0.so.0 => not found
    libpthread.so.0 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libpthread.so.0 (0x00007f06fdd9e000)
    libc.so.6 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib/libc.so.6 (0x00007f06fdbdd000)
    /lib64/ld-linux-x86-64.so.2 => /nix/store/v8q6nxyppy1myi3rxni2080bv8s9jxiy-glibc-2.32-40/lib64/ld-linux-x86-64.so.2 (0x00007f06fddc3000)
dhaavi commented 3 years ago

The UI at localhost:817 just spins though:

That's because the browser does not have privileged API access to the Portmaster. You'll have to configure API Keys and then open http://127.0.0.1:817/api/v1/auth/basic to provide the key and get a session token. Alternatively, you can enable the Development Mode.

06kellyjac commented 3 years ago

Okiedokie, I'll try that out. As a note it might be nice if it detected failed requests due to not being authed, or just detect spinning for ages, and have a link to those docs pages

edit: all those settings seem to require controlling them in the ui..

ppacher commented 3 years ago

@06kellyjac you can enable devmode via the command line as well. Just run portmaster-core with -devmode. I.e. portmaster-start core --data /var/lib/portmaster -- -devmode. You can also use /etc/default/portmaster and add PORTMASTER_ARGS="-devmode" if your using our systemd service. Sorry for any typos, I'm on mobile with German auto-correction 🤦

06kellyjac commented 3 years ago

Ahh I ran sudo ./portmaster-start --data /tmp/portmaster core --help but not sudo ./portmaster-start --data /tmp/portmaster core -- --help. Danke @ppacher

Looking good:

image

Is /etc/default/portmaster a config file? Anywhere I can read more about it? If we end up writing a NixOS module to run portmaster in server mode we'll want to configure it decoratively with args or the config file.

06kellyjac commented 3 years ago

Ah I think I found the relevant line from the systemd service https://github.com/safing/portmaster-packaging/blob/master/linux/debian/portmaster.service#L21

ppacher commented 3 years ago

The file can basically just add additional arguments when starting portmaster. Almost all in-depth configuration is done through the UI and stored in different ways.

dit7ya commented 2 years ago

What is the status of the NixOS package at the moment?

06kellyjac commented 2 years ago

I've not touched it for quite a while. I'll need to refresh my brain on where it was and what issues were outstanding. Depending on if there's still significant difficulties it might be best to try a FHS wrapper

SuperSandro2000 commented 2 years ago

I am going to give this a try. We probably don't package the updater at all because that will not work with NixOS anyway.

06kellyjac commented 2 years ago

Portmaster is totally designed around the updater so you might need to make some big patches that would be best to upstream.

Goodluck either way :)

SuperSandro2000 commented 2 years ago

Portmaster is totally designed around the updater so you might need to make some big patches that would be best to upstream.

After digging a bit into the program I need to agree with you.

Depending on if there's still significant difficulties it might be best to try a FHS wrapper

That would be an easy solution but I don't really like it.

As you have probably seen already on the logs and maybe also heard, NixOS is a bit different and generic build binaries from other distros don't just work on it.

I build the portmaster go program locally but right after the start it downloaded other binaries which I already had build locally. That's a really big blocker if we want to build the program fully from source. Would it be possible to get support for pointing the binaries at other locations? That would really help us with the exec errors above. I could also patch around that and try to link the programs to where portmaster expects them but thats a bit hacky and will probably break at some point.

After that I would think about a way to get the electron app to work. Downloading lists from the internet is probably not a problem and something we would most likely keep. Also we would probably hardcode the datadir to /var/lib/portmaster.

Also dumping my nix file here

```nix { lib, buildGoModule, fetchFromGitHub }: buildGoModule rec { pname = "portmaster"; version = "0.8.13"; src = fetchFromGitHub { owner = "safing"; repo = "portmaster"; rev = "v${version}"; sha256 = "sha256-ZdvuNIza1LPck6WtIvx4MC3DQMhCA1vo6iH2YTc2TJ0="; }; vendorSha256 = "sha256-UBjoAb+zUcQzSra1mQFY09A/MDXPFr0To3NpkD7W16k="; ldflags = let t = "github.com/safing/portbase/info"; in [ "-s" "-w" "-X ${t}.commit=${src.rev}" "-X ${t}.buildOptions=unknown" "-X ${t}.buildUser=nixbld" "-X ${t}.buildHost=nix" "-X ${t}.buildDate=01.01.1970" "-X ${t}.buildSource=${src.url}" ]; # can't be run in the sandbox doCheck = false; meta = with lib; { description = "Puts you back in charge over all your computer's network connections"; homepage = "https://github.com/safing/portmaster"; license = licenses.agpl3; maintainers = with maintainers; [ SuperSandro2000 ]; }; } ```
lisandrocarmona commented 1 year ago

I had some fixing to do and enabled static linking for portmaster-start and portmaster-core in the process. The newest release of these two components is now statically linked. Unfortunately, statically linking the UI components is not feasible, as they bring loads of dependencies with them. I'm not sure how these would be best built for NixOS.

Hi, after reading all the thread, I'm not sure if we can download the binaries of the portmaster-start and portmaster-core and use it on Nixos. In other non-Nixos distros, I've tested the access to UI setting via Developer Mode (API) and browser, but I was not successful. Portmaster is being the last challenge for me to move into Nixos. Is there any news on making it compatible? Thanks.

lisandrocarmona commented 1 year ago

Some movements on this. I've managed to install portmaster with the following changes:

On configuration.nix:

  systemd.services.portmaster = {
    enable = true;
    unitConfig = {
      Description = "Portmaster by Safing";
      Documentation = [ "https://safing.io" "https://docs.safing.io" ];
      Before = "nss-lookup.target network.target shutdown.target";
      After = "systemd-networkd.service";
      Conflicts = [ "shutdown.target" "firewalld.service" ];
      Wants = "nss-lookup.target";
    };
    serviceConfig = {
      Type = "simple";
      Restart = "on-failure";
      RestartSec = "10";
      LockPersonality = "yes";
      MemoryDenyWriteExecute = "yes";
      NoNewPrivileges = "yes";
      PrivateTmp = "yes";
      PIDFile = "/root/portmaster/core-lock.pid";
      Environment = [ "LOGLEVEL=info" "PORTMASTER_ARGS" ];
      EnvironmentFile = "/etc/default/portmaster";
      ProtectSystem = "true";
      ReadWritePaths = [ "/var/lib/portmaster" "/root/portmaster" "/run/xtables.lock" ];
      RestrictAddressFamilies = "AF_UNIX AF_NETLINK AF_INET AF_INET6";
      RestrictNamespaces = "yes";
      ProtectHome = "read-only";
      ProtectKernelTunables = "yes";
      ProtectKernelLogs = "yes";
      ProtectControlGroups = "yes";
      PrivateDevices = "yes";
      AmbientCapabilities = "cap_chown cap_kill cap_net_admin cap_net_bind_service cap_net_broadcast cap_net_raw cap_sys_module cap_sys_ptrace cap_dac_override cap_fowner cap_fsetid";
      CapabilityBoundingSet = "cap_chown cap_kill cap_net_admin cap_net_bind_service cap_net_broadcast cap_net_raw cap_sys_module cap_sys_ptrace cap_dac_override cap_fowner cap_fsetid";
      SystemCallArchitectures = "native";
      SystemCallFilter = "@system-service @module";
      SystemCallErrorNumber = "EPERM";
      ExecStart = "/root/portmaster/portmaster-start --data /root/portmaster core -- $PORTMASTER_ARGS";
      ExecStopPost = "/root/portmaster/portmaster-start recover-iptables";
      };

Also on configuration.nix, I've installed two packages (similar to Arch-based or Debian-based distros). Make sure firewalld and firewalld-gui are NOT installed as they will conflict:

  environment.systemPackages = with pkgs; [
  libnetfilter_queue
  libappindicator-gtk3
  ...
  ]

On hardware-configuration.nix:

  fileSystems."/opt/safing/portmaster" = {
    device = "/root/portmaster";
    options = [ "bind" ];
  };

Then running on root:

    mkdir -p /root/portmaster
    wget -O /tmp/portmaster-start https://updates.safing.io/latest/linux_amd64/start/portmaster-start
    mv /tmp/portmaster-start /root/portmaster/portmaster-start
    chmod a+x /root/portmaster/portmaster-start
    /root/portmaster/portmaster-start update --data=/root/portmaster
    /root/portmaster/portmaster-start core

But then I can't move forward:

[control] 2022/12/26 11:50:24 starting /root/portmaster/updates/linux_amd64/core/portmaster-core_v1-0-4 --data /root/portmaster
221226 08:50:24.172 ▶ BOF
221226 08:50:30.106 les/worker:098 ▶ ERRO 001 resolver: service-worker name record delayed cache writer failed (1): could not start database cache (type bbolt): timeout - restarting in 2s
221226 08:50:30.271 v/location:299 ▶ WARN 003 netenv: failed to get IPv4 device location from traceroute: failed to send icmp packet: write ip4 0.0.0.0->1.1.1.1: sendto: operation not permitted
221226 08:50:30.313 sts/lookup:019 ▶ WARN 005 intel/filterlists: not searching for cache:intel/filterlists/lists/asn/15169 because filterlists not loaded
221226 08:50:30.313 sts/lookup:019 ▶ WARN 007 intel/filterlists: not searching for cache:intel/filterlists/lists/ipv6/2800:3f0:4001:828::200a because filterlists not loaded
221226 08:50:30.313 sts/lookup:019 ▶ WARN 009 intel/filterlists: not searching for cache:intel/filterlists/lists/country/(code) because filterlists not loaded
221226 09:06:47.562 solver-tcp:420 ▶ WARN 011 resolver: read error from Cloudflare (dot://cloudflare-dns.com:853#config): read tcp 192.168.1.117:60428->1.1.1.2:853: read: connection timed out
221226 09:08:45.324 solver-tcp:420 ▶ WARN 013 resolver: read error from Cloudflare (dot://cloudflare-dns.com:853#config): read tcp 192.168.1.100:12495->1.1.1.2:853: read: connection reset by peer

The messages:

operation not permitted
filterlists not loaded
read error from Cloudflare

Make me think that some declarations are missing, but I don't know which ones.

If anybody can help, I'll be grateful.

lisandrocarmona commented 1 year ago

Well, new tests. If I run an update command, everything works:

/root/portmaster/portmaster-start update
230103 15:49:18.703 ▶ BOF
230103 15:49:19.622 r/updating:165 ▶ INFO 001 updates: updated index all/intel/intel.json with 10 entries
230103 15:49:20.228 r/updating:089 ▶ INFO 002 updates: verified signature of https://updates.safing.io/stable.v2.json
230103 15:49:20.229 r/updating:165 ▶ INFO 003 updates: updated index stable.json with 48 entries
230103 15:49:20.229 r/updating:204 ▶ INFO 004 updates: everything up to date
230103 15:49:20.506 r/electron:054 ▶ INFO 005 updates: fixed SUID permission for chrome-sandbox
230103 15:49:20.516 ◀ EOF

However, if I try to start the program, I receive an error that the executable does not exist:

/root/portmaster/portmaster-start app
[control] 2023/01/03 18:49:42 starting /root/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-5/portmaster-app_v0-2-5 --data /root/portmaster
[control] 2023/01/03 18:49:42 app/portmaster-app.zip failed with: failed to start app/portmaster-app.zip: fork/exec /root/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-5/portmaster-app_v0-2-5: no such file or directory
Error: failed to start app/portmaster-app.zip: fork/exec /root/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-5/portmaster-app_v0-2-5: no such file or directory
4JX commented 1 year ago

However, if I try to start the program, I receive an error that the executable does not exist:

The error about the portmaster-app_v0-2-5 executable has to do with the interpreter, see patchelf: Before:

# file portmaster-app_v0-2-5
interpreter /lib64/ld-linux-x86-64.so.2

After:

# file portmaster-app_v0-2-5
interpreter /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/ld-linux-x86-64.so.2

Ideally the executable's build process/downloading should (instead of being downloaded from the internet) be done on a nix derivation to automatically apply the necessary patching though.

dnkmmr69420 commented 1 year ago

There should be an official nixos package

knarkzel commented 1 year ago

There should be an official nixos package

Where is this located?

storopoli commented 1 year ago

https://github.com/NixOS/nixpkgs

taotien commented 1 year ago

the PR adding it has been closed, not merged https://github.com/NixOS/nixpkgs/pull/203020

nyabinary commented 1 year ago

What currently blocking this from being a thing?

dhaavi commented 1 year ago

A package maintainer.

More precisely, the lack thereof.

nyabinary commented 11 months ago

Another attempt to try and package it: https://github.com/NixOS/nixpkgs/pull/264454

06kellyjac commented 11 months ago

Im pretty sure all the previous blockers around self-updating etc. mentioned by @SuperSandro2000 and myself still apply. The design is pretty anti-nixos really.

If someone wants to try to get it working properly in a FHS that's probably your best bet

nyabinary commented 11 months ago

Im pretty sure all the previous blockers around self-updating etc. mentioned by @SuperSandro2000 and myself still apply. The design is pretty anti-nixos really.

If someone wants to try to get it working properly in a FHS that's probably your best bet

Is there a guide for how to package something in a FHS?

nyabinary commented 11 months ago

So currently trying to figure out how to disable this by default: https://docs.safing.io/portmaster/settings#core/automaticUpdates

Sntx626 commented 9 months ago

I'll just add the resulting information of my try on running portmaster under nixos and add my thoughts to this issue's history.

I tried manually "installing" portmaster. Just so that I can verify that it can start.

Installation

I started by following the official docs for manual installation, however I modified the commands slightly, so here's what I used:

mkdir -p /opt/safing/portmaster

curl -o /tmp/portmaster-start https://updates.safing.io/latest/linux_amd64/start/portmaster-start
sudo mv /tmp/portmaster-start /opt/safing/portmaster/portmaster-start
sudo chmod a+x /opt/safing/portmaster/portmaster-start

# Download resources
sudo /opt/safing/portmaster/portmaster-start --data /opt/safing/portmaster update

That all worked without problems.

Running the Core Service

Running the core service also works without any patches.

sudo /opt/safing/portmaster/portmaster-start core

Starting the Portmaster App

However the problems start when you want to run the portmaster app.

/opt/safing/portmaster/portmaster-start app
[pmstart] 2023/12/18 04:56:38 starting /opt/safing/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-6/portmaster-app_v0-2-6 --data /opt/safing/portmaster --enable-features=UseOzonePlatform,WaylandWindowDecorations --ozone-platform=wayland
[pmstart] 2023/12/18 04:56:38 /opt/safing/portmaster/logs/app/2023-12-18-04-56-38.log: writing logs failed after 0 bytes: read |0: file already closed
[pmstart] 2023/12/18 04:56:38 app/portmaster-app.zip failed with: failed to start app/portmaster-app.zip: fork/exec /opt/safing/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-6/portmaster-app_v0-2-6: no such file or directory
Error: failed to start app/portmaster-app.zip: fork/exec /opt/safing/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-6/portmaster-app_v0-2-6: no such file or directory

I then proceeded to wrap it with nix-alien and that almost works.

nix run "github:thiagokokada/nix-alien#nix-alien" -- /opt/safing/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-6/portmaster-app_v0-2-6 --data /opt/safing/portmaster --enable-features=UseOzonePlatform,WaylandWindowDecorations --ozone-platform=wayland
Portmaster data directory: /opt/safing/portmaster
[3248805:1218/005810.641484:ERROR:object_proxy.cc(577)] Failed to call method: org.freedesktop.DBus.Properties.Get: object_path= /org/freedesktop/portal/desktop: org.freedesktop.DBus.Error.InvalidArgs: No such interface “org.freedesktop.portal.FileChooser”
[3248805:1218/005810.641523:ERROR:select_file_dialog_linux_portal.cc(280)] Failed to read portal version property

While I don't get a gui application to show up, the tray indicator still starts and works. This error might very well be the fault of my system/configuration though.

I've attached the buildFHSUserEnv expression generated by nix-alien:

{ pkgs ? import
    (builtins.fetchTarball {
      name = "nixpkgs-unstable-20231204192800";
      url = "https://github.com/NixOS/nixpkgs/archive/2c7f3c0fb7c08a0814627611d9d7d45ab6d75335.tar.gz";
      sha256 = "sha256-6ovz0pG76dE0P170pmmZex1wWcQoeiomUZGggfH9XPs=";
    })
    { }
}:

let
  inherit (pkgs) buildFHSUserEnv;
in
buildFHSUserEnv {
  name = "portmaster-app_v0-2-6-fhs";
  targetPkgs = p: with p; [
    alsa-lib.out
    at-spi2-atk.out
    cairo.out
    cups.lib
    dbus.lib
    expat.out
    glib.out
    gtk3.out
    libdrm.out
    libxkbcommon.out
    mesa.out
    nspr.out
    nss_latest.out
    pango.out
    xorg.libX11.out
    xorg.libXcomposite.out
    xorg.libXdamage.out
    xorg.libXext.out
    xorg.libXfixes.out
    xorg.libXrandr.out
    xorg.libxcb.out
  ];
  runScript = "/opt/safing/portmaster/updates/linux_amd64/app/portmaster-app_v0-2-6/portmaster-app_v0-2-6";
}

Accessing the Portmaster UI

To get to interact with portmaster, I restarted the core service in devmode to access the webui.

sudo /opt/safing/portmaster/portmaster-start core -- -devmode

or directly

sudo /opt/safing/portmaster/updates/linux_amd64/core/portmaster-core_v1-6-2 --data=/opt/safing/portmaster -devmode

This allows me to access the webui at localhost:817 and fully use portmaster.

image

My Thoughts

I think we could start with only packaging the core binary of portmaster. From that we can start module development by using the -devmode flag.

We should then figure out the module for the portmaster core part.

Since it doesn't seem to depend on dynamic libraries. We can also change the --data path to point to a location better suited to nixos.

The module should include a systemd service that automatically starts the portmaster core. Once we got that sorted out, portmaster is basically ready to use and configure.

After that, we should look into packaging the app and notifier part of portmaster. Possibly using the buildFHSUserEnv expression generated by nix-alien.

In my opinion we should start with getting something working first, so that multiple devs can try it out under nixos. Afterwards we can look into declaratively configuring portmaster, especially since that part might involve working with upstream (portmaster in this case).

Sntx626 commented 8 months ago

Part 2 - Hacking portmaster into your nixos configuration

Like my previous post, this post is aimed at getting something working first, then figuring out all the other problems (like making the expressions "properly pure").

Getting the portmaster-start binary and wrapping it

I first follow the "official installation steps" by downloading the portmaster-start binary and setting the execute bit: portmaster-start.

Afterwards I create wrapper script for the portmaster-start binary: portmaster-start-wrapped. All this does is ensuring that the dataDir is always set correctly (since I want to use /var/lib/portmaster).

Setting up portmaster-core as a system service

The set-up the portmaster-core as a system service is simply copying the official systemd service and wrapping it as a nixos module: pkgs-sys/portmaster.nix.

(Systemd somehow refuses to start portmaster automatically after a reboot, that's why the after, before, conflicts and wants is a bit chaotic).

Notice how I set -devmode in L57. That's because I couldn't get the portmaster-app to work (more on that in the following chapter).

Linking portmaster-app and portmaster-notifier into userspace

To get the portmaster-notifier (and portmaster-app in the future) into userspace, I simply use the already created portmaster-start-wrapped script: pkgs-usr/portmaster.nix.

The portmaster-app doesn't run with the previously mentioned error. Since the error might be with my system/config and not with portmaster, it would be nice if other people could try to replicate the error.

Future steps

Personally my next step would be finishing the systemd service for portmaster-notifier to automatically start when reaching graphical-session.target or comparable.

I think further milestones should be:

nyabinary commented 8 months ago

Part 2 - Hacking portmaster into your nixos configuration

Like my previous post, this post is aimed at getting something working first, then figuring out all the other problems (like making the expressions "properly pure").

Getting the portmaster-start binary and wrapping it

I first follow the "official installation steps" by downloading the portmaster-start binary and setting the execute bit: portmaster-start.

Afterwards I create wrapper script for the portmaster-start binary: portmaster-start-wrapped. All this does is ensuring that the dataDir is always set correctly (since I want to use /var/lib/portmaster).

Setting up portmaster-core as a system service

The set-up the portmaster-core as a system service is simply copying the official systemd service and wrapping it as a nixos module: pkgs-sys/portmaster.nix.

(Systemd somehow refuses to start portmaster automatically after a reboot, that's why the after, before, conflicts and wants is a bit chaotic).

Notice how I set -devmode in L57. That's because I couldn't get the portmaster-app to work (more on that in the following chapter).

Linking portmaster-app and portmaster-notifier into userspace

To get the portmaster-notifier (and portmaster-app in the future) into userspace, I simply use the already created portmaster-start-wrapped script: pkgs-usr/portmaster.nix.

The portmaster-app doesn't run with the previously mentioned error. Since the error might be with my system/config and not with portmaster, it would be nice if other people could try to replicate the error.

Future steps

Personally my next step would be finishing the systemd service for portmaster-notifier to automatically start when reaching graphical-session.target or comparable.

I think further milestones should be:

* Wrapping `portmaster-app` into an fhs wrapper

* Natively building

  * `portmaster-core`
  * `portmaster-app`
  * `portmaster-notifier`

* Creating a script for `portmaster-start recover-iptables`

I have a pr https://github.com/NixOS/nixpkgs/pull/264454 to package it, but I'm currently a bit stuck, so if you want to help, that would be great :)