Linux jail with alias cannot communicate with other FreeBSD jails without alias in the same virtual network.

[OrvilleQ]

Hello.

I used a modified version of https://github.com/AppJail-makejails/alpine-linux to deploy alpine linux onto my arm64 machine. Things working great but I cannot access other jail in the same virtual network due to alias, but the application I deployed on the linux jail depend on postgresql which is running on a FreeBSD jail.

I tried to combine alias with multiple virtual network but the jail refuse to start, bridge also seems not working on Linux jail either. I tried lots of combination but no one work.

So is there a way make Linux jail using alias communicate with other jails in the same virtual network? Or we still need more develop to achieve that? Or it's even impossible?

[DtxdF]

Hi @OrvilleQ

Since LinuxJails use aliases, communication must be with NAT or with all jails using the same interface through aliases. Aliasing has some drawbacks, that's why many users use VNET as it offers isolation and other cool things.

Of course, there is a way to cover this case:

# kldload linux linux64 fdescfs tmpfs linprocfs linsysfs pty
...
# appjail makejail \
        -j alpine \
        -f gh+AppJail-makejails/alpine-linux \
        -o template=/usr/local/share/examples/appjail/templates/linux.conf \
        -o alias \
        -o virtualnet=":appjail0 address:10.0.0.50 default" \
        -o nat
...
# appjail jail list
STATUS  NAME         TYPE               VERSION       PORTS     NETWORK_IP4
UP      alpine       generic            -             -         10.0.0.50
UP      flatnotes    thin               14.0-RELEASE  -         10.0.0.4
UP      ubuntu       linux+debootstrap  jammy         -         10.0.0.5
UP      filebrowser  thin               14.0-RELEASE  8080/tcp  10.0.0.2

At this point you can only communicate with the outside, but with the following rule in your pf.conf(5), you can communicate with other jails:

/etc/pf.conf:

nat-anchor "appjail-nat/jail/*"
nat-anchor "appjail-nat/network/*"
rdr-anchor "appjail-rdr/*"

nat on ajnet inet from 10.0.0.50 to 10.0.0.0/10 -> 10.0.0.1

That's why I use the virtualnet="... address:10.0.0.50 ..." option, to set a static and specific IPv4 address within the virtual network, so it's much easier to make a new rule.

Reload pf(4)'s rules:

service pf reload

And try if you can communicate with any other jail:

# appjail cmd jexec alpine ping -c4 10.0.0.4
PING 10.0.0.4 (10.0.0.4): 56 data bytes
64 bytes from 10.0.0.4: seq=0 ttl=64 time=0.086 ms
64 bytes from 10.0.0.4: seq=1 ttl=64 time=0.057 ms
64 bytes from 10.0.0.4: seq=2 ttl=64 time=0.052 ms
64 bytes from 10.0.0.4: seq=3 ttl=64 time=0.057 ms

--- 10.0.0.4 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
# appjail cmd jexec alpine ping -c4 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=113 time=104.831 ms
64 bytes from 8.8.8.8: seq=1 ttl=113 time=129.898 ms
64 bytes from 8.8.8.8: seq=2 ttl=113 time=111.267 ms
64 bytes from 8.8.8.8: seq=3 ttl=113 time=105.241 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 104.831/112.809/129.898 ms

Of course, I'm hardcoding the addresses, but we can accomplish the same thing using AppJail's tools and scripts to get some data, but it adds complexity that you may not need or want.

---

I'll add this use case as soon as possible to AppJail Documentation. Please let me know if it solves your problem.

Thanks for reporting!

Note: Remember that even though Linuxlator allows us to run LinuxApps, this is a very, very experimental feature of FreeBSD, so maybe your app worked fine yesterday, but not today. I recommend that you port your application to FreeBSD or use a VM with a lightweight Linux distro like Alpine Linux.