minipot not working on recent FreeBSD installs.

[cweimann]

I tried this on both FreeBSD 14.0 and 13.2. I suspect the problem is the consul config (new version of consul) and nomad doesn't start because there is no consul running. The below run is from the FreeBSD 14.0 box.

root@hp800g3:/home/cweimann # pkg info | egrep "consul|nomad|traefik|minipot|pot"
consul-1.16.2                  Service discovery and configuration made easy
minipot-0.2.3                  Mono-server installation of pot, nomad and consul
nomad-1.6.2                    Cluster manager and scheduler
nomad-pot-driver-0.9.1         Nomad driver to support pot jails
pot-0.15.6                     Container framework for FreeBSD
potnet-0.4.4_34                Utility to help sysutils/pot to manage the internal network
traefik-2.10.5                 High availability reverse proxy and load balancer
root@hp800g3:/home/cweimann #
root@hp800g3:/home/cweimann # minipot-init
0
Creating a backup of your /etc/rc.conf
/etc/rc.conf -> /etc/rc.conf.bkp-pot
syslogd_flags: -b 127.0.0.1 -b 10.192.0.1 -a 10.192.0.0/10 -> -b 127.0.0.1 -b 10.192.0.1 -a 10.192.0.0/10
Creating a backup of your /etc/pf.conf
/etc/pf.conf -> /etc/pf.conf.bkp-pot
auto-magically editing your /etc/pf.conf
Please, check that your PF configuration file /etc/pf.conf is still valid!
nomad_user:  -> root
nomad_env:  -> PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/sbin:/bin
nomad_args:  -> -config=/usr/local/etc/nomad/minipot-server.hcl
consul_enable:  -> YES
nomad_enable:  -> YES
traefik_enable:  -> YES
traefik_conf:  -> /usr/local/etc/minipot-traefik.toml
root@hp800g3:/home/cweimann # minipot start
-sh: minipot: not found
root@hp800g3:/home/cweimann # minipot-start
Stopping syslogd.
Waiting for PIDS: 1167.
Starting syslogd.
Stopping cron.
Waiting for PIDS: 1455.
Starting cron.
Starting consul.
Starting nomad.
Starting traefik.
root@hp800g3:/home/cweimann #
root@hp800g3:/home/cweimann # sockstat -4l
USER     COMMAND    PID   FD  PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
traefik  traefik    12974 7   tcp46  *:8080                *:*
traefik  traefik    12974 8   tcp46  *:9002                *:*
root     syslogd    12909 6   udp4   127.0.0.1:514         *:*
root     syslogd    12909 7   udp4   10.192.0.1:514        *:*
root     sshd        1451 4   tcp4   *:22                  *:*
root@hp800g3:/home/cweimann #

[pizzamig]

Hi. I have the same version of consul and no problem with that. Can you please share /usr/local/etc/consul.d/minipot-agent.jsonand the output of the command grep ^consul /etc/rc.conf minipot is also going to configure the system to write logs in /var/log/consul/consul.log

[grembo]

@cweimann I tried here as well and it worked out of the box. So maybe the environment is polluted with other configurations or something about the setup causes some hashicorp fingerprinter to go wrong (#wildspeculation). Maybe sharing the output of ifconfig and cat /etc/rc.conf and cat /etc/pf.conf could help to figure out what's going on.

[cweimann]

Doing a fresh FreeBSD install.

"pot vnet-init" doesn't work. I think that ought to be "pot vnet-start". Ok, I found something. After "pot de-init" the bridge is still there and minipot-init must get confused by that. Skipping that and going straight to minipot-init and I now get consul and traefik but no nomad. Still looking....

Ok, this commit to nomad's startup script on FreeBSD broke it.

https://cgit.freebsd.org/ports/commit/sysutils/nomad/files/nomad.in?id=1e8c5858e52fcb3b1664e67740b1f1396ea6ac5d

Specifically the nomad_command setting. I added -data-dir=${nomad_dir} back in. They way they have changed it your setting of nomad_args winds up removing the nomad data-dir setup and nomad won't start.

So, my first problem was that I did the klara bits testing pot and wound up with a bridge that confused minipot-init. After fixing that I found a problem with the FreeBSD nomad package.

I can only presume things worked for you because you were didn't do the pot test bits and had packages from before 2023/09/22.

Knowing it worked for you was helpful :).

[cweimann]

ok... nginx.job doesn't work. It just spins on "Deployment "c5ef80af" in progress..." It seems that it uses image = "https://pot-registry.zapto.org/registry/" and that hostname dosen't resolve.

[grembo]

@cweimann I'd suggest to use the job description from

https://github.com/bsdpot/nomad-pot-driver?tab=readme-ov-file#complete-job-example