search

BastilleBSD / bastille

Bastille is an open-source system for automating deployment and management of containerized applications on FreeBSD.
https://bastillebsd.org
BSD 3-Clause "New" or "Revised" License
783 stars 130 forks source link

Bastille console does not work after upgrading jail to 14.1-RELEASE #704

Open paulz-gh opened 4 weeks ago

paulz-gh commented 4 weeks ago

[MANDATORY] Describe the bug [MANDATORY] I upgraded my system from 13.3 to 14.1-RELEASE, including pkg-static update -f. I have run bastille bootstrap 14.1-RELEASE.

When I update a jail by changing the mount from 13.3-RELEASE to 14.1-RELEASE the jail seems to start but the bastille console command does not connect me to the console. Reverting the fstab entry back to 13.3-RELEASE makes the bastille console command work again.

[MANDATORY] Bastille and FreeBSD version (paste bastille -v && freebsd-version -kru output) $ bastille -v && freebsd-version -kru 0.10.20231125 14.1-RELEASE 14.1-RELEASE 14.1-RELEASE

[MANDATORY] How did you install bastille? (port/pkg/git)

pkg [optional] Steps to reproduce?

[13:14:39:freebsd:paulz:/var/log/bastille]
$ doas bastille  start unifi
pfctl: Invalid argument.
[unifi]:
unifi: created

[13:14:58:freebsd:paulz:/var/log/bastille]
$ doas bastille  console unifi
[unifi]:

[13:15:04:freebsd:paulz:/var/log/bastille]
$

[optional] Expected behavior Console prompt for jail console

[optional] Screenshots If applicable, add screenshots to help explain your problem.

[optional] Additional context

I can start a shell inside the jail using jexec $JID sh. The unifi service I want to run inside that jail does not seem to be started and when I start it I get the error:

# service unifi start
Starting unifi.
su: pam_start: System error
/usr/local/etc/rc.d/unifi: WARNING: failed to start unifi
michael-o commented 3 weeks ago

Does a fresh jail work?

paulz-gh commented 3 weeks ago

Yes a new jail created with bastille -C 14.1-RELEASE works. I tried one of my other 13.3-RELEASE jails and that also fails. So it's not just the one I tried first.

I also created a new 13.3-RELEASE jail and edited fstab to point to 14.1-RELEASE. That also fails to get a console.

The console log for that jail shows no errors: First entries are for 13.3 last is for 14.1 $ cat /var/log/bastille/testjail3_console.log ELF ldconfig path: /lib/casper /lib /usr/lib /usr/lib/compat 32-bit compatibility ldconfig path: /usr/lib32 /usr/lib32 Updating motd:. Creating and/or trimming log files. Clearing /tmp (X related). Updating /var/run/os-release done. Starting syslogd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron.

Thu Jun 6 19:10:45 CEST 2024 Stopping cron. Waiting for PIDS: 57930. . Terminated ELF ldconfig path: /lib/casper /lib /usr/lib /usr/lib/compat 32-bit compatibility ldconfig path: /usr/lib32 /usr/lib32 Updating motd:. Creating and/or trimming log files. Clearing /tmp (X related). Updating /var/run/os-release done. Starting syslogd. Starting cron.

Thu Jun 6 19:10:47 CEST 2024 Stopping cron. Waiting for PIDS: 58718. . Terminated ELF ldconfig path: /lib /usr/lib /usr/lib/compat 32-bit compatibility ldconfig path: /usr/lib32 /usr/lib32 Updating motd:. Creating and/or trimming log files. Clearing /tmp (X related). Updating /var/run/os-release done. Starting syslogd. Starting cron.

Thu Jun 6 19:12:02 CEST 2024

paulz-gh commented 3 weeks ago

I did some testing and I can start a root shell in updated jail but login command fails. I see following in /var/log/messages: Jun 7 11:09:51 unifi login[11869]: in openpam_load_module(): no pam_opie.so found Jun 7 11:09:51 unifi login[11869]: pam_start(): System error Jun 7 11:12:07 unifi login[11932]: in openpam_load_module(): no pam_opie.so found Jun 7 11:12:07 unifi login[11932]: pam_start(): System error

Looks like a missing pam module. I have copied /etc/pam.d/* to the jail /etc/pam.d and now console works.

michael-o commented 3 weeks ago

As far as I remember pam_opie has ben removed/replaced. Did freebsd-udpate fail here? See 14.0 RNs: The one-time password facility OPIE, opie(4), has been removed from the base system. If you still wish to use it, install the security/opie port. Otherwise, make sure to remove or comment out any mention of pam_opie and pam_opieaccess from your PAM policies. etcupdate(8) will normally take care of this for the stock policies. 0aa2700123e2

I don't believe that this is a Bastille issue.

paulz-gh commented 3 weeks ago

Hi, Maybe I missed something but the procedure on https://bastille.readthedocs.io/en/latest/chapters/upgrading.html#revert-upgrade-downgrade-process does not mention anything about updating the /etc content inside the jails. The jails /etc/pam.d/* references to opie cause the login command in the jail to fail.

michael-o commented 3 weeks ago

Hi, Maybe I missed something but the procedure on https://bastille.readthedocs.io/en/latest/chapters/upgrading.html#revert-upgrade-downgrade-process does not mention anything about updating the /etc content inside the jails. The jails /etc/pam.d/* references to opie cause the login command in the jail to fail.

Consider it incomplete. It is imperative to study official RNs before upgrading...

paulz-gh commented 3 weeks ago

Hi, Maybe I missed something but the procedure on https://bastille.readthedocs.io/en/latest/chapters/upgrading.html#revert-upgrade-downgrade-process does not mention anything about updating the /etc content inside the jails. The jails /etc/pam.d/* references to opie cause the login command in the jail to fail.

Consider it incomplete. It is imperative to study official RNs before upgrading...

Agreed. But maybe good to add a reminder for that on that page....

michael-o commented 3 weeks ago

Hi, Maybe I missed something but the procedure on https://bastille.readthedocs.io/en/latest/chapters/upgrading.html#revert-upgrade-downgrade-process does not mention anything about updating the /etc content inside the jails. The jails /etc/pam.d/* references to opie cause the login command in the jail to fail.

Consider it incomplete. It is imperative to study official RNs before upgrading...

Agreed. But maybe good to add a reminder for that on that page....

Given that no issues were addressed for the past 7+ months and the maintainers have disappeared I don't expect this to be fixed anytime soon.

stafwag commented 3 weeks ago

I had the same issue a few months back.

This might be related to the /etc/ directory isn't updated. So it might be that etcupdated; https://man.freebsd.org/cgi/man.cgi?etcupdate needs to be executed.

There is already a PR open for this: https://github.com/BastilleBSD/bastille/pull/660

You can still access the "console by' executing jexec.

# jls
# jexec <JID> /bin/sh

I switched from thin jails to tick jails and this also resolves the issues as /etc is updated as part of the tick jail upgrade process.