Closed Loader23 closed 3 years ago
MichaIng
commented
3 years ago Many thanks for your report.
I don't know much about DFS, it's intended to be mountable as Samba/CIFS mount? How does the fstab entry for that mount look like, respectively the mount command arguments?
Loader23
commented
3 years ago Yes, normal cifs mount. Thats the entry:
//IP/Folder /mnt/Folder cifs username=user,password=pass,iocharset=utf8,uid=dietpi,gid=dietpi,file_mode=0770,dir_mode=0770,vers=3.1.1,_netdev,nodfs,nofail,noauto,x-systemd.automount
I just saw the "nodfs" flag but removing it makes no difference.
MichaIng
commented
3 years ago And what is the output if you do:
mount /mnt/FolderCouldn't chdir to /mnt/Folder: No such device
Joulinar
commented
3 years ago I guess Folder is/was a placeholder and would need to be replace with the existing target folder located in /mnt
Loader23
commented
3 years ago I have of course replaced the placeholder :D
MichaIng
commented
3 years ago First of all, pinging that CIFS share IP from the DietPi system works, right?
Can you try to remove the nodfs option? It seems to have been required for very old servers but is not even listed on current man pages as valid option, should hence definitely not be required for SMB 3.1.1 protocol. If that does not help, please try to remove the vers=3.1.1 option as well to allow auto-negotiation.
Loader23
commented
3 years ago Yes a ping is working fine. Removing the nodfs did not work and removing the vers is also not working.
I just tested it with another Pi on version 6.33.3. The Pi never had a mount before. I added the line to fstab and its working fine. I Updated to 6.34.3 -> broken.
Joulinar
commented
3 years ago what happen if on the Pi running 6.33.3 you are going to do apt update && apt upgrade without changing DietPi version. What kernel you are running on 6.33.3? uname -a before doing any update?
Loader23
commented
3 years ago That was the last Pi i had on 6.33.3 :-(
Joulinar
commented
3 years ago no backup? no saved image stored?
Loader23
commented
3 years ago I have an Image of dietpi 6.32.x but installing that would also update it to the latest version at startup.
MichaIng
commented
3 years ago EDIT: Ah you mean it's an image that has never been bootet yet. Okay, then the blow does not work. Although you could boot it without network access, so that first run update/setup cannot be done, then exit to console. Then you can enable network and do the APT upgrades manually.
Please run apt update && apt upgrade on that system only, at best after a backup 😉, and store the list of upgraded packages somewhere to hopefully find the culprit. Since the last DietPi update did not include any related changes, I'm also pretty sure that it is an APT package related to SMB/CIFS mounts, or the kernel upgrade from 5.4 to 5.10.
EDIT2: Ah, can you please paste cat /var/tmp/dietpi/logs/dietpi-update.log from any of the affected systems?
Loader23
commented
3 years ago Maybe i have an Update from 6.33.3.^^ I have to check that tomorrow, the device is offline now. I will post the kernel info then. Here is the dietpi-update.log:
Joulinar
commented
3 years ago it's doing the Raspberry Pi kernel update. propaly we could try to switch back to 5.4. It's highly recommended to clone your SD card before or do an appropriate backup. 😉
apt update
apt install rpi-update
sudo rpi-update 0642816ed05d31fb37fc8fbbba9e1774b475113f
rebootThis will downgrade the kernel to 5.4.79
Loader23
commented
3 years ago The downgrade to 5.4.79 did not help. But I could recover that one Pi to 6.33.3, the kernel is 5.4.72 there. After downgrade to 5.4.72 its still not working.
Joulinar
commented
3 years ago Ok If you are ok let's try some more test. Keep the backup of your system on a save spot. On the current system running v6.33.3, just do an apt update && apt upgrade
Loader23
commented
3 years ago After apt update && apt upgrade + reboot its not working anymore.
Joulinar
commented
3 years ago Than it is one of the apt packages and not the DietPi update. Now we would need to find out which one. Can you go back to the backup before running apt upgrade? And share the list of packages to be upgraded (without actually upgrading them)
Loader23
commented
3 years ago Here it is :-)
apt base-files ca-certificates chromium-browser chromium-codecs-ffmpeg-extra device-tree-compiler firmware-atheros firmware-brcm80211 firmware-iwlwifi firmware-misc-nonfree firmware-realtek iproute2 libapt-pkg5.0 libcups2
libdns-export1104 libegl-mesa0 libegl1-mesa libgbm1 libgl1-mesa-dri libglapi-mesa libglx-mesa0 libgnutls30 libisc-export1100 libjpeg62-turbo libldap-2.4-2 libldap-common libp11-kit0 libraspberrypi-bin libraspberrypi0 libsqlite3-0
libssl1.1 libsystemd0 libudev1 libxml2 libzstd1 openssl raspberrypi-bootloader raspberrypi-kernel raspberrypi-sys-mods rpi-eeprom sudo systemd systemd-sysv tzdata udev unzip xserver-common xserver-xorg-corequite a long list 😃
@MichaIng would it make sense to try to install one by one to see where it is failing?
Loader23
commented
3 years ago After installing raspberrypi-kernel it is no longer working. Does this Log help?
With the 6.33.3 Pi I can update the Kernel to 5.4.81 and its still working. At least until kernel 5.10.0, then its broke again. But a downgrade on a fresh Dietpi Install is not working.
MichaIng
commented
3 years ago So upgrading the kernel alone broke it, but downgrading the kernel alone did not fix it 🤔? But you did a reboot after kernel upgrade/downgrade, didn't you?
Loader23
commented
3 years ago Yes, of course.^^ Downgrading the 6.33.3 did work though, downgrading a fresh install did not.
Joulinar
commented
3 years ago Is this something to report back to Raspberry Foundation as issue? Interesting would be as well how plain Raspberry OS behave. Before and after apt upgrade. As well as checking the kernel downgrade.
MichaIng
commented
3 years ago Interesting would be as well how plain Raspberry OS behave.
I agree. I'm unsure how to setup such a Windows DFS to replicate/test, which would be great before forwarding to RPi devs. Probably we find some related report on their forum or GitHub.
Ah, when mounting fails, does dmesg show any related kernel log entry (at the end of the output)?
Loader23
commented
3 years ago These are the last lines of dmesg after a reboot:
[ 14.846642] CIFS: Attempting to mount //IP/Folder
[ 15.654995] CIFS: VFS: cifs_mount failed w/return code = -2
[ 15.727164] CIFS: Attempting to mount //IP/Folder
[ 16.288569] CIFS: VFS: cifs_mount failed w/return code = -126Mount error codes: https://manpages.debian.org/buster/mount/mount.8.en.html#RETURN_CODES
The first error could be related to missing network. Can you please paste the following:
journalctl -u mnt-Folder.mount -u mnt-Folder.automount -u network.target -u network-online.targetAll parts of the mount point path need to be concatenated with a dash, so /mnt/this/point would be mnt-this-point.mount and mnt-this-point.automount.
Probably a change we do with upcoming release does already fix that part:
mkdir -p /etc/systemd/system/ifup@.service.d
curl -sSfL 'https://raw.githubusercontent.com/MichaIng/DietPi/dev/rootfs/etc/systemd/system/ifup%40.service.d/dietpi.conf' > /etc/systemd/system/ifup@.service.d/dietpi.conf_netdev mount option should assure that the mount is not done before network-online.target is reached.network-online.target is not reached before all network adapters have been fully configured, hence give that target much more meaning.But actually, the x-systemd.automount mount option means that the share is not mounted before the mount point is accessed the first time. Do you have any service or task at boot that accesses that mount so early (~15 seconds after boot?). Usually such network shares are accessed somehow later, which is why the issue with missing network rarely shows up.
But mounting the drive fails as well later + we have a second error code (126), which is a combination of all errors but "incorrect invocation or permissions (1)", not helpful 😄.
Let's have a look at the sec mount option, which is about how the password is transferred and where the default seems to depend on the kernel version: https://manpages.debian.org/testing/cifs-utils/mount.cifs.8.en.html#OPTIONS
sec=arg
Security mode. Allowed values are: none - attempt to connection as a null user (no name) krb5 - Use Kerberos version 5 authentication krb5i - Use Kerberos authentication and forcibly enable packet signing ntlm - Use NTLM password hashing ntlmi - Use NTLM password hashing and force packet signing ntlmv2 - Use NTLMv2 password hashing ntlmv2i - Use NTLMv2 password hashing and force packet signing ntlmssp - Use NTLMv2 password hashing encapsulated in Raw NTLMSSP message ntlmsspi - Use NTLMv2 password hashing encapsulated in Raw NTLMSSP message, and force packet signing
The default in mainline kernel versions prior to v3.8 was sec=ntlm. In v3.8, the default was changed to sec=ntlmssp.
If the server requires signing during protocol negotiation, then it may be enabled automatically. Packet signing may also be enabled automatically if it's enabled in /proc/fs/cifs/SecurityFlags.
I was trying to find out whether there was a change recently: https://wiki.samba.org/index.php/LinuxCIFSKernel#5.5_kernel_.28module_version_2.24.2C_released_January_26th.2C_2020._61_changesets.29
5.9 kernel (module version 2.28 30 changesets)
Fix idsfromsid mount option when creating directories. Various DFS (global namespace) fixes.
But no recent word about sec default. However, could you try:
mount -o sec=ntlmssp /mnt/Folderand in case go through the other possible options from the list above? Else we must assume that the "various DFS fixes" contain a regression.
Loader23
commented
3 years ago Journal:
-- Logs begin at Wed 2021-02-24 15:15:07 CET, end at Wed 2021-02-24 16:27:16 CET. --
Feb 24 15:15:10 Pi01 systemd[1]: Reached target Network.
Feb 24 15:15:32 Pi01 systemd[1]: Reached target Network is Online.I have to test the change you have posted tomorrow (device offline now) but mount -o sec=ntlmssp also resulted in No such device
MichaIng
commented
3 years ago journalctl -u mnt-Folder.mount -u mnt-Folder.automount doesn't output anything (with adjusted path elements)?
Does systemctl status mnt-Folder.mount -u mnt-Folder.automount show something instead?
I have to test the change you have posted tomorrow (device offline now)
Okay. Probably I find time to setup a DFS here myself in the meantime.
Loader23
commented
3 years ago Sorry, I missread the dash part.^^Thats the Log:
Feb 24 15:15:10 Pi01 systemd[1]: Reached target Network.
Feb 24 15:15:32 Pi01 systemd[1]: mnt-Folder.automount: Got automount request for /mnt/Folder, triggered by 447 (cp)
Feb 24 15:15:32 Pi01 systemd[1]: Reached target Network is Online.
Feb 24 15:15:32 Pi01 systemd[1]: Mounting /mnt/Folder...
Feb 24 15:15:33 Pi01 mount[449]: mount error(2): No such file or directory
Feb 24 15:15:33 Pi01 mount[449]: Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Feb 24 15:15:33 Pi01 systemd[1]: mnt-Folder.mount: Mount process exited, code=exited, status=32/n/a
Feb 24 15:15:33 Pi01 systemd[1]: mnt-Folder.mount: Failed with result 'exit-code'.
Feb 24 15:15:33 Pi01 systemd[1]: Failed to mount /mnt/Folder.
Feb 24 15:15:33 Pi01 systemd[1]: mnt-Folder.automount: Got automount request for /mnt/Folder, triggered by 497 (find)
Feb 24 15:15:33 Pi01 systemd[1]: Mounting /mnt/Folder...
Feb 24 15:15:34 Pi01 mount[498]: mount error(126): Required key not available
Feb 24 15:15:34 Pi01 mount[498]: Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Feb 24 15:15:34 Pi01 systemd[1]: mnt-Folder.mount: Mount process exited, code=exited, status=32/n/a
Feb 24 15:15:34 Pi01 systemd[1]: mnt-Folder.mount: Failed with result 'exit-code'.
Feb 24 15:15:34 Pi01 systemd[1]: Failed to mount /mnt/Folder.
Feb 24 15:33:36 Pi01 systemd[1]: mnt-Folder.automount: Got automount request for /mnt/Folder, triggered by 777 (sftp-server)
Feb 24 15:33:36 Pi01 systemd[1]: Mounting /mnt/Folder...
Feb 24 15:33:36 Pi01 mount[778]: mount error(2): No such file or directory
Feb 24 15:33:36 Pi01 mount[778]: Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Feb 24 15:33:36 Pi01 systemd[1]: mnt-Folder.mount: Mount process exited, code=exited, status=32/n/a
Feb 24 15:33:36 Pi01 systemd[1]: mnt-Folder.mount: Failed with result 'exit-code'.
Feb 24 15:33:36 Pi01 systemd[1]: Failed to mount /mnt/Folder.
Feb 24 15:34:40 Pi01 systemd[1]: mnt-Folder.automount: Got automount request for /mnt/Folder, triggered by 523 (bash)
Feb 24 15:34:40 Pi01 systemd[1]: Mounting /mnt/Folder...
Feb 24 15:34:41 Pi01 mount[798]: mount error(2): No such file or directory
Feb 24 15:34:41 Pi01 mount[798]: Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Feb 24 15:34:41 Pi01 systemd[1]: mnt-Folder.mount: Mount process exited, code=exited, status=32/n/a
Feb 24 15:34:41 Pi01 systemd[1]: mnt-Folder.mount: Failed with result 'exit-code'.
Feb 24 15:34:41 Pi01 systemd[1]: Failed to mount /mnt/Folder.
Feb 24 15:34:41 Pi01 systemd[1]: mnt-Folder.automount: Got automount request for /mnt/Folder, triggered by 523 (bash)
Feb 24 15:34:41 Pi01 systemd[1]: Mounting /mnt/Folder...
Feb 24 15:34:41 Pi01 mount[803]: mount error(126): Required key not available
Feb 24 15:34:41 Pi01 mount[803]: Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Feb 24 15:34:41 Pi01 systemd[1]: mnt-Folder.mount: Mount process exited, code=exited, status=32/n/a
Feb 24 15:34:41 Pi01 systemd[1]: mnt-Folder.mount: Failed with result 'exit-code'.
Feb 24 15:34:41 Pi01 systemd[1]: Failed to mount /mnt/Folder.
Feb 24 16:26:21 Pi01 systemd[1]: mnt-Folder.automount: Got automount request for /mnt/Folder, triggered by 1431 (mount.cifs)
Feb 24 16:26:21 Pi01 systemd[1]: Mounting /mnt/Folder...
Feb 24 16:26:21 Pi01 mount[1432]: mount error(2): No such file or directory
Feb 24 16:26:21 Pi01 mount[1432]: Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Feb 24 16:26:21 Pi01 systemd[1]: mnt-Folder.mount: Mount process exited, code=exited, status=32/n/a
Feb 24 16:26:21 Pi01 systemd[1]: mnt-Folder.mount: Failed with result 'exit-code'.
Feb 24 16:26:21 Pi01 systemd[1]: Failed to mount /mnt/Folder.Just to clearify, my test device is offline but i can read out logs on one of the live machines I just don't want to change something on it and break it.^^
MichaIng
commented
3 years ago Okay, so indeed there is a cp and a find command during boot that trigger the automount, so adding the ifup@.service configuration might solve part of the issue. Probably you can identify the service/script that accesses that mount during boot and can delay it, to not have it facing an empty mount point unexpectedly. But the kernel-related one is to be investigated.
Loader23
commented
3 years ago The cp and find command during boot is caused by commands inside /var/lib/dietpi/dietpi-autostart/custom.sh for the Kiosk System. Installing ifup@.service.d and rebooting did not fix the cp and find errors. But there is also a cron job for that so I can live with that.^^
MichaIng
commented
3 years ago I made some tests and even with x-systemd.automount, any access to a network mount point, that triggers a mount request, is delayed until the network is online (the target reached). I even caused hanging test systems with this by adding a mount point access to our preboot script, which then was waiting forever until network, since network on the other hand waited for preboot to finish 😄.
However, let's try to completely rule out any network init related effect by disabling that service temporarily and see if a single manual access still fails to mount the share properly:
systemctl disable dietpi-autostart_custom
reboot
ls -l /mnt/Folder
df
journalctl -u mnt-Folder.mount -u mnt-Folder.automountI have set up a fresh dietpi with no autostart and the mount does still fail. No matter if setup via manual fstab entry or via drive manager.
MichaIng
commented
3 years ago Okay, and downgrading the kernel does as well not resolve it on that system?
Loader23
commented
3 years ago No :-(
MichaIng
commented
3 years ago You tried the other sec= options as well, right?
I had a look through the other upgraded packages an libgnutls30 is used by the Samba library. Luckily the older package version is available in Debian snapshots. It won't be armv6hf but armv7hf, but in case or RPi 4 that shouldn't matter:
cd /tmp
wget 'https://snapshot.debian.org/archive/debian/20210101T024031Z/pool/main/g/gnutls28/libgnutls30_3.6.7-4%2Bdeb10u5_armhf.deb'
dpkg -i libgnutls30_*_armhf.deb
rm libgnutls30_*_armhf.debEverything else looks completely unrelated to me. libcups2 is used by the library as well, but for printer servers. Of course one could go through the whole package list and downgrade everything from the snapshots archive, as something must have changed. I had another look through the patches of DietPi v6.34 but there is nothing even close to be related.
So I'm out of ideas then 😢.
Since we do not know whether it's RPi-related or not, probably best is to ask on a more general forum for help about this, like StackExchange superuser: https://superuser.com/ Probably someone which more Samba experience has further debugging ideas or knows more about possible issues in combination with DFS.
Joulinar
commented
3 years ago Still would be interesting to see how plan Raspberry OS is behaving
Loader23
commented
3 years ago I have setup Raspbian Lite, made update and upgrade and added the line to fstab. The mount line is unchanged except the uid and gid. After reboot the mount is available with no problems. with dmesg I get this warning, but its working anyway.
[ 38.314167] CIFS: VFS: Autodisabling the use of server inode numbers on new server
[ 38.314177] CIFS: VFS: The server doesn't seem to support them properly or the files might be on different servers (DFS)
[ 38.314185] CIFS: VFS: Hardlinks will not be recognized on this mount. Consider mounting with the "noserverino" option to silence this message.I did not get this warning with dietpi though.
With DietPi 7.0 its also not working and additionally the Kiosk System is broken now too with this error: no screens found(EE)
MichaIng
commented
3 years ago Did you do an apt full-upgrade and reboot on Raspberry Pi OS, before testing it?
Loader23
commented
3 years ago Hmm, no.^^ Just looked at the kernel, that was updated.^^
MichaIng
commented
3 years ago Is done of those two packages installed?
dpkg -l | grep -E (keyutils|winbind)Ah the second one will be it: https://packages.debian.org/buster/winbind
This package provides winbindd, a daemon which integrates authentication and directory service (user/group lookup) mechanisms from a Windows domain on a Linux system.
At least sounds like it. Strange only that it was not required when you used DietPi v6.33 🤔.
Loader23
commented
3 years ago Thats the solution! 😄 I have installed keyutils (not winbind) and its working! Still confusing why it was working with the old version and stopped working after Kernel Update. Iam going to include this package in all my DietPi scripts. Thanks alot to both of you 😃 👍
MichaIng
commented
3 years ago Great! It's for kernel key management, so then it's likely due to the kernel upgrade, although strange that a downgrade did not work. We'll keep it in mind an in case create some hint in the docs.
Creating a bug report/issue
Required Information
G_DIETPI_VERSION_CORE=6 G_DIETPI_VERSION_SUB=34 G_DIETPI_VERSION_RC=3 G_GITBRANCH='master' G_GITOWNER='MichaIng'busterLinux Dietpi 5.10.11-v7l+ #1399 SMP Thu Jan 28 12:09:48 GMT 2021 armv7l GNU/LinuxSteps to reproduce
Expected behaviour
Actual behaviour
Extra details