limo-app / limo

A simple Qt based mod manager
GNU General Public License v3.0
119 stars 3 forks source link

Hard Link fails to create despite both directories being on the same partition #13

Open Retroeer opened 2 months ago

Retroeer commented 2 months ago

I have Skyrim installed on a NTFS drive, as previously I would just use Windows to mod Skyrim, but with the release of this I decided to give this a shot, however, I get this error image I'm on EndeavourOS and am using the Flatpak version of limo, I've tried flatseal to see if I could get around this, but that doesn't seem to fix it either

WMan22 commented 2 months ago

Have you followed this guide first?

KutaG59 commented 2 months ago

Having the same issue, but using BTRFS drive. Tried moving the game + staging to my home directory, but it didn't fix. Changed some permissions in flatseal assuming it couldn't use those directory's, but that also didn't fix it.

Drives are mounted (with fstab) to /run/media/kuta , the /media directory is owned by my user, in which flatseal has permissions to access. Limo, as far as I can see and know, is using the same directory's on the same drive. Screenshot_20240826_101147 Screenshot_20240826_101216 Screenshot_20240826_101227 Screenshot_20240826_101242

The only form of any logs I could get was running the flatpak through my Terminal


kf.i18n: KLocalizedString: Using an empty domain, fix the code. msgid: "&Yes" msgid_plural: "" msgctxt: "@action:button"
kf.i18n: KLocalizedString: Using an empty domain, fix the code. msgid: "&No" msgid_plural: "" msgctxt: "@action:button"
kuta@kutalinux:~$ ```
phen0mX commented 2 months ago

Having the same issue, but using BTRFS drive.

Works perfectly fine on my BTRFS drive, so that's probably unrelated. Unless it's some setting I am not aware of.

limo-app commented 2 months ago

The flatpak should have access to all those locations and this apparently seems to be unrelated to BTRFS. This error is shown when the c++ filesystem implementation throws while trying to create a hard link between those locations.

Just to make sure and get this out of the way first: Does manually hard linking a test file between those locations work for you?

lastrogue commented 2 months ago

I am also having the same issue with my steam deck. I am on what I believe is btrfs. as that is what the partition shows for the / path. steam deck is weird though. So I could be reading that wrong.

I can create a hard link between the two folders manually from my terminal. I also tried adding the folder paths via flatseal as directories it can access. Still getting the same error.

Edit: my staging directory is in my user's home directory right below the .local directory where the game files are installed in Steam.

Retroeer commented 2 months ago

The flatpak should have access to all those locations and this apparently seems to be unrelated to BTRFS. This error is shown when the c++ filesystem implementation throws while trying to create a hard link between those locations.

Just to make sure and get this out of the way first: Does manually hard linking a test file between those locations work for you?

Went ahead and followed the guide @WMan22 linked, this works better than what I had before, but now that my compat folder is a symlink between my home/user directory and my ntfs partition's steamlibrary, trying to create a hardlink between the 2 locations results in an invalid cross-device link, ext4 to ntfs isn't great I'd imagine, unfortunately I can't swap everything to ext4 :(

limo-app commented 2 months ago

The new update, which i will push to flathub later today, will include more debug output for failed links. Hopefully that will help to identify the issue.

trying to create a hardlink between the 2 locations results in an invalid cross-device link

I am planning to implement sym-link support for 1.0.6. While this may cause issues for some applications, since they don't behave exactly like hard links, it should be a better solution for cases where mods are stored on different file systems.

limo-app commented 2 months ago

Version 1.0.5 is now on flathub. Could somebody who already updated and was affected by the linking issue please try again and copy the new error message? This should help narrow down the cause.

lastrogue commented 2 months ago

Thanks for the response. I'll give this a try in a few hours when I have access to the system I had the issue on and will report back.

Retroeer commented 2 months ago

This is the new error,

2024-08-29 14:48:42 [Error]: The error was: 'filesystem error: cannot create hard link: Invalid cross-device link [/media/SataDrive/SteamLibrary/steamapps/common/Skyrim Special Edition/Data/_lmm_write_test_file_] [/home/retro/.steam/steam/steamapps/compatdata/489830/pfx/drive_c/users/steamuser/AppData/Local/Skyrim Special Edition/_lmm_write_test_file_]'. (Code: 18. Message: 'Invalid cross-device link')
limo-app commented 2 months ago

This is the new error,

It looks like you are trying to create a hard link across file systems from a drive mounted at /media to your home partition. This is impossible. For a short explanation why, see here. Your issue should be solved by making sure the staging directory and the target directory of the deployer (i.e. the game directory) are on the same partition.

Retroeer commented 2 months ago

That's the issue though, if I put the staging directory on the same drive as the game, it attempts to path to my ext4's compatdata folder, which is sym-linked to my game's drive If I put the staging directory in my home directory, this causes it to path to the game's drive Both setups result in an invalid link

limo-app commented 2 months ago

I am not entirely clear on your setup. Is the following correct?

limo-app commented 2 months ago

I just discovered an oversight: The Loot Deployer also checks if it can create a hard link, which is obviously not needed. If the above description of your setup is correct than this causes your issue. I will fix this with the next update.

lastrogue commented 2 months ago

Sorry for the delay for my testing. In my case, they are on the same drive but I still get this message.

2024-08-30 07:25:04 [Error]: Could not create hard link from "/home/deck/limo_staging" to "/home/deck/.local/share/Steam/steamapps/common/Skyrim Special Edition". Ensure that both directories are on the same partition!
2024-08-30 07:25:04 [Error]: The error was: 'filesystem error: cannot create hard link: Invalid cross-device link [/home/deck/limo_staging/_lmm_write_test_file_] [/home/deck/.local/share/Steam/steamapps/common/Skyrim Special Edition/_lmm_write_test_file_]'. (Code: 18. Message: 'Invalid cross-device link')

I have manually been able to create a hard link on a text file between those two locations.

limo-app commented 2 months ago

I have manually been able to create a hard link on a text file between those two locations.

This is strange. The new error message reflects what the underlying OS implementation returns when the c++ standard library tries to create a hard link. I honestly don't see how i could have caused a bug here. It also seems unlikely that the std lib has a bug like that. Since this does not happen on my setup, i am having a hard time identifying the cause. This could also somehow be related to the flatpak sandbox. I will search for any reports of similar bugs for other applications. For now the only thing that would help is if somebody affected by this compiles the non flatpak version and tries with that. This way we could at least rule out the sandboxing as a cause.

lastrogue commented 2 months ago

This could also somehow be related to the flatpak sandbox.

I did use flatseal to grant the limo flatpak access to those specific directories. Which is recursive in my experience.

But it is absolutely possible there is something unique.

This is also SteamOS, and is odd in it's immutable nature. So there could be something there. I can try compiling it, but I will possibly lose it after a reboot.

limo-app commented 2 months ago

This is also SteamOS, and is odd in it's immutable nature

That might be the issue, will look into this. Are you using a non standard file system for the steam deck? It seems hard links work for some other steam deck users.

I did use flatseal to grant the limo flatpak access to those specific directories.

While this won't hurt, it should not be necessary, since, in this case, all directories are in your home directory. Some users have their drives mounted in really unusual places, which is why i am thinking about lifting the file system restrictions that Limo currently has in the next update entirely. Currently Limo only has access to /mnt, /media, /run/media and your home directory. There is an option for flatpaks that enables access to the entire file system. Since Limo already needs the ability to run commands on the host in order for the tools section to work, this restriction could be circumvented anyways.

lastrogue commented 2 months ago

Are you using a non standard file system for the steam deck? It seems hard links work for some other steam deck users.

I am using the standard file system for a steam deck install. I just did a clean re-install of Steam OS on it to test out Limo on it when I saw the post about it on Reddit.

I am on the Steam Beta release, not stable. This is to take advantage of the steam family groups.

That prompts me to try to see if I can revert to the stable release. I'll give that a go and report back.

lastrogue commented 2 months ago

Going back to stable release for SteamOS did not resolve the issue.

I was on OS version 3.6.9 per the /etc/os-release file while on beta. I went back to version 3.5.19 when I reverted to Stable.

For good measure, here is my lsblk -f output:

lsblk -f NAME FSTYPE FSVER LABEL FSAVAIL FSUSE% MOUNTPOINTS sda zram0 [SWAP] nvme0n1 ├─nvme0n1p1 vfat FAT16 esp ├─nvme0n1p2 vfat FAT16 efi ├─nvme0n1p3 vfat FAT16 efi ├─nvme0n1p4 btrfs rootfs ├─nvme0n1p5 btrfs rootfs 1.8G 58% / ├─nvme0n1p6 ext4 1.0 var ├─nvme0n1p7 ext4 1.0 var 200.3M 6% /var └─nvme0n1p8 ext4 1.0 home 32% /var/log /var/tmp /var/lib/systemd/coredump /var/lib/steamos-log-submitter /var/lib/flatpak /var/cache/pacman /var/lib/docker /srv /root /opt /nix /home

It looks like I should correct my original statement that I'm on btrfs. /home is on nvme0n1p8 and is ext4.

limo-app commented 2 months ago

So far all of my attempts to reproduce this bug have failed, sorry. I will continue to look for solutions, but it seems like at the moment your best option will be to wait for the next update in 2-3 days that will add sym link support. Maybe i can find a different way to create hard links that might at least give insight into why this is happening.

lastrogue commented 2 months ago

Thanks, looking forward to the symbolic link to see if that helps. I even attempted to move the staging directory closer to the Bin directory to see if for some reason my home drive has some strange split going on. I did run into another bug that caused limo to crash and required me to re-import and recreate deployers. I'll create a post on in a moment if I can duplicate it (EDIT: I'm not able to duplicate this second bug at this time).

But after changing the staging directory and trying again, this was the error output:

2024-08-31 20:19:52 [Info]: Startup complete 2024-08-31 20:20:29 [Info]: Adding new application 'The Elder Scrolls V: Skyrim Special Edition' 2024-08-31 20:22:29 [Info]: Adding new deployer 'Bin' 2024-08-31 20:25:02 [Info]: Importing mod '/home/deck/Downloads/skse64_2_02_06.7z' 2024-08-31 20:25:02 [Info]: Mod imported 2024-08-31 20:26:22 [Info]: Installing mod 'skse64_2_02_06.7z' 2024-08-31 20:26:22 [Warning]: Mod 'skse64_2_02_06.7z' has been split because it contains a sub-directory managed by deployer 'Data'. 2024-08-31 20:26:22 [Info]: Installation complete 2024-08-31 20:26:54 [Info]: Deploying mods for 'The Elder Scrolls V: Skyrim Special Edition' 2024-08-31 20:26:54 [Error]: Could not create hard link from "/home/deck/.local/share/Steam/steamapps/common/limostaging" to "/home/deck/.local/share/Steam/steamapps/common/Skyrim Special Edition/". Ensure that both directories are on the same partition! 2024-08-31 20:26:54 [Error]: The error was: 'filesystem error: cannot create hard link: Invalid cross-device link [/home/deck/.local/share/Steam/steamapps/common/limostaging/_lmm_write_testfile] [/home/deck/.local/share/Steam/steamapps/common/Skyrim Special Edition/_lmm_write_testfile]'. (Code: 18. Message: 'Invalid cross-device link') 2024-08-31 20:26:54 [Info]: Mods deployed

priahoud commented 2 months ago

Hey! I am currently testing Limo on my Linux machine and I ran into the same problem.

In my setup, Limo and Steam are both installed from Flatpak. In that scenario, it should in theory be sufficient to just give Limo permission to see the folder at ~/.var/app/com.valvesoftware.Steam/data/Steam/steamapps and the staging folder and as long as both are located on the same drive, everything should work. However, it doesn't because Flatpak access permissions are realized by bind-mounting those directories into the sandbox - and hard links don't work across bind-mounted directories. You can easily verify that without a sandbox by creating two bind-mounted directories on the host and trying to create a hard link across those directories.

I got curious what all the cautions and warnings about soft vs. hard linking was all about and I think that stems from the sub-par implementation of soft links on windows machines. Linux and macOS should have no problem whatsoever to use soft links instead. So for the last three days, I used a patched Limo Flatpak which just creates soft- instead of hard-links and I can report that it works flawlessly.

Looking forward for Version 1.0.6!

limo-app commented 2 months ago

hard links don't work across bind-mounted directories.

If this causes the issues then using filesystem=host for Limo might solve it, i will add that to 1.0.6.

So for the last three days, I used a patched Limo Flatpak which just creates soft- instead of hard-links and I can report that it works flawlessly.

Thanks for sharing this. I didn't find any issues either, so not implementing them from the start was a mistake on my part.

priahoud commented 2 months ago

If this causes the issues then using filesystem=host for Limo might solve it, i will add that to 1.0.6.

That wouldn't help in case of both Apps being Flatpaks though. filesystem=host doesn't really give you full access to the host file system. It is still a filtered list of directories which doesn't include the prefixes of other Flatpak apps under .var/app - and rightly so! So I think leaving the permissions as they are is fine for most cases. People with more specialised setups can easily grant additional permissions from the command line or via Flatseal.

Thanks for sharing this. I didn't find any issues either, so not implementing them from the start was a mistake on my part.

If you do implement symlinks as an additional Option, maybe you might want to go for a global Switch in Settings/Deployers instead of one per Deployer, or at least make it possible to switch the deployment Type after Deployers are already setup, which as far as I can see is currently not the case.

Like this: Settings

limo-app commented 2 months ago

If you do implement symlinks as an additional Option, maybe you might want to go for a global Switch in Settings/Deployers instead of one per Deployer, or at least make it possible to switch the deployment Type after Deployers are already setup, which as far as I can see is currently not the case.

Sym links have been added in de92404 which is part of 1.0.6 and should be live on flathub in a few hours. Deployment method can now be edited for existing deployers. While i can see the advantage of a single global toggle, i believe having this as a per deployer option is preferable for flexibility.