exFAT support

[FurretUber]

Feature Request

I confirm:

• [X] that I haven't found another request for this feature.
• [X] that I have checked whether there are updates for my system available that contain this feature already.

Description

Support for exFAT filesystem, which now has a kernel module, works both on Windows and Linux and no longer requires FUSE. CPU overhead is much lower than ntfs-3g, being very noticeable on loading games.

The dosdevices directory normally has many files with illegal characters (mainly :) for exFAT, being impossible to use an exFAT partition with the compatdata directory, unless if a symbolic link is made pointing to a directory in a partition with a different filesystem type.

Justification [optional]

This use case of sharing very large installed programs happens more often with games, so it may be seen more relevant to the Proton project than to the Wine project.

With Ubuntu 20.04 release, exFAT became natively supported on Ubuntu. Compared to ntfs-3g, which is most often used to have files (and games) available to Windows and Linux, there is a much lower CPU load when using the exFAT filesystem. This results in much lower loading times compared to using ntfs-3g. One game where this is very noticeable is Microsoft Flight Simulator X: Steam Edition (314160).

What seems to be one problem is that exFAT does not support the : character, which is used for the files inside the dosdevices directory. This causes some errors where it fails to create and find files/directories. The following log has the errors when trying to open PHOTON CUBE:

steam-867510.log

Risks [optional]

Being c:, d: the normal way on Windows, a change to make dosdevices show c without the : would require some way to add the : again somehow when searching files. There is a possibility this can cause a regression, and this code seems that was added at least 15 years ago.

A possible issue is that more problems with exFAT may arise, so doing this (seemingly complex) change to dosdevices may not be enough.

Lack of journaling.

References [optional]

https://source.winehq.org/git/wine.git/?a=search&h=HEAD&st=commit&s=dosdevices

[FurretUber]

I'm a bit unsure on where to suggest this, but I was thinking on ways to solve this issue using more reasonable methods. One idea I experimented and seemed to solve this was to make an IMG file that would have an EXT4 filesystem. The idea is to mount this EXT4 filesystem as the dosdevices directory, which seems to be the only blocker.

To be able to mount partitions as non-root, udisksctl was used. As there is no direct control to where it will be mounted, symbolic links are used, then.

This approach does not require a different partition with the compatdata, as everything would be on the own exFAT partition, even the dosdevices. It would still end with a broken symbolic link, however, and I'm not sure how problematic that would be.

The bash script below would be executed as a normal user after trying to run the game at least once. The game would fail the first time, but most of the wineprefix was created. The script below has the purpose to make a functional dosdevices directory for Proton, which makes the game work when running it a second time.

Currently, it isn't written to be generic, as it is more a proof of concept. PHOTON CUBE (867510) works after running the game once (which makes it fail), then running the script below and then opening the game again. I'm confident the proton python script would have the variables to make specific images for every game and setting the EXT4 label (the -L option) unique, which is required for the udisksctl.

#!/bin/bash

##      Diretório inicial/Initial directory      ##
## /media/usuario/Seagate/SteamLibrary/steamapps ##

## PHOTON CUBE ##
##    867510   ## 

cd compatdata/867510/pfx/
dd if=/dev/zero of=dosdevices.img bs=1M count=16
mkfs.ext4 -L PHOTONCUBE -E root_owner=$(id -u):$(id -g) dosdevices.img
udisksctl loop-setup -f dosdevices.img
## Que horror/The horror/Terrível/Terrible ##
sleep 1
udisksctl mount --block-device "/dev/disk/by-label/PHOTONCUBE"
rm -rf "dosdevices"
ln -s "/media/$USER/PHOTONCUBE" dosdevices
rm -rf "dosdevices/c:"
ln -s "$(pwd)/drive_c" "$(pwd)/dosdevices/c:"

[FurretUber]

I tried to work on this idea a bit further, adding these to the proton Python script. The known limitations are:

1) Heavily relies on udisks; 2) Relies on symbolic links; 3) The path of the mounted image is hardcoded to /media/$USER/SteamAppId. Works for Ubuntu but will fail for the distributions that use /run/media; 4) The image isn't unmounted when the game is closed; 5) Microsoft Flight Simulator X: Steam Edition (always this game!) doesn't work.

The games I tested and worked were: Project CARS 2 (378860), WITCH-BOT MEGLILO (437020), Quake Champions (611500), Megadimension Neptunia VIIR (774511), PHOTON CUBE (867510) and LunarLux Chapter 1 (1258970).

One thing that I consider to add is an environment variable that, when set, makes the prefix use this approach with the IMG file, instead of being the only option (as the diff makes it to be).

The diff file with the changes: test-proton-exfat.diff.zip

While this is being useful to hone my Python skills (and discover mu-editor isn't that good once the files are larger), by no means this is ready or high quality. My question is: is this approach of relying on an image file and udisks viable? I wonder if it's worthy to continue improving it.

[walterav1984]

UPDATE: on UDF colons ':' are supported also symlinks just checked it on ubuntu 20.04

Hi @FurretUber have you tried UDF which is read and write on Windows, Linux and other OS. The only downside is that you have to use a whole disk instead of a partition/volume (there are workarounds see issue 24 of format-udf) and depending on the hardware blocksize you are limited to 2TB(512K) or 16TB(4K). You have to use the following format tool to make it work:

https://github.com/JElchison/format-udf

[faulknercs]

As a workaround, it's possible to just mount some directory from the ext4 (or any which supports symlinks) partition into the compatdata directory at exfat partition.

In my case I have exfat partition at /mnt/shared/ and also steam installation at home directory, which is on ext4, so I've just used the directory from the last one:

sudo mount --bind \
  ~/.steam/debian-installation/steamapps/compatdata \
  /mnt/shared/Steam/steamapps/compatdata

Also it can be added to the fstab:

# Steam Proton workaround
/home/artem/.steam/debian-installation/steamapps/compatdata /mnt/shared/Steam/steamapps/compatdata none bind 0 0

[devurandom]

The dosdevices problem appears to affect all external drives independent of the file system, cf. #4330.

[oyvindln]

Even if there was a workaround for colons, exfat does not support symbolic links either which may or may not be an issue.

It would be nice if steam and/or proton would at the very least pop up a warning about trying to install/run a windows game on a exfat partition. Took a while to realize this was what was causing a game to just fail to launch for me.

Guess an easier workaround than changing the underlying code also would be to allow specifying the location of the compat directory to put it somewhere else.

[annnoo]

I am not sure If this comment helps anyone (or if this helps in this discussion), but I've stumbled upon this issue because I am currently facing similair issues - which I somehow managed to fix quite easily.

You can specify the compat data path by using the variable STEAM_COMPAT_DATA_PATH. If you want to specify it in steam just add the following to the launch options so that the variable get set STEAM_COMPAT_DATA_PATH=~/<your_compat_folder> %command% (yep %command% at the end is needed)

[Arch-0]

Replying to https://github.com/ValveSoftware/Proton/issues/3835#issuecomment-985527790

I know this is a few years old but this is a very clever solution, much better (and easier) than setting up launch arguments for each game that I want to play.

Is there a reason why this isn't done by default? I have games installed on the same drive from other stores that I play through Heroic and neither Wine or Proton have any issues with them.

In any case, thank you for the useful tip!

[PhilippMDoerner]

STEAM_COMPAT_DATA_PATH=~/<your_compat_folder> %command%

What I find fascinating about this is that this just sets the environment variable. A variable which I e.g. already have defined in my .zshrc file. This would lead me to assume that it should be getting picked when running, but it does not.

Is there any way to have this as a kind of global prefix applied to any game? I want them all to store their AppData on my local drive because my secondary one is exfat.