~/.hydrogen does not follow XDG standards

ghost commented 6 years ago

Hydrogen makes a .hydrogen directory.

.hydrogen
├── data
└── hydrogen.conf

1 directory, 1 file

XDG standards specify that config files should go in ~/.config and data in ~/.local/share.

Therefor, the ~/.hydrogen/data directory should be moved to ~/.local/share/hydrogen and hydrogen.conf should be moved to ~/.config/hydrogen.conf by default.

elpescado commented 6 years ago

That's because Hydrogen predates XDG spec, and .hydrogen dir has been kept for backwards compatibility.

ghost commented 6 years ago

Then how about using symlinks?

mauser commented 6 years ago

Hi!

Moving the config file to a different problem shouldn't be such a large problem, we could save the file in ~/.config/hydrogen.conf and favour this location over .hydrogen when loading the config file.

The data could be a larger problem since due to all of those references in song files, but a symlink for backward compatibility could work.

jeremyz commented 6 years ago

Hi, It's on my TODO list, I did send a mail about that very same point to the ML on Tue, 17 Apr 2018. About the data, maybe, on h2 startup if there is a .hydrogen/data directory, we ask the user if we can move that directory to ~/.local/share/hydrogen, otherwise h2 exit.

trebmuh commented 6 years ago

About the data, maybe, on h2 startup if there is a .hydrogen/data directory, we ask the user if we can move that directory to ~/.local/share/hydrogen, otherwise h2 exit.

@jeremyz I do understand that we want to transition (and encourage users to transition) from ~/.hydrogen/data to ~/.local/share/hydrogen/data, but don't understand why H2 should exit if not?

elpescado commented 6 years ago

I like this idea, but have some questions/remarks.

Should we ask user to upgrade or just upgrade silently?
Leave old .hydrogen directory during migration or leave it so users can revert to older version without losing config?
What happens if user has both .hydrogen and XDG configs? Load only XDG? Load both and prefer XDG? What if .hydrogen/data contains drumkits A and B and .local/data/hydrogen has drumkits B and C - what drumkits should be available to user - B,C (XDG only)? A,B,B,C (union)? A,B,C (unique union)?

hydrogen.conf should be moved to ~/.config/hydrogen.conf by default.

To be precise, it should be moved to ($XDG_CONFIG_HOME or $HOME/.config)/hydrogen.conf

Therefore I fear this not as simple as it seems at first sight and on the other hand benefits are not that big.

riri commented 4 years ago

One solution would be to degrade gracefully for each of the two points (configuration and data) :

First check command line arguments
If not specified, check and load from what should be nowadays (XDG compliant)
If not exists, check and load from old location
If not exists, create at new location (XDG compliant)

For configuration:

-c path-to-config-file (new one to define) set? use it
$XDG_CONFIG_HOME/hydrogen.conf exists? load from it
$HOME/.hydrogen/hydrogen.conf exists? load from it
nothing exists, create $XDG_CONFIG_HOME/hydrogen.conf

For data:

-P path-to-data-dir set? use it
$XDG_DATA_HOME/hydrogen exists? use this as base directory for data
$HOME/.hydrogen/data exists? use this as base directory for data
nothing exists, create $XDG_DATA_HOME/hydrogen

That would need a bit of work in Filesystem helper. I don't think other source files should be impacted, except the defaults in project configuration, but I gave just a brief look at the source code, and I don't really know much about Qt and Cmake :)

I don't think an automatic move should be used. At most, a question could be asked to the user on startup if the configuration or the data are the old locations, if they want to migrate to new location.

Or that could be a specific task available in the preferences (people keep the choice to stay with the old location or migrate)

jeremyz commented 4 years ago

I've just started working on it https://github.com/hydrogen-music/hydrogen/commits/xdg

jeremyz commented 4 years ago

Hi @mauser, @trebmuh, @elpescado, @riri, and others

can you check xdg branch please.

on a unix box, you can try it like that : XDG_CONFIG_HOME=$HOME/fake_cfg XDG_DATA_HOME=$HOME/fake_data ./build.sh z you should end up with $HOME/.hydrogen/hydrogen.conf copied to $HOME/fake_cfg/hydrogen.conf and data copied from $HOME/.hydrogen/data/ to $HOME/fake_data/hydrogen. songs and playlists filename attributes should be fixed ... is there other xml attribute to fix ?

riri commented 4 years ago

I have never built hydrogen, and it seems I am missing some parts on ArchLinux (although dev packages are always included). I get: no rule to make the target "translations". Don't worry about that, I will check how hydrogen is packaged.

Isn't there the autosave file in user's directory? (I can't find one in my current installation, but I remember seeing it) This one should be in $XDG_DATA_HOME/hydrogen

jeremyz commented 4 years ago

Hi, you need qt5-tools, songs autosave files are in $XDG_DATA_HOME/hydrogen/songs

riri commented 4 years ago

I could build using my own build script, and it works. All data are transferred in the locations defined by XDG variables :

$HOME/.config/hydrogen.conf
$HOME/.local/share/hydrogen : content of old $HOME/.hydrogen/data

but data is copied.... isn't it too much if the user has a huge set of data ? Shouldn't it be moved instead?

jeremyz commented 4 years ago

is the data OK, or have you paths in some xml files that still reference the old path ? grep -R 'hydrogen/data' ~/.local/share/hydrogen/

moving data would be much more easier, but I fear having user reports saying 'launched H2 0.9.X I've lost all my data'. I've hadded 2 bool in Filesystem.h config_migrated and data_migrated, so we can add a msgbox that says, data has been copied from X to Y, do you want us to delete X ?

if we move data and set symlinks old_data -> new_data, running 0.9.x should work, but songs generated from it would have layer filename that wont work when going back to 1.x.y ...

riri commented 4 years ago

is the data OK, or have you paths in some xml files that still reference the old path ? grep -R 'hydrogen/data' ~/.local/share/hydrogen/

All seems ok, no reference to the old path :)

moving data would be much more easier, but I fear having user reports saying 'launched H2 0.9.X I've lost all my data'. I've hadded 2 bool in Filesystem.h config_migrated and data_migrated, so we can add a msgbox that says, data has been copied from X to Y, do you want us to delete X ?

To me the message method is good, the user then knows about paths that changed. If they dont bother with geek things, they will choose whatever is the default. Those aware and sensitive to this will make their choice : move (delete old) or copy (do not delete old).

if we move data and set symlinks old_data -> new_data, running 0.9.x should work, but songs generated from it would have layer filename that wont work when going back to 1.x.y ...

For this, once users update to 1.x.y (and then has there data migrated), they wouldn't generate new songs with 0.9.x, except hydrogen developers and testers. I don't think that's so important (but I see only my usage).

Edit: I have just checked one song xml content, and in each instrument nodes, there are only the name of the drumkit and and filename (without path) for layers.

jeremyz commented 4 years ago

Edit: I have just checked one song xml content, and in each instrument nodes, there are only the name of the drumkit and and filename (without path) for layers.

I see now that full path is used in autosave files, should not be, will fix that ...

elpescado commented 4 years ago

The only thing that I'm afraid of is that copying user data, especially drumkits, might result in shuffling large amount of data around. That in turn might take time, freezing GUI and delaying startup in meanwhile. I'm not sure if that's real problem today, in the age of crazy fast SSDs, though. If so, maybe we should show dialog box (possibly with progress bar?) explaining what's going on?

Plus, what happened when those two copies go out of sync?

Alternatively, we already load drumkits from two locations: one system (/usr/local/share) and one user (~/.whatever/data). Maybe we could just add third location, XDG-compliant one, and prefer that when writing. Any thoughts?

polirritmico commented 3 years ago

Any update on this?

polirritmico commented 2 years ago

I made a more suitable patch, this time uses QStandardPaths to get all the XDG paths and make the files and the folders accordly. The patch splits the configuration, data and cache folders following the XDG specs:

The config file hydrogen.conf is created into $XDG_CONFIG_HOME/hydrogen/
The data subfolders (drumkits, plugins, playlists, etc.) and the hydrogen.log file are created inside $XDG_DATA_HOME/hydrogen
The cache folder go into $XDG_CACHE_HOME/hydrogen/

Here's the patch for Hydrogen 1.1.1: https://gist.github.com/polirritmico/450bdc4f24d441df3b660fe852b0590b

Tested and working on Gentoo. Regards.

polirritmico commented 2 years ago

The code on the PR uses this logic:

Search for old hydrogen folder (~/.hydrogen). If the folder is found uses it.
Else, follow the XDG structure (cache, data, config) to make and use the paths.

elpescado commented 2 years ago

Does this impact macOS/Windows?

polirritmico commented 2 years ago

A new PR with small changes. I also create a patch for hydrogen-1.1.1 version to declutter my Linux home folder.

Does this impact macOS/Windows?

Yes, if no "old path" is found it should uses the paths provided by QStandarPaths:	QStandardPaths	macOs	Windows
AppConfigLocation	~/Library/Application Support/hydrogen	C:/Users/USER/AppData/Roaming/hydrogen	~/.config/hydrogen
AppLocalDataLocation	~/Library/Application Support/hydrogen	C:/Users/USER/AppData/Local/hydrogen	~/.local/share/hydrogen
CacheLocation	~/Library/Caches/hydrogen	C:/Users/USER/AppData/Local/hydrogen/cache	~/.cache/hydrogen

But I don't have a Windows or macOS machine to test it.

theGreatWhiteShark commented 2 years ago

TBH I do not understand the point of this issue. Changing the config folder after more than 20 years feels like something that can only bring harm (and lot's of nuisance issues/bug reports) or at least I do not see the problem it solves.

Does this impact macOS/Windows?

I would strongly suggest to not move the config folder for macOS and Windows. Linux users may have already experience with applications moving their stuff in order to comply with freedesktop standards. But on other platforms users might only be annoyed.

What's about *BSD variants? Right now we don't have a precompiler directive to distinguish them from Linux builds.

The code on the PR uses this logic:

1. Search for old hydrogen folder (`~/.hydrogen`). If the folder is found uses it.

2. Else, follow the XDG structure (cache, data, config) to make and use the paths.

I would also suggest an error dialog in case both directories are present. I still remember the mess 10 years ago when last using a Gnome-based desktop. Config files were all over the place, some in custom folders, some in XDG compliant one, and you never knew which one took precedence.

I also concur with @mauser that ~/.local/share/hydrogen should be linked to ~/.hydrogen/data. Else, song files could not be guaranteed to loaded correctly anymore (prior to version 0.9.7. songs used absolute for all samples - incl. the drumkits found in the data folder).

polirritmico commented 2 years ago

I would strongly suggest to not move the config folder for macOS and Windows.

Agree, now it shouldn't affect Windows and macOS systems (PR #1624).

TBH I do not understand the point of this issue. Changing the config folder after more than 20 years feels like something that can only bring harm (and lot's of nuisance issues/bug reports) or at least I do not see the problem it solves.

The point is follow the XDG standard so Linux users found configuration, data and cache files on the expected folders. Then for example if the user runs out of space on the hard drive it could easily clean .cache and expect no data or configuration lost. Another example is for backups scripts collecting all configurations in one folder at once and all data in one place. Also the standard follows environment variables so that the user can configure them in the way that is most convenient for his system. For example export XDG_DATA_HOME=/path/to/fast_drive/

A clutter home it's something that concerns many users:

What's about *BSD variants? Right now we don't have a precompiler directive to distinguish them from Linux builds.

I don't have any experience with *BSD, but a lot of Linux software follow the XDG standard so I don't think it would bother too much. Also, if the old .hydrogen path is found nothing should change.

I would also suggest an error dialog in case both directories are present. I still remember the mess 10 years ago when last using a Gnome-based desktop. Config files were all over the place, some in custom folders, some in XDG compliant one, and you never knew which one took precedence.

If all programs follow the standard then there shouldn't be problems like that in the first place. Anyway I could work on that; would it be enough to add an warning in the log file?

I also concur with @mauser that ~/.local/share/hydrogen should be linked to ~/.hydrogen/data. Else, song files could not be guaranteed to loaded correctly anymore (prior to version 0.9.7. songs used absolute for all samples - incl. the drumkits found in the data folder).

Could someone provide a <=0.9.7 song file so I can look a solution for that case?

If more information or more adjustments are required, please comment so I could try to solve it.

theGreatWhiteShark commented 2 years ago

The point is follow the XDG standard so Linux users found configuration, data and cache files on the expected folders. Then for example if the user runs out of space on the hard drive it could easily clean .cache and expect no data or configuration lost. Another example is for backups scripts collecting all configurations in one folder at once and all data in one place. Also the standard follows environment variables so that the user can configure them in the way that is most convenient for his system. For example export XDG_DATA_HOME=/path/to/fast_drive/

Hmmmm. I'm still not not that convinced. But it's not because I dislike the standard. For a new or still alpha-stable application I would agree without any discussion. It's just, Hydrogen predated the XDG standard, is around for 20 years, and there is already a small ecosystem building around it. Smaller Python/Perl/Bash scripts of users (some even publicly available on Github), personal backup or system installation scripts, test pipelines of maintainer's of different distribution etc. If any of those relies on accessing the ~/.hydrogen folder things will break and a lot of those will probably stay broken. So, adopting the standard does feel more like breaking stuff than increasing interoperability (at least from my point of view).

But, on the other hand, even Emacs is young and agile enough to adopt it :grinning:

Anyway I could work on that; would it be enough to add an warning in the log file?

A popup dialog similar to the warning shown when using a development version of Hydrogen (https://github.com/hydrogen-music/hydrogen/blob/master/src/gui/src/MainForm.cpp#L2052) would be better. Typical users do not read the log.

Could someone provide a <=0.9.7 song file so I can look a solution for that case?

There is one in the repo used for unit tests. https://github.com/hydrogen-music/hydrogen/blob/master/src/tests/data/song/test_song_0.9.6.h2song

hydrogen-music / hydrogen

~/.hydrogen does not follow XDG standards #643