search

rust-cli / team

CLI working group
https://rust-cli.github.io/book
MIT License
297 stars 34 forks source link

Config file management #7

Open spacekookie opened 6 years ago

spacekookie commented 6 years ago

(moderated summary by WG)

Context

Plan of Action

In-ecosystem resources

External inspiration

Challenges

@spacekookie's original poist

In the first meeting, we discussed the task of file management on different platforms (particularly configurations) and how it can be made better.

@Eijebong summarised it like this

If I need a config file, I don't want to know that it should be ${XDG_CONFIG:${XDG_HOME}/.config:/home/${user}/.config on linux, %AppDir%/App/ on windows and something else on osx [...]

There is a crate for "determing system configuration" (app-dirs-rs) but it seems unmaintained and not up to date

i30817 commented 6 years ago

It's true that is a concern more of other files that interact more with user paths that the typical config file (though 'last used recent path' in unix is a frequent exception), but i'm thinking that point 1 and 3 are sufficiently valuable that you want to give the ability for them to be used in those paths.

soc commented 6 years ago

@sharkdp It would be even more correct to use Apple's plist format if you write configuration to ~/Library/Preferences, but that's why the API avoids making any judgement on what you do with the paths directories gives you.

(If I have some time in the future, I'd love to propose some RFC for Redox that overhauls config handling (using a config: scheme) there and extend directories to support it.)

@i30817 I pushed https://github.com/soc/paths-rs which is a rough sketch of my approach.

pickfire commented 6 years ago

https://crates.io/crates/dirs looks like a good choice for me even with it's limited features but the API is a lot simpler compared to https://crates.io/crates/directories.

soc commented 6 years ago

Yes, that's intentional, dirs just exposes individual functions, but if you want to derive paths for applications then you should absolutely use directories' ProjectDirs instead.

kbd commented 6 years ago

Should command-line tools on macOS use ~/Library/Preferences or ~/.config?

I would strongly expect any command line tool to use XDG standard location for config. There is no command line tool I know of that stores its config in ~/Library/Preferences. I know I'm not alone in source-controlling my dotfiles in ~/.config -- while not source-controlling anything in ~/Library/Preferences -- so storing config in ~/Library/Preferences makes that less convenient.

Besides the lack of consistency in storing things in one location on Linux and another on Mac, if Rust utilities alone start storing things in ~/Library/Preferences it's going to be a Java Swing-style uncanny-valley sign of "this is a Rust app".

soc commented 6 years ago

@kbd As a bit of data: dirs and directories have been out for a while, and I have received no complaints about those libraries adhering to macOS conventions in general.

lavifb commented 5 years ago

I would like to second @kbd. Command line tools are often cross-platform for nix and as a result the config files are stored similarly to the way they do on Linux, at ~/.config. It doesn't make much sense to store them in a harder to reach and less nix-y place like ~/Library/Preferences.

reitermarkus commented 5 years ago

I can only confirm what @kbd and @lavifb are saying. I have yet to see a CLI tool which uses ~/Library/Preferences for configuration files.

bitwalker commented 5 years ago

Just wanted to chime in with my two cents here as well, this actually caught me off guard recently with a tool I was using, llvmenv, which I was making some changes to. It instructed me (and I would've expected no different) to put files in $XDG_CONFIG_HOME/llvmenv, and that a variety of other associated files (cache, data) would be put in other XDG directories accordingly. I did so, but the tool wasn't picking up my config, and I assumed it was a bug of some kind. It was only after I dug through llvmenv, traced through to dirs, that I realized what was going on. It is highly unusual in my experience to find command line tooling with support for XDG that does not honor it on macOS - so unusual that I don't think the author of that tool even realized this was happening on macOS.

If the default directories end up in ~/Library or wherever, then I think it should be a convention that if the XDG_* env vars are set, that those directories are used instead. If they are set, that is a strong indication that the user expects those paths to be used in place of whatever other conventions the OS uses.

I basically never assume configs or other data associated with my shell environment are going to end up anywhere other than in the XDG base directories I have set, or in the worst case, my home directory - but I would never even think to look in ~/Library/Preferences for that kind of thing.

For tooling I build, I certainly will be overriding this behaviour in dirs or any other such library I use so that XDG is always honored first.

soc commented 5 years ago

I closed https://github.com/soc/directories-rs/issues/47, please have a look at the explanation in https://github.com/soc/directories-rs/issues/47#issuecomment-478337412 for details.

In this case, ~/Library/Caches is the problem.

In general, if I had a buck for every time I heard "I shouldn't need to follow the platform's standards because of X" I could have retired multiple times already. Stuff like this is way we can't have nice things.

sharkdp commented 5 years ago

@soc I understand that you do not want to include this in directories-rs, which probably also targets other (GUI) applications.

However, as we have heard from multiple people in this thread (and multiple people in bats issue tracker), the official standard (~/Library/Preferences) on MacOS does not seem to be followed by any command line applications, whereas the XDG-standard seems to be used quite a lot. In this case, I'd rather listen to the actual users and follow what other command-line applications do, instead of following the "official standard". This is why I feel that there is a need for a directories-rs-like library that uses the XDG-standard on both Linux/MacOS and the "Known Folders" on Windows.

lavifb commented 5 years ago

Also I'm not sure I understand the cache problem. Is it a big problem that cached files for CLI apps might be backed up? CLI caches do not tend to be that big in my experience.

But even if it were a big problem, this is not a reason config files can't be placed in ~/.config. People don't manually look at or edit caches very often. Config files, on the other hand, are very important especially in CLI applications. There is often no other way to configure these apps. Placing these files in a convenient and generally agreed upon location for users shouldn't be dismissed out of hand.

soc commented 5 years ago

@sharkdp

However, as we have heard from multiple people in this thread (and multiple people in bats issue tracker), the official standard (~/Library/Preferences) on MacOS does not seem to be followed by any command line applications, whereas the XDG-standard seems to be used quite a lot. In this case, I'd rather listen to the actual users and follow what other command-line applications do, instead of following the "official standard".

The difference between Linux and macOS is that Linux is a system written by developers, for developers.

On the other hand., macOS is built by Apple, owned by Apple and controlled by Apple. The opinions of developers frankly doesn't matter on macOS – they are not calling the shots. If Apple ever decides to clamp down on applications not following their rules, I'm not going to take responsibility for the ensuing breakage.

Just as a reminder,

does not seem to be followed by any command line applications

is an excuse I heard more often than I care to count on Linux, too.

directories-rs does exactly what it says on the tin: it follows the platforms standards.

soc commented 5 years ago

@lavifb

Is it a big problem that cached files for CLI apps might be backed up? CLI caches do not tend to be that big in my experience.

Pretty much every dependency manager on this planet is a CLI app. Have a look at the caches of Maven, npm or cargo. The cache directories are huge.

lavifb commented 5 years ago

@lavifb

Is it a big problem that cached files for CLI apps might be backed up? CLI caches do not tend to be that big in my experience.

Pretty much every dependency manager on this planet is a CLI app. Have a look at the caches of Maven, npm or cargo. The cache directories are huge.

But all of those store cache in the home dir on MacOS:

More importantly, their config files are not buried somewhere deep in preferences but are all easily accessible in the home dir.

soc commented 5 years ago

Yes, I hope you now understand the issue with that (on both Linux and macOS).

lavifb commented 5 years ago

@soc I think we just have different philosophies for this kind of library. Thanks for explaining your reasoning

@sharkdp

This is why I feel that there is a need for a directories-rs-like library that uses the XDG-standard on both Linux/MacOS and the "Known Folders" on Windows.

I do, however, think there is room for another crate that makes a different tradeoff here.

sharkdp commented 5 years ago

This is why I feel that there is a need for a directories-rs-like library that uses the XDG-standard on both Linux/MacOS and the "Known Folders" on Windows.

I do, however, think there is room for another crate that makes a different tradeoff here.

Absolutely. That's why I said that I completely understand that @soc does not want to change directories-rs.

soc commented 5 years ago

Alternatively we could encourage developers to write applications which are well-behaved citizens of the platform they run on.

In the beginning of medicine people also thought there was room for not washing hands before a procedure. Thankfully, washing hands became non-optional, even though many considered it to be surprising and unexpected at that time.

I'm optimistic that – with an effort to educate developers – a similar success story can be achieved here.

kbd commented 5 years ago

Counterpoint: the reason we disagree isn't because we need to be "educated" that we're "share-croppers" who need to learn to be "well-behaved", but because we view the platform differently.

Many use macOS because it's the best way to get a unix with a good gui. Remember, with the command line on Mac we're talking about an open-source unix that effectively shares a userspace with Linux.

That shared GNU/Linux/POSIX userspace is what we think of when we use the command line on Mac. It's why we'd prefer to share system conventions with it when it makes sense to.

XAMPPRocky commented 5 years ago

This topic isn't being very productive, and I don't think it should be discussed further on this issue. We'd be better served by working towards one of other the aspects of configuration file management we've yet to address, rather than circling around a difference in opinion.

BurntSushi commented 5 years ago

While this may not be satisfactory for a number of reasons, I'd like to say that ripgrep's approach to configuration files has generally side-stepped all of the issues mentioned here, and generally has received zero complaints from users. The trick is to opt out of encoding knowledge about xdg (or similar) at all in the first place. Namely, ripgrep requires users to set an environment variable that points to the location of the config file.

That location can be anywhere. The user can choose any arbitrary location. People like me who gave up caring about the cleanliness of their home directory can plop it at $HOME/.ripgreprc, while macOS users rebelling against their overlords can drop it into an xdg compatible location if it suits them.

As I said initially, this is not satisfactory in all cases for a number of reasons. One such example is that this really only applies to configuration. If your application needs to make use of cache directories, for example, then forcing the user to set an environment variable is probably bad UX and not something they'd expect. My approach also only works if the configuration file is entirely optional (which is honestly probably a generally good idea anyway).

Anywho, I don't mean to propose this as a panacea, but it's something that can work well in a specific circumstance for maintainers that don't want to be on the wrong side of a horde of users because your program put the config file in the wrong place.

epage commented 5 years ago

Regarding layered configs: I've been playing with this with cargo-release and for various reasons decided against the existing crates.

Aspects of cargo-release

To get an idea of how this works, see

And some solution exploration

I feel like a toolbox approach to support layered config would be more useful than an all-in-one / opinionated solutions. For example

Alxandr commented 5 years ago

I just published my first rust library on crates.io which is a config library that I plan to use in another application I'm developing. It's currently lacking quite a bit in documentation (but there is some), but it provides configuration (either configured by the application, or by convention) from multiple places (user folders, application folders, environment variables and cli arguments). Binding to structs is still not available though (but should be reasonably straight forward).

Sample usage looks like this:

#[derive(ClapConfig)]
pub struct Server {
  #[preftool_clap(help = "Server host")]
  host: String,

  #[preftool_clap(help = "Server port")]
  port: usize,

  #[preftool_clap(help = "Server certificate (enables TLS)")]
  cert: Option<String>,
}

fn main() -> std::io::Result<()> {
  let mut builder = TerminalLoggerBuilder::new();
  builder.level(Severity::Debug);
  builder.destination(Destination::Stdout);
  let log = builder.build().unwrap();

  let config = AppConfigBuilder::new("preftool-app-config-simple", Some(log))?
    .with_args::<Server>()
    .configure(|app| app.version("0.0.1"))
    .add_format(TomlConfigFormat)
    .build()?;

  println!("{:#?}", config);
  Ok(())
}

This will read config from toml files, environment variables, and application arguments. Code is available here: https://github.com/YoloDev/preftool.

soc commented 5 years ago

@Alxandr Very interesting library!

One note, please do not use dirs-sys directly – I updated the README to that effect: https://github.com/soc/dirs-sys-rs#compatibility

Alxandr commented 5 years ago

@soc issue is that neither one of those two provides me the information that I need though. I will probably have to go back to just having the functions embedded instead. I switched to dirs-sys to get away from having to do conditional dependencies based on OS. The only thing I use home_dir, is_absolute_path and 2 of the windows dirs.

soc commented 5 years ago

Feel free to use it at your own risk. I just wanted to make sure that the level of non-guarantees has been communicated. :-)

Alxandr commented 5 years ago

Fair enough. I would love a stable API with the low-level primitives I though ;-).

cjbassi commented 5 years ago

I just made/published platform-dirs which provides a simple API for retrieving platform dependent application directories and user directories. It also allows for specifying if an application is CLI based and uses the XDG specified application directories on macOS if so. There's some other differences relative to dirs-rs and directories-rs at the bottom of the readme too. It's still beta-level so the API may change and it may have bugs FYI. Any and all suggestions welcome!

epage commented 3 years ago

From Discord mike7c2

It's pretty neat at the user end, I didn't have to do much at all to get it running, it does layering off cli args, config file args and environment args, it has some tools for other layers too but I didn't look at them

I really liked being able to define my config as a structure - that was easy and natural, really bolstered by the fact it populates comments from the struct into the CLI arg definitions and stuff. I do wonder if I should be going a step further and using a yaml/dhall schema to define my arguments but at the same time I don't think that added level of descriptiveness will get me anything

I didnt check in detail but it definitly pulled in quite a bit of stuff, exe size got big adding it but that isn't a real issue for me

The thing that bugged me most about it was not being able to make nested config fields, it needs to work with a flat structure so to work with it you need to flatten any nested keys

Overall I'm just trying to find something which works good enough I can build into a template I will build some GRPC services on top of
runiq commented 3 years ago

Reformatted for your reading pleasure:

It's pretty neat at the user end, I didn't have to do much at all to get it running, it does layering off cli args, config file args and environment args, it has some tools for other layers too but I didn't look at them

I really liked being able to define my config as a structure - that was easy and natural, really bolstered by the fact it populates comments from the struct into the CLI arg definitions and stuff. I do wonder if I should be going a step further and using a yaml/dhall schema to define my arguments but at the same time I don't think that added level of descriptiveness will get me anything

I didnt check in detail but it definitly pulled in quite a bit of stuff, exe size got big adding it but that isn't a real issue for me

The thing that bugged me most about it was not being able to make nested config fields, it needs to work with a flat structure so to work with it you need to flatten any nested keys

Overall I'm just trying to find something which works good enough I can build into a template I will build some GRPC services on top of