Config woes meta-ticket - add ASTROPY_CONFIG_DIR and ASTROPY_CACHE_DIR

[embray]

Description

This ticket does a few things:

• Attempts to synthesize various open tickets related to configuration system woes
• Proposes adding support for ASTROPY_CONFIG_DIR and ASTROPY_CACHE_DIR environment variables, allowing total customization of these paths by users if desired
• De-emphasizes use of XDG_ environment variables for configuring where Astropy-related files go, as it is a somewhat system-biased solution and could cause problems for other apps when used by unwary users (while still maintaining support for the XDG directory specification and documenting it in more "intermediate" config documentation)
• Ditto implementation of #8245 to vendor the appdirs module rather than maintaining our own config path code (or at least a more minimal amount).

I believe this proposal would, in one stroke, resolve:

• 6511

• 6803

• 10244

and maybe some others. Also related is #10090 which I don't think is in conflict with this proposal--it would still just suppress loading any config file. Originally I thought of incorporating #8245 into this, but upon further reflection I think it would be counter-productive.

Detailed proposal

Much of this proposal is already in line with the status quo, but I wanted to write out the logic clearly:

1. By default all Astropy-related directories remain under the path ~/.astropy/. This includes ~/.astropy/config/ and ~/.astropy/cache; this is the current status quo, and is applicable to all platforms, where its platform-specific expansion depends on how os.path.expanduser works on that platform (see also #10705)

2. Astropy will honor the XDG base directory specification if and only if XDG_CONFIG_HOME and/or XDG_CACHE_HOME are explicitly set in the environment. Otherwise it falls back the the default behavior in 1. If either one of the XDG_*_HOME variables is set we commit to the XDG specification. For example if XDG_CONFIG_HOME is set, then the astropy config dir is $XDG_CONFIG_HOME/astropy, and the astropy cache dir is ~/.cache/astropy and not ~/.astropy/cache.

   • Use of XDG_* environment variables will be de-emphasized in the docs, though this logic will be made clear in an admonition box for Linux/BSD users.

3. If ASTROPY_CONFIG_DIR is set, it overrides both 1. and 2. For example setting ASTROPY_CONFIG_DIR=/etc/astropy fixes that path and overrides any other processing. Same for ASTROPY_CACHE_DIR. These variables can both be used independently.

   • These will be documented as the recommended way to change astropy's default directories.

4. 10090 is maintained within this scheme as well. If ASTROPY_SUPPRESS_CONFIG is set then all config file loading is suppressed, overriding all of 1., 2., and 3.

[embray]

As an alternative to 1. and 2., using appdirs à la #8245 could be considered as well. In this case the rest of the logic doesn't change much:

1. On non-Windows, non-MacOS systems only, the XDG layout is used by default.
2. ASTROPY_CONFIG_DIR and ASTROPY_CACHE_DIR would still completely override 1. (i.e. it would bypass whatever appdirs does on the current platform).
3. ASTROPY_SUPPRESS_CONFIG would still override all of the above.

I remain :-1: on this counter-proposal, but it's still worth offering up for discussion.

[pllim]

I am okay with dropping appdirs as an option. No one ever stepped up to do it in all these years, so obviously the problem it was meant to fix isn't that bad.

[embray]

I mostly made mention of that since I did run across the issue for it, and for a hot minute it looked like a good idea until I changed my mind. Just listing it here since, assuming others agree, we can decide to close that issue (since it would be obsoleted by the other solution I proposed).

[embray]

@Cadair any thoughts on this, since you opened #8245? @astrofrog you seemed interested as well.

[saimn]

Following a discussion on Slack with @pllim (cc @astrofrog and @Cadair as well), I would like to push on the idea (already proposed in #10090) of stopping writing and updating config files by default, and doing nothing when trying to read it if it is missing.

The current mechanism is unnecessarily complicated, causing some issues (warnings on the v4.0.X branch #10854, race condition if multiple process attempt to read/write the config file #7290), for no real benefit. The only benefit is for the users to have an example config file with the possible parameters, but this can be solved with documentation (already done in #10148, https://docs.astropy.org/en/latest/config/astropy_config.html) and a way to generate the file if needed.

Also, doing nothing if the file is missing has zero impact. And it actually already works on the dev version, deleting it will cause no error or warning. The default file is only generated for released versions.

[astrofrog]

I agree that we should not write the config file by default, and users should have to opt in to creating it if they want to customize anything.

[astrofrog]

@saimn it might be worth just asking on astropy-dev if anyone has objections to your idea (mentioning the benefits) and doing a PR if not.

[saimn]

I would be happy to do so, but I think we must first find an agreement here since there is also #10090 which is going in another direction ? I started a branch and can push it for discussion.

[astrofrog]

In my view both are needed: a way to suppress any loading of config files, and avoiding writing them by default, so I see the two PRs as complementary.

[saimn]

@astrofrog - But if the file is not written by default, the natural way to suppress loading the config file is to delete or rename it. I'm not sure we need an environment variable for this ?

[astrofrog]

One might want to just temporarily suppress the loading in which case renaming or removing would be too heavy handed?

[saimn]

Renaming the file is simple enough, and that's what I would think to do for any software. Also there is the proposal above for ASTROPY_CONFIG_DIR, which could be also used for this, if allowing it to point to a non-existing file.