POETRY_CACHE_DIR or cache-dir config are not used at all.

mdgilene commented 4 years ago

[x] I am on the latest Poetry version.
[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: Windows 10
Poetry version: 1.0.5
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/mdgilene/0da1568f8ac90615e89cd1331115d80a

Issue

Setting the cache-dir location using either the environment variable POETRY_CACHE_DIR or with poetry config cache-dir /path/to/new/cache does not work. Poetry still uses the default location C:\Users\<user>\AppData\Local\pypoetry\Cache.

This will not work for me as there are size constraints on my user profile directory. Beacuse of this I need to relocate this directory to another drive.

I believe this to be cause by the fact that the environment/config settings are never read when determining the cache location.

poetry/utils/appdirs.py

def user_cache_dir(appname):
    r"""
    Return full path to the user-specific cache dir for this application.
        "appname" is the name of application.
    Typical user cache directories are:
        macOS:      ~/Library/Caches/<AppName>
        Unix:       ~/.cache/<AppName> (XDG default)
        Windows:    C:\Users\<username>\AppData\Local\<AppName>\Cache
    On Windows the only suggestion in the MSDN docs is that local settings go
    in the `CSIDL_LOCAL_APPDATA` directory. This is identical to the
    non-roaming app data dir (the default returned by `user_data_dir`). Apps
    typically put cache data somewhere *under* the given dir here. Some
    examples:
        ...\Mozilla\Firefox\Profiles\<ProfileName>\Cache
        ...\Acme\SuperApp\Cache\1.0
    OPINION: This function appends "Cache" to the `CSIDL_LOCAL_APPDATA` value.
    """
    if WINDOWS:
        # Get the base path
        path = os.path.normpath(_get_win_folder("CSIDL_LOCAL_APPDATA"))

        # Add our app name and Cache directory to it
        path = os.path.join(path, appname, "Cache")
    elif sys.platform == "darwin":
        # Get the base path
        path = expanduser("~/Library/Caches")

        # Add our app name to it
        path = os.path.join(path, appname)
    else:
        # Get the base path
        path = os.getenv("XDG_CACHE_HOME", expanduser("~/.cache"))

        # Add our app name to it
        path = os.path.join(path, appname)

    return path

pwoolvett commented 4 years ago

Far from ideal, but as a workaround you can temporarily set/replace XDG_CACHE_HOME, eg:

XDG_CACHE_HOME=/path/to/another/drive/cache poetry add black

(tested on 1.0.9)

finswimmer commented 4 years ago

Hello,

I tested it with the latest preview release without any problems. Is someone else able to reproduce the issue?

fin swimmer

inseopark commented 4 years ago

I've found something strange behavior on this topic. Using poetry version 1.1.2, on Windows, I wanted run some script runnning with System privileges, I want to set cache-dir set other than Windows system directory.

When testing with normal user account,

Without environment variable set :

C:\Users\ispark_skcc\AppData\Local\pypoetry\ used 
$ poetry shell
$ poetry add semver
...
$ gtree -dL 2 C:\Users\ispark_skcc\appdata\local\pypoetry\Cache
C:\Users\ispark_skcc\appdata\local\pypoetry\Cache
|-- artifacts
|   `-- d1
`-- cache
`-- repositories

With POETRY_CACHE_DIR set to other locations only artifacts directory goes into POETRY_CACHE_DIR, but cache\repositories\pypi generated in %LOCALAPPDATA%

$ gtree -dL 3 C:\Users\ispark_skcc\appdata\local\pypoetry\Cache
C:\Users\ispark_skcc\appdata\local\pypoetry\Cache
`-- cache
`-- repositories
    `-- pypi

mohan43u commented 3 years ago

only artifacts directory goes into POETRY_CACHE_DIR, but cache\repositories\pypi generated in %LOCALAPPDATA%

I can confirm this is still happening in Linux. Even after setting POETRY_CACHE_DIR, I see ~/.cache/pypoetry/cache/repositories/pypi/ created with contents.

Poetry Version: 1.1.4

TBBle commented 3 years ago

Perhaps it makes more sense to just drop the "cache-dir" config option, rather than trying to plumb it through the other places that are directly using locations.CACHE_DIR and locations.REPOSITORY_CACHE_DIR?

Honouring XDG_CACHE_HOME is probably sufficient on Posix platforms, and Windows and OS X have pretty clear cache locations as well.

mdgilene commented 3 years ago

I'm sorry but "just remove it" is not a valid solution. It is common practice among many development tools that utilize a local cache to allow the user to configure where this cache is stored.

My workplace uses network storage for users home directories so that settings can be used on multiple machines. This however means that we are VERY limited on how much can be stored on the user profile (<500Mb). I HAVE to move this cache to another location or I just simply cannot use the tool.

mohan43u commented 3 years ago

Honouring XDG_CACHE_HOME is probably sufficient on Posix platforms, and Windows and OS X have pretty clear cache locations as well.

changing XDG_CACHE_HOME (also XDG_CONFIG_HOME and XDG_DATAHOME) breaks other apps when we run those apps under 'poetry shell'. Changing `XDG*_HOME` variables for other apps under 'poetry shell' is unnecessary and may lead to unwanted results.

The change In my PR provides alternate to XDG_*_HOME variables in poetry. So, when POETRYHOME is available, poetry will use it instead of `XDG*_HOME` variables.

TBBle commented 3 years ago

I don't think POETRY_HOME is the right thing to leverage for this, because it's surprising that it would override the XDG_CACHE_HOME default behaviour, and also I'd expect nothing to change in POETRY_HOME except when using the self-updater (and then only if Poetry was installed by get-poetry.py).

Edit: I see that POETRY_HOME is only set by the user for certain uses of get-poetry.py, and the self-updater, but it's not actually ensured to be available at any other time as nothing else in the system uses it or is aware of it. So it definitely seems like the wrong thing to use for the cache directory.

Anyway, fair 'nuff, if the cache-dir config is useful, then it should only take a few fixes to make it apply, by making sure that nothing directly accesses location.CACHE_DIR or location.REPOSITORY_CACHE_DIR, and instead all such code uses config.get("cache-dir"), which correctly defaults to location.CACHE_DIR.

It might also be good to fix the default for virtualenvs.path, which is documented as {cache-dir}/virtualenvs but {cache-dir} here actually means "The default cache-dir", not "the value of the cache-dir configuration", at least on inspection of the config-defaulting code. It's possible this is actually not a problem in practice, the curcumstance where I was changing the cache-dir, I am also using virtualenvs.in-project so virtualenvs.path is ignored, so I didn't see any issues with the virtualenv directory.

On the last comment, I'm not sure what poetry shell has to do with this. If I'm changing XDG_CACHE_HOME, of course I'm changing it for everything. That's the whole point. It doesn't break other apps, it causes them to cache in that directory instead. I wouldn't want this changing under something like poetry shell, that'd be awful and surprising.

My use-case is that I'm running poetry inside a container, and the default XDG_CACHE_HOME is not writeable (weird container setup, running as non-root but $HOME is /) so redirecting $XDG_CACHE_HOME is the correct solution.

But in the general case, the cache-env setting should be either fixed or removed. I personally don't mind which, as I don't have a use-case where I need to move just the Poetry cache.

@mdgilene In your circumstance, you probably should be changing $XDG_CACHE_HOME rather than Poetry's cache-dir setting, otherwise you'll have to change each application's own cache setting one by one, and are stuck for apps that don't have that feature. Changing $XDG_CACHE_HOME will fix all (FreeDesktop-spec compliant) apps at once, which is what I would think you want for the general "cache needs to be redirected elsewhere" use-case.

You probably do want to change the virtualenv.path setting if you're putting the cache somewhere temporary; I'm not sure I agree with Poetry that the env is a "cache", but I can see the logic of it.

mohan43u commented 3 years ago

I don't think POETRY_HOME is the right thing to leverage for this, because it's surprising that it would override the XDG_CACHE_HOME default behaviour, and also I'd expect nothing to change in POETRY_HOME except when using the self-updater (and then only if Poetry was installed by get-poetry.py).

The PR doesn't modify POETRY_HOME dir, it creates hidden directories under POETRY_HOME and do the change under those hidden directories.

This way, all I have to do in my bootstrap script is set POETRY_HOME to one location under the project's root directory, so that everything poetry creates (including get-poetry.py) contained within the project's root directory.

On the last comment, I'm not sure what poetry shell has to do with this. If I'm changing XDG_CACHE_HOME, of course I'm changing it for everything. That's the whole point.

the point is why someone has to change XDG_CACHE_HOME in the first place in order to make poetry store its cache and virtualenv dirs in a different place.

Also, there is no way to make poetry store its config in a different place other than changing XDG_CONFIG_HOME.

In my openion, XDG_*_HOME dirs are fallback dirs when the user not explicitly specify where to store the files. But poetry is using XDG_*_HOME dirs assuming user is not going to care about providing the locations.

TBBle commented 3 years ago

Maybe this is a difference of opinion on XDG_*_HOME, which to me are the way to tell all my apps if I want them putting data somewhere other than ~/.cache or ~/.local etc. They're not fallbacks, but the first config target, since I rarely want to tell individual apps to put their caches somewhere different than other apps.

Your use of POETRY_HOME makes your PR make more sense, since I'd never thought of using it to put Poetry inside my project (that seems the opposite of the intention of get-poetry.py, which seems focussed on a single, isolated Poetry install). The use-case I had seem for POETRY_HOME was for putting Poetry somewhere other than ~/.poetry, e.g. somewhere system wide shared by multiple users, rather than for multi-installing it.

But I think in the general case, mutating POETRY_HOME outside the self-updater will hit issues for people where POETRY_HOME is not writable, i.e. the container use-case I mentioned, where I was using get-poetry.py earlier, and would have used POETRY_HOME to install it in /poetry as root, but then used it as a regular user.

"creates hidden directories under POETRY_HOME" is mutating the contents of POETRY_HOME, and will fail if POETRY_HOME is not writable by the user, and would be surprising if I delete and reinstall a get-poetry.py-installed Poetry and expect to not lose my caches. Which I would expect, because caches by default should live in ~/.cache per the FreeDesktop spec. Using POETRY_HOME first, since it's not specifically a config for the Poetry cache, but also controls other behaviour, would be in violation of the FreeDesktop spec, and generally surprising to users.

Anyway, that PR, and discussion about the use-cases of POETRY_HOME, doesn't really affect this issue, which is that the existing cache-dir config option, which should override the common default of $XDG_CACHE_DIR/pypoetry, is not used consistently within Poetry, and that common default leaks through.

Adding another conditional default doesn't fix this issue, it just adds another possible workaround.

parched commented 3 years ago

Is there any workaround for this on windows? Perhaps some environment variable like XDG_CACHE_DIR I can set?

I'm running multiple instances of Poetry at one once and I'm getting a conflict in the cache directory.

  [Errno 17] File exists: 'C:\\Windows\\system32\\config\\systemprofile\\AppData\\Local\\pypoetry\\Cache\\cache\\repositories\\pypi\\_http\\4\\8\\8\\4\\4\\48844afbd98c85dd6f00d0efdc493230a4f783137c66108a422a9d24'
  at f:\gitlab-runner\builds\c1fMCazL\1\james.duley\python-template\.poetry\venv\lib\site-packages\cachecontrol\caches\file_cache.py:46 in _secure_open_write

So I want to use a different cache directory for each invocation. Although, this is probably a separate issue that needs to be addressed.

arielnmz commented 2 years ago

Can confirm this happens in version 1.1.13 in a docker container:

FROM python:3.9-slim-buster

ENV \
    POETRY_HOME="/opt/poetry" \
    POETRY_CACHE_DIR="/opt/poetry/.cache" \
    POETRY_VERSION=1.1.13 \
    # Poetry config
    POETRY_NO_INTERACTION=true \
    POETRY_NO_DEV=true \
    POETRY_VIRTUALENVS_CREATE=false \
    POETRY_VIRTUALENVS_IN_PROJECT=false \
    POETRY_EXPERIMENTAL_NEW_INSTALLER=false 

CMD ["bash"]

Firing up the container with user 1000 and running poetry lock fails with this error:

$ docker-compose run --rm --user=1000:1000 my-service bash
$ poetry lock

Skipping virtualenv creation, as specified in config file.
Updating dependencies
Resolving dependencies... (0.0s)

  LockFailed

  failed to create /.cache/pypoetry/cache/repositories/pypi/_http/2/3/2/5/7/25a703cfdb14-dbb50740.82384136640494608341

It fails because ofc user 1000 does not have access to /.cache, why is it trying to create a directory there?

gshpychka commented 2 years ago

@abn has this been acknowledged? Can we configure the repository cache?

mohan43u commented 2 years ago

this issue should be fixed now. poetry will not create repositories directory (cache/repositories) outside of cache-dir after this commit https://github.com/python-poetry/poetry/pull/5672 , see https://github.com/python-poetry/poetry/pull/5301 for more details.

abn commented 2 years ago

Resovled by #5672.

adam-grant-hendry commented 2 years ago

This is happening for me on Windows with version 1.2.0b2. Any help would be greatly appreciated:

PS> poetry config --list
cache-dir = "C:\\Users\\username\\.poetry\\Cache"
experimental.new-installer = true
experimental.system-git-client = false
installer.max-workers = null
installer.no-binary = null
installer.parallel = true
virtualenvs.create = false
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "C:\\Users\\username\\.poetry\\venv"
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"

When adding to my packages, poetry updates from a cache stored in %LOCALAPPDATA%:

PS> poetry add --group dev instaviz
Using version ^0.6.0 for instaviz

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 2 installs, 33 updates, 0 removals

  • Updating frozenlist (1.3.0 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/9e/6f/48/9c41b19462554c732ac11adcb6c12e9b911989c263486d33dd8ad69d41/frozenlist-1.3.0-cp38-cp38-win_amd64.whl -> 1.3.0)
  • Updating idna (3.3 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/f7/09/c9/8e80952436ff87855ddff4891a35ecf913b2b5dc97911f02a070ff2b1f/idna-3.3-py3-none-any.whl -> 3.3)
  • Updating multidict (6.0.2 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/2b/3f/f0/c9166f505465acaba68c94bdb3f4001777bb73290a85cddc10d17fb6a9/multidict-6.0.2-cp38-cp38-win_amd64.whl -> 6.0.2)
  • Updating pyparsing (3.0.9 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/4a/b9/34/28557b740fb51bd635f6794a95d02e23de02724e07f03c3ad49cebcf86/pyparsing-3.0.9-py3-none-any.whl -> 3.0.9)
  • Updating aiosignal (1.2.0 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/4c/98/1b/ee90d29611d2aaeb4a2b40c1509ca578cccf552d05a53563f4f8bf9bb3/aiosignal-1.2.0-py3-none-any.whl -> 1.2.0)
  • Updating async-timeout (4.0.2 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/31/90/f6/e2df01736b095fcbabf93cf579a4d9b8d7a4a78c4c63fad4d32320b197/async_timeout-4.0.2-py3-none-any.whl -> 4.0.2)
  • Updating attrs (21.4.0 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/9a/27/ab/6895b786f345452394b42c1fdbe863803d0396e675b632c01cb13cad92/attrs-21.4.0-py2.py3-none-any.whl -> 21.4.0)
  • Updating six (1.16.0 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/48/e6/04/8118155ae3ec3a16dd2a213bbf7a7d8a62c596b2e90f73a22c896269f1/six-1.16.0-py2.py3-none-any.whl -> 1.16.0)
  • Updating packaging (21.3 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/7f/d5/4c/1d3c4776012fc0bbe1f86e69bde2c5e36d2d32caa2e4845ac1fc08ab3f/packaging-21.3-py3-none-any.whl -> 21.3)
  • Updating charset-normalizer (2.0.12 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/ae/14/8e/087a104edb7ec627884755272de815d8ee4fd53b8745fdb12a087dd2e1/charset_normalizer-2.0.12-py3-none-any.whl -> 2.0.12)
  • Updating tomli (2.0.1 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/36/bd/11/98fa2af866c73d00f2498ade62c7fcf28db8656714b7b7a44e9cb2f855/tomli-2.0.1-py3-none-any.whl -> 2.0.1)
  • Updating yarl (1.7.2 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/a2/6a/27/e9e84dba14a393ddac771dd9ae65439811ec491d75b056170ce6150ec8/yarl-1.7.2-cp38-cp38-win_amd64.whl -> 1.7.2)
  • Updating aiohttp (3.8.1 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/73/ef/d8/f992ece989563634df5070549065fcdac89ed7d2ecdaf35697139624a9/aiohttp-3.8.1-cp38-cp38-win_amd64.whl -> 3.8.1)
  • Updating cycler (0.11.0 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/37/9c/25/cbaf98f95cc74b0a7617cef17520e3f063dbabd9f0f8a04f93a15f6632/cycler-0.11.0-py3-none-any.whl -> 0.11.0)
  • Updating pillow (9.1.1 C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/56/93/0d/15304aaa1b5631edde44db83af439458145ca775bb4a04a4596b43c17a/Pillow-9.1.1-cp38-cp38-win_amd64.whl -> 9.1.1)

mohan43u commented 2 years ago

check "Default Directories" section in this doc https://python-poetry.org/docs/master/configuration/ . It provides details about how you can override the default directories

adam-grant-hendry commented 2 years ago

@mohan43u Thank you, but I think I have those set already?

The above printout of config --list shows cache-dir set to the folder I want, and

PS> $env:POETRY_CACHE_DIR
C:\Users\username\.poetry\Cache

is set as a system environment variable. I even cleared my cache

PS> poetry cache clear --all .

and regardless poetry updates from 33 wheels stored in %LOCALAPPDATA% from the existing version to the same version.

Using -vvv option when running poetry update doesn't give me any details for the 33 updates.

NOTE: In the above, I've replaced my actual username with "username". Poetry shows the right path to my %USERPROFILE%.

adam-grant-hendry commented 2 years ago

@mohan43u @abn After disabling the experimental.new-installer, poetry began warning that the .dist-info folders for these packages were invalid and should be removed. In each of these, the direct_url.json file points to a wheel in %LOCALAPPDATA%.

For example, numpy was one of these and the numpy-1.22.4.dist-info/direct_url.json was

{"archive_info": {}, "url": "file:///C:/Users/username/AppData/Local/pypoetry/Cache/artifacts/7e/42/12/cd512fbb1274bf257081cb661ba45881d717adc618df1a73d3a5532da8/numpy-1.22.4-cp38-cp38-win_amd64.whl"}

I believe if poetry is resolving and/or working with these json files that it is resolving them incorrectly and/or writing to them incorrectly. I'm not sure how poetry uses them. It clearly reads them (otherwise it wouldn't give an invalid distribution message).

mohan43u commented 2 years ago

@adam-grant-hendry You may need to set three environment variables to alter poetry's default directories instead of single POETRY_CACHE_DIR

POETRY_CONFIG_DIR
POETRY_DATA_DIR
POETRY_CACHE_DIR

adam-grant-hendry commented 2 years ago

You may need to set three environment variables to alter poetry's default directories instead of single POETRY_CACHE_DIR

Oh interesting....

Why is it that I need to set all 3?
Why doesn't simply using poetry config to set the values and have them stored in the config.toml suffice? Why do I also need to set environment variables?

suvoooo commented 1 year ago

Is this the right way to set cache and config dir when installing? curl -sSL https://install.python-poetry.org | POETRY_HOME=/new/path/to/poetry POETRY_CONFIG_DIR=/new/path/to/poetry/.config POETRY_DATA_DIR=/new/path/to/poetry/ POETRY_CACHE_DIR=/new/path/to/poetry/.cache python3 -

because the cache and config are still set to ~/.cache/pypoetry and ~/.config/pypoetry. Where I'm doing it wrong?

github-actions[bot] commented 8 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

python-poetry / poetry

POETRY_CACHE_DIR or cache-dir config are not used at all. #2445

Issue