Invoke 2.1.0 / 2.1.1 is not able to read invoke.yaml configuration file

[vaz-ar]

Hello,

I updated my virtualenv this morning and was faced with the following issue:

❯ inv git.checkout-main
Traceback (most recent call last):
  File "/home/arnaud/.virtualenvs/invoke/lib64/python3.11/site-packages/invoke/config.py", line 117, in __getattr__
    return self._get(key)
           ^^^^^^^^^^^^^^
  File "/home/arnaud/.virtualenvs/invoke/lib64/python3.11/site-packages/invoke/config.py", line 177, in _get
    value = self._config[key]
            ~~~~~~~~~~~~^^^^^
  File "/home/arnaud/.virtualenvs/invoke/lib64/python3.11/site-packages/invoke/config.py", line 168, in __getitem__
    return self._get(key)
           ^^^^^^^^^^^^^^
  File "/home/arnaud/.virtualenvs/invoke/lib64/python3.11/site-packages/invoke/config.py", line 177, in _get
    value = self._config[key]
            ~~~~~~~~~~~~^^^^^
KeyError: 'git'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/arnaud/.virtualenvs/invoke/bin/inv", line 8, in <module>
    sys.exit(program.run())
             ^^^^^^^^^^^^^
  File "/home/arnaud/.virtualenvs/invoke/lib64/python3.11/site-packages/invoke/program.py", line 398, in run
    self.execute()
  File "/home/arnaud/.virtualenvs/invoke/lib64/python3.11/site-packages/invoke/program.py", line 583, in execute
    executor.execute(*self.tasks)
  File "/home/arnaud/.virtualenvs/invoke/lib64/python3.11/site-packages/invoke/executor.py", line 140, in execute
    result = call.task(*args, **call.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arnaud/.virtualenvs/invoke/lib64/python3.11/site-packages/invoke/tasks.py", line 138, in __call__
    result = self.body(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arnaud/toolbox/work/python/invoke/tasks/git.py", line 53, in checkout_main
    for workspace in ctx.git.workspaces:
                     ^^^^^^^
  File "/home/arnaud/.virtualenvs/invoke/lib64/python3.11/site-packages/invoke/config.py", line 129, in __getattr__
    raise AttributeError(err)
AttributeError: No attribute or config key found for 'git'

Valid keys: ['run', 'runners', 'sudo', 'tasks', 'timeouts']

Valid real attributes: ['cd', 'clear', 'config', 'cwd', 'from_data', 'pop', 'popitem', 'prefix', 'run', 'setdefault', 'sudo', 'update']

I have the following packages installed in my venv:

❯ python -m pip list
Package            Version
------------------ ---------
astroid            2.15.3
autopep8           2.0.1
certifi            2022.12.7
charset-normalizer 3.1.0
dill               0.3.6
distro             1.8.0
flake8             6.0.0
idna               3.4
invoke             2.1.1
isort              5.12.0
lazy-object-proxy  1.9.0
markdown-it-py     2.2.0
mccabe             0.7.0
mdurl              0.1.2
pip                23.1.2
platformdirs       2.6.2
pycodestyle        2.10.0
pyflakes           3.0.1
Pygments           2.14.0
pylint             2.17.2
requests           2.30.0
rich               13.3.5
setuptools         62.6.0
tabulate           0.9.0
tomlkit            0.11.6
urllib3            1.26.15
wheel              0.40.0
wrapt              1.14.1

I have the following config file in my home directory:

❯ cat ~/.invoke.yaml
# BEGIN ANSIBLE MANAGED BLOCK
tasks:
  search_root: /home/arnaud/toolbox/work/python/invoke
# END ANSIBLE MANAGED BLOCK

and in the /home/arnaud/toolbox/work/python/invoke I have

❯ ls -l /home/arnaud/toolbox/work/python/invoke
total 12
-rw-r--r-- 1 arnaud arnaud 4837 Apr 27 15:04 invoke.yaml
drwxr-xr-x 2 arnaud arnaud 4096 Apr 27 15:31 tasks

with the following in my invoke.yaml file:

git:
  workspaces:
    - /home/arnaud/git/CI
    - /home/arnaud/git/Products

This is working without any issue if I downgrade invoke to invoke<2.1.0 (2.0.1).

I can provide additional information if needed, just let me know.

[kuwv]

@vaz-ar I wasn't able to reproduce this. You might have to provide additional details.

Example:

from invoke import task

@task
def check(ctx):
    print(ctx.test)

Config file:

$ cat ~/.invoke.yaml 
---
test: value

$ cat $PWD/invoke.yaml 
---
test: value

Both outputs gave:

$ inv check
value

[vaz-ar]

@vaz-ar I wasn't able to reproduce this. You might have to provide additional details.

Example:

from invoke import task

@task
def check(ctx):
    print(ctx.test)

Config file:

$ cat ~/.invoke.yaml 
---
test: value

$ cat $PWD/invoke.yaml 
---
test: value

Both outputs gave:

$ inv check
value

Hello,

Thanks for your reply.

Here is the smallest example I was able to make to reproduce the problem, I uploaded it to gist: https://gist.github.com/vaz-ar/dac367545b16b905f070d56391acaae3

the https://gist.github.com/vaz-ar/dac367545b16b905f070d56391acaae3#file-dir_tree-txt show the directory structure.

I reproduce the problem by placing mysefl in the /tmp/inv_test dir and running inv testcheck.check

[vaz-ar]

I just tested and the issue is also there with 2.1.2

[kuwv]

I'll check it out tonight.

[retnuh]

We have the issue as well; our builds broke when we went from 2.0.0 -> 2.1.2.

The structure of our project is

tasks/
|-- __init__.py
|-- a.py
|-- b.y

And we would call invoke a.foo or whatever, but use an invoke.yml in the top level directory. In 2.1.2, it seems like the invoke.yml has to be in the tasks/ directory to get recognized.

[kuwv]

Apologies, took longer than I expected to get some free time.

Confirmed:

invoke.loader.find: FilesystemLoader find starting at '/tmp/example'
invoke.loader.find: Found module: ModuleSpec(name='tasks', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fe419984760>, origin='/tmp/example/tasks/__init__.py', submodule_search_locations=['/tmp/example/tasks'])
invoke.config._load_file: Didn't see any /tmp/example/tasks/invoke.yaml, skipping.
invoke.config._load_file: Didn't see any /tmp/example/tasks/invoke.yml, skipping.
invoke.config._load_file: Didn't see any /tmp/example/tasks/invoke.json, skipping.

[bitprophet]

Linking this to #934 since it's proximately related (same root cause - switching up how we find/import "the tasks") though I'm not yet sure if the right fix is anything like that one's was (which was literally about fixing import'ability, and this is more about ensuring the right location is looked at for the conf files).

Jesse's PR uses cwd which is only incidentally correct if you're sitting in the project dir; the best fix would be to identify the 'sibling' location to the tasks package. I am looking at this now. Also, why did tests not turn this up? We don't have the same excuse as the import problem, AIUI, so there may just be a missing test.

[bitprophet]

Fix identified, tests updated/added (real problem was in the Loader class, the changes to use importlib missed this case, as expected because there was a missing test or two), changelogged. Testing on the dayjob machine before I merge.

[bitprophet]

2.1.3 now on PyPI!

[vaz-ar]

Just installed it, I can confirm that it works as expected, many thanks!