Tasks run from wrong folder on update with 6.0.0a6

[chris-sanders]

I switched from using copier 5.1.0 to the latest 6.0.0a6 to avoid copier deleting a folder that it didn't create. However with the v6 version I'm having an issue with tasks getting executed by invoke during a copier update.

With version 5 the tasks run as expected, but with v6 during an update tasks that call invoke don't run from the destination folder but from the template sub-folder, where it can't find the expected files.

The template can be found at the v6 branch here: https://github.com/charmed-kubernetes/pytest-operator-template/tree/v6

You can run the tox -e integration and you'll see that it fails when it invokes one of the scripts because it can't find the file, which is in the destination folder the task is just not being run in the correct location.

[yajo]

Thanks for testing the alpha and for reporting, I'll add this to the v6 roadmap.

[chris-sanders]

I've been looking at this more and I've now triggered it on the v5 aslo. But I think I see what's actually happening. As part of the update process the template is cloned into an empty directory and then diffed against the actual final directory. During the initial copy into the tmp directory it's running the tasks and I have tasks which expect files in the destination directory which fail.

I'm not sure how to best handle that. I can catch the errors and just exit clean, but I actually wanted to fail if the destination folder doesn't match expectation. I'm applying this template as a delta on top of another folder structure and If the structure doesn't match what I expect I would like it to fail.

[yajo]

There are a couple of ideas that come to my mind.

1. Use invoke's --search-root to specify where to look for it. I use it in one of our templates and it works fine.

2. I actually wanted to fail if the destination folder doesn't match expectation

   Currently Copier only runs tasks after the copy. What you want is probably a better system that runs them before, to see if the project expectations are matched. That progress is being tracked in #240 and I hope it's ready for v6 final.

[yajo]

You can run the tox -e integration and you'll see that it fails when it invokes one of the scripts because it can't find the file, which is in the destination folder the task is just not being run in the correct location.

Tests fail with something that doesn't seem related to copier:

``` ╰─ pipx run tox -e integration integration create: /tmp/tmp.srxrxoudz4/pytest-operator-template/.tox/integration integration installdeps: invoke, copier>=6.0.0a6, pytest, black, pyyaml integration installed: appdirs==1.4.4,attrs==20.3.0,black==20.8b1,click==7.1.2,colorama==0.4.4,copier==6.0.0a6,iniconfig==1.1.1,invoke==1.5.0,iteration-utilities==0.10.1,Jinja2==2.11.3,MarkupSafe==1.1.1,mypy-extensions==0.4.3,packaging==20.9,pathspec==0.8.1,pluggy==0.13.1,plumbum==1.7.0,prompt-toolkit==3.0.18,py==1.10.0,pydantic==1.8.1,Pygments==2.8.1,pyparsing==2.4.7,pytest==6.2.2,PyYAML==5.4.1,pyyaml-include==1.2.post2,questionary==1.9.0,regex==2021.3.17,toml==0.10.2,typed-ast==1.4.2,typing-extensions==3.7.4.3,wcwidth==0.2.5 integration run-test-pre: PYTHONHASHSEED='2696812564' integration run-test: commands[0] | pytest -x -v /tmp/tmp.srxrxoudz4/pytest-operator-template/tests/integration/ --provider machine ============================================================================================================ test session starts ============================================================================================================= platform linux -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /tmp/tmp.srxrxoudz4/pytest-operator-template/.tox/integration/bin/python cachedir: .tox/integration/.pytest_cache rootdir: /tmp/tmp.srxrxoudz4/pytest-operator-template, configfile: tox.ini collected 8 items tests/integration/test_template.py::TestTemplate::test_pytest ERROR [ 12%] =================================================================================================================== ERRORS =================================================================================================================== _________________________________________________________________________________________________ ERROR at setup of TestTemplate.test_pytest _________________________________________________________________________________________________ session_folder = PosixPath('/tmp/pytest-of-yajo/pytest-0/session0') @pytest.fixture(scope="session") def charm_dir(session_folder): charm_dir = session_folder / "charm-dir" tmp_dir = Path("charm-dir") tmp_dir.mkdir() > subprocess.check_call( ["charmcraft", "init", "--author", "Pytest Conftest"], cwd=tmp_dir ) tests/integration/conftest.py:55: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python3.9/subprocess.py:368: in check_call retcode = call(*popenargs, **kwargs) /usr/lib64/python3.9/subprocess.py:349: in call with Popen(*popenargs, **kwargs) as p: /usr/lib64/python3.9/subprocess.py:951: in __init__ self._execute_child(args, executable, preexec_fn, close_fds, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , args = ['charmcraft', 'init', '--author', 'Pytest Conftest'], executable = b'charmcraft', preexec_fn = None, close_fds = True, pass_fds = () cwd = PosixPath('charm-dir'), env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1, p2cwrite = -1, c2pread = -1, c2pwrite = -1, errread = -1, errwrite = -1, restore_signals = True, gid = None, gids = None uid = None, umask = -1, start_new_session = False def _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session): """Execute program (POSIX version)""" if isinstance(args, (str, bytes)): args = [args] elif isinstance(args, os.PathLike): if shell: raise TypeError('path-like args is not allowed when ' 'shell is true') args = [args] else: args = list(args) if shell: # On Android the default shell is at '/system/bin/sh'. unix_shell = ('/system/bin/sh' if hasattr(sys, 'getandroidapilevel') else '/bin/sh') args = [unix_shell, "-c"] + args if executable: args[0] = executable if executable is None: executable = args[0] sys.audit("subprocess.Popen", executable, args, cwd, env) if (_USE_POSIX_SPAWN and os.path.dirname(executable) and preexec_fn is None and not close_fds and not pass_fds and cwd is None and (p2cread == -1 or p2cread > 2) and (c2pwrite == -1 or c2pwrite > 2) and (errwrite == -1 or errwrite > 2) and not start_new_session and gid is None and gids is None and uid is None and umask < 0): self._posix_spawn(args, executable, env, restore_signals, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite) return orig_executable = executable # For transferring possible exec failure from child to parent. # Data format: "exception name:hex errno:description" # Pickle is not used; it is complex and involves memory allocation. errpipe_read, errpipe_write = os.pipe() # errpipe_write must not be in the standard io 0, 1, or 2 fd range. low_fds_to_close = [] while errpipe_write < 3: low_fds_to_close.append(errpipe_write) errpipe_write = os.dup(errpipe_write) for low_fd in low_fds_to_close: os.close(low_fd) try: try: # We must avoid complex work that could involve # malloc or free in the child process to avoid # potential deadlocks, thus we do all this here. # and pass it to fork_exec() if env is not None: env_list = [] for k, v in env.items(): k = os.fsencode(k) if b'=' in k: raise ValueError("illegal environment variable name") env_list.append(k + b'=' + os.fsencode(v)) else: env_list = None # Use execv instead of execve. executable = os.fsencode(executable) if os.path.dirname(executable): executable_list = (executable,) else: # This matches the behavior of os._execvpe(). executable_list = tuple( os.path.join(os.fsencode(dir), executable) for dir in os.get_exec_path(env)) fds_to_keep = set(pass_fds) fds_to_keep.add(errpipe_write) self.pid = _posixsubprocess.fork_exec( args, executable_list, close_fds, tuple(sorted(map(int, fds_to_keep))), cwd, env_list, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, errpipe_read, errpipe_write, restore_signals, start_new_session, gid, gids, uid, umask, preexec_fn) self._child_created = True finally: # be sure the FD is closed no matter what os.close(errpipe_write) self._close_pipe_fds(p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite) # Wait for exec to fail or succeed; possibly raising an # exception (limited in size) errpipe_data = bytearray() while True: part = os.read(errpipe_read, 50000) errpipe_data += part if not part or len(errpipe_data) > 50000: break finally: # be sure the FD is closed no matter what os.close(errpipe_read) if errpipe_data: try: pid, sts = os.waitpid(self.pid, 0) if pid == self.pid: self._handle_exitstatus(sts) else: self.returncode = sys.maxsize except ChildProcessError: pass try: exception_name, hex_errno, err_msg = ( errpipe_data.split(b':', 2)) # The encoding here should match the encoding # written in by the subprocess implementations # like _posixsubprocess err_msg = err_msg.decode() except ValueError: exception_name = b'SubprocessError' hex_errno = b'0' err_msg = 'Bad exception data from child: {!r}'.format( bytes(errpipe_data)) child_exception_type = getattr( builtins, exception_name.decode('ascii'), SubprocessError) if issubclass(child_exception_type, OSError) and hex_errno: errno_num = int(hex_errno, 16) child_exec_never_called = (err_msg == "noexec") if child_exec_never_called: err_msg = "" # The error must be from chdir(cwd). err_filename = cwd else: err_filename = orig_executable if errno_num != 0: err_msg = os.strerror(errno_num) > raise child_exception_type(errno_num, err_msg, err_filename) E FileNotFoundError: [Errno 2] No such file or directory: 'charmcraft' /usr/lib64/python3.9/subprocess.py:1823: FileNotFoundError ========================================================================================================== short test summary info =========================================================================================================== ERROR tests/integration/test_template.py::TestTemplate::test_pytest - FileNotFoundError: [Errno 2] No such file or directory: 'charmcraft' !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ============================================================================================================== 1 error in 0.42s ============================================================================================================== ERROR: InvocationError for command /tmp/tmp.srxrxoudz4/pytest-operator-template/.tox/integration/bin/pytest -x -v tests/integration --provider machine (exited with code 1) __________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________ ERROR: integration: commands failed ```

So I ran the test manually:

``` ╰─ copier --version copier 6.0.0a5.post28+386e50e ╰─ cd (mktemp -d) ╰─ git clone https://github.com/charmed-kubernetes/pytest-operator-template -b v6 Clonando en 'pytest-operator-template'... remote: Enumerating objects: 193, done. remote: Counting objects: 100% (193/193), done. remote: Compressing objects: 100% (98/98), done. remote: Total 193 (delta 84), reused 165 (delta 62), pack-reused 0 Recibiendo objetos: 100% (193/193), 29.41 KiB | 792.00 KiB/s, listo. Resolviendo deltas: 100% (84/84), listo. ╰─ copier copy ./pytest-operator-template/ dst No git tags found in template; using HEAD as ref 🎤 The class name for the charm, must be a valid python class name. This is in src/charm.py for existing charms. class_name? Format: str MyClass 🎤 charm_type? Format: str container create . create tox.ini create tests create tests/unit create tests/unit/test_charm.py.tmpl create tests/unit/empty create tests/integration create tests/integration/test_charm.py create tests/data create tests/data/empty create tasks.py create src create src/charm.py.tmpl create [[_copier_conf.answers_file]].tmpl create .github create .github/workflows create .github/workflows/tests.yaml.tmpl > Running task 1 of 3: invoke check-yaml Traceback (most recent call last): File "/usr/bin/invoke", line 33, in sys.exit(load_entry_point('invoke==1.4.1', 'console_scripts', 'invoke')()) File "/usr/lib/python3.9/site-packages/invoke/program.py", line 384, in run self.execute() File "/usr/lib/python3.9/site-packages/invoke/program.py", line 566, in execute executor.execute(*self.tasks) File "/usr/lib/python3.9/site-packages/invoke/executor.py", line 129, in execute result = call.task(*args, **call.kwargs) File "/usr/lib/python3.9/site-packages/invoke/tasks.py", line 127, in __call__ result = self.body(*args, **kwargs) File "/tmp/tmp.srxrxoudz4/dst/tasks.py", line 40, in check_yaml config = yaml.safe_load(config_file.read_text()) File "/usr/lib64/python3.9/pathlib.py", line 1255, in read_text with self.open(mode='r', encoding=encoding, errors=errors) as f: File "/usr/lib64/python3.9/pathlib.py", line 1241, in open return io.open(self, mode, buffering, encoding, errors, newline, File "/usr/lib64/python3.9/pathlib.py", line 1109, in _opener return self._accessor.open(self, flags, mode) FileNotFoundError: [Errno 2] No such file or directory: 'config.yaml' Traceback (most recent call last): File "/var/home/yajo/mydevel/copier/.venv/bin/copier", line 5, in CopierApp.run() File "/var/home/yajo/mydevel/copier/.venv/lib/python3.9/site-packages/plumbum/cli/application.py", line 614, in run inst, retcode = subapp.run(argv, exit=False) File "/var/home/yajo/mydevel/copier/.venv/lib/python3.9/site-packages/plumbum/cli/application.py", line 609, in run retcode = inst.main(*tailargs) File "/var/home/yajo/mydevel/copier/copier/cli.py", line 40, in _wrapper return method(*args, **kwargs) File "/var/home/yajo/mydevel/copier/copier/cli.py", line 251, in main self.parent._worker( File "/var/home/yajo/mydevel/copier/copier/main.py", line 568, in run_copy self._execute_tasks( File "/var/home/yajo/mydevel/copier/copier/main.py", line 189, in _execute_tasks subprocess.run(task_cmd, shell=use_shell, check=True, env=local.env) File "/usr/lib64/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'invoke check-yaml' returned non-zero exit status 1. ```

The interesting part is:

File "/tmp/tmp.srxrxoudz4/dst/tasks.py", line 40, in check_yaml
  config = yaml.safe_load(config_file.read_text())

Which shows clearly that invoke finds the tasks.py file and that it gets executed inside the destination directory.

Also, the failure:

FileNotFoundError: [Errno 2] No such file or directory: 'config.yaml'

Is correct:

╰─ env LANG=C ls dst/config.yaml
ls: cannot access 'dst/config.yaml': No such file or directory

So, AFAICS, all these are bugs in the template itself, not something related to copier.

Regarding the issue at hand, our CI asserts that the tasks are executed in the dst directory always. You can check the test here, and see that it's passing. Unless there's a bug in the test, this issue is fixed in our side: https://github.com/copier-org/copier/blob/93d95a6a7bda7ac3bc201356e63e7cfacd810dad/tests/test_tasks.py#L8-L43

Feel free to reopen if needed 😊