Calling ecbundle-create from Python script seems not to propagate $PWD properly

uwefladrich commented 5 months ago

What happened?

When ecbundle-create is called from a Python script (as opposed to a shell script or directly on the command line) and

the bundle.yml file uses $PWD in dir: specs, and
the current working directory is changed before calling ecbundle-create, then

the dir: argument in the bundle file is not correctly resolved and ecbundle-create fails with

A directory [...] is provided for project [...] but it does not exist.

This does not happen when ecbundle-create is called from a Shell script.

What are the steps to reproduce the bug?

A minimal setup of files looks like this:

minimal-example/
├── ecbundle-create.py
├── ecbundle-create.sh
└── subdir/
    ├── arch/
    ├── bundle.yml
    ├── external/
    │   └── ecbuild/  # contains ecbuild clone
    └── source/

Where ecbundle-create.py and ecbundle-create.sh are the scripts I am testing, subdir/bundle.yml is the bundle file using the $PWD references, and subdir/external/ecbuild contains a cloned ecbuild repository. It is not really important to use ecbuild, any other project would do.

The bundle file is minimal:

name: ecbundletest
version: 0.1

projects:
  - ecbuild:
      dir: ${PWD}/external/ecbuild
      bundle: false

Now, from within minimal-example, I can run the ecbundle-create.sh script:

#!/usr/bin/sh
cd subdir
ecbundle-create

and everything works fine. The complete bundle is created in subdir/source/ and all links are correctly set.

However, running a Python script with (arguably) the same functionality:

import shutil
import subprocess
from pathlib import Path

subprocess.run(
    [shutil.which("ecbundle-create")],
    cwd=Path("subdir"),
)

results in

> python ecbundle-create.py 

Downloading bundle with 1 threads for 
   [...]/minimal-example/subdir/bundle.yml
A directory [.../minimal-example/external/ecbuild] is provided for project [ecbuild] but it does not exist.

Time elapsed for downloading: 00:00:00

Creating bundle for 
    [...]/minimal-example/subdir/bundle.yml

Bundle does not need updating at src-dir
    [...]/minimal-example/subdir/source

projects:
    - ecbuild (symbolic link to [.../minimal-example/external/ecbuild])

!!! Errors occured !!!

(common path prefixes replaced by "..." for readability)

It can be seen that ecbundle-create looks for the ecbuild code in minimal-example/external/, not in minimal-example/subdir/external/, as it should.

Version

2.1.1

Platform (OS and architecture)

Red Hat Enterprise Linux 9.3 (Plow)

Relevant log output

No response

Accompanying data

No response

Organisation

SMHI/EC-Earth

uwefladrich commented 5 months ago

Note that I at first suspected the way I am starting the ecbundle-create process from Python to be the problem (i.e. using subprocess.run() with no further arguments).

However, I tested with a small script:

#!/usr/bin/bash
echo "PWD=${PWD}"

and called it from Python in a subdirectory:

import subprocess
from pathlib import Path

subprocess.run(
    [Path("pwdtest.sh").resolve()],
    cwd=Path("subdir"),
)

and that works, i.e. $PWD points to the correct subdirectory.

I also checked if the shell=true argument to subprocess.run() made any difference, but it didn't.

uwefladrich commented 5 months ago

Note (mostly to myself): I am guessing that the os.path.expandvars() call in https://github.com/ecmwf/ecbundle/blob/814873bf2c770b2693b08181243dd05d110e0c2c/ecbundle/project.py#L26 is responsible for expanding the environment variables in dir: ....

uwefladrich commented 5 months ago

It seems that subprocess.Popen() will not set $PWD when using cwd=...: https://github.com/python/cpython/issues/48307 :disappointed:

uwefladrich commented 5 months ago

Since this is not an issue of ecbundle in the first place, I close the issue.

ecmwf / ecbundle