Building using `pip` does not copy certain files (related to config / data )

[abhaasgoyal]

As of now, running the latest version of benchcab af43dad provides error in missing certain files related to config/data

Reproducing bug

Assuming that you have a working conda environment from .conda/benchcab-dev.yaml:

conda create -n benchcap-dev --file benchcab-dev.yaml

1. Install benchcap from the home directory

   pip install --user .

2. Running benchcab provides the following error

   (benchcab-dev) [ag9761@gadi-login-07 bench_example]$ benchcab run
   Traceback (most recent call last):
   File "/home/659/ag9761/.local/bin/benchcab", line 8, in <module>
   sys.exit(main())
   File "/home/659/ag9761/.local/lib/python3.9/site-packages/benchcab/main.py", line 30, in main
   parse_and_dispatch(parser)
   File "/home/659/ag9761/.local/lib/python3.9/site-packages/benchcab/main.py", line 20, in parse_and_dispatch
   func(**args)
   File "/home/659/ag9761/.local/lib/python3.9/site-packages/benchcab/benchcab.py", line 314, in run
   self.fluxsite(config_path, no_submit, verbose, skip)
   File "/home/659/ag9761/.local/lib/python3.9/site-packages/benchcab/benchcab.py", line 299, in fluxsite
   self.checkout(config_path, verbose)
   File "/home/659/ag9761/.local/lib/python3.9/site-packages/benchcab/benchcab.py", line 185, in checkout
   config = self._get_config(config_path)
   File "/home/659/ag9761/.local/lib/python3.9/site-packages/benchcab/benchcab.py", line 102, in _get_config
   self._config = read_config(config_path)
   File "/home/659/ag9761/.local/lib/python3.9/site-packages/benchcab/config.py", line 94, in read_config
   validate_config(config)
   File "/home/659/ag9761/.local/lib/python3.9/site-packages/benchcab/config.py", line 54, in validate_config
   schema = bu.load_package_data("config-schema.yml")
   File "/home/659/ag9761/.local/lib/python3.9/site-packages/benchcab/utils/__init__.py", line 43, in load_package_data
   raw = pkgutil.get_data("benchcab", os.path.join("data", filename)).decode("utf-8")
   File "/home/659/ag9761/.conda/envs/benchcab-dev/lib/python3.9/pkgutil.py", line 639, in get_data
   return loader.get_data(resource_name)
   File "<frozen importlib._bootstrap_external>", line 1039, in get_data
   FileNotFoundError: [Errno 2] No such file or directory: '/home/659/ag9761/.local/lib/python3.9/site-packages/benchcab/data/config-schema.yml'

Further Steps Looking into why benchcap/data is not being copied to the local build. One of the reasons could be related to the build system in setup.py / related files

[SeanBryan51]

@abhaasgoyal I'm not sure I can reproduce your error. The following commands ran successfully for me:

module load conda
conda env remove --name benchcab-dev
conda env create --file .conda/benchcab-dev.yaml
conda activate benchcab-dev
pip install --user .
/bin/bash benchcab/data/test/integration.sh

This could be to do with how conda has been configured. Can you try these instructions for setting up personal conda environments on Gadi? http://climate-cms.wikis.unsw.edu.au/Conda#Creating_personal_environments

Above commands were tested using the latest version in main:

$ git fetch origin; git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

[abhaasgoyal]

Hi @SeanBryan51 , could you try running benchcap run on CABLE-LSM/bench_example, in the meantime I'll try running the integration script as well as your set of commands.

[abhaasgoyal]

Nevermind, it seems to be working now in a clean conda environment.

Happy to mark the issue as closed