Dependencies missing: `dask`, `distributed`, `colorlog`

[jkikstra]

dask

{
    "name": "ModuleNotFoundError",
    "message": "No module named 'dask'",
    "stack": "---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[4], line 1
----> 1 from concordia import RegionMapping

File c:\\Users\\kikstra\\Documents\\GitHub\\emissions_harmonization_historical\\.venv\\Lib\\site-packages\\concordia\\__init__.py:8
      5 from importlib.metadata import version as _version
      7 from .report import add_hypothesis, add_plotly_header, add_sticky_toc, embed_image
----> 8 from .utils import (
      9     CondordiaMagics,
     10     RegionMapping,
     11     VariableDefinitions,
     12 )
     15 try:
     16     __version__ = _version(\"concordia\")

File c:\\Users\\kikstra\\Documents\\GitHub\\emissions_harmonization_historical\\.venv\\Lib\\site-packages\\concordia\\utils.py:10
      7 from pathlib import Path
      8 from typing import Self, TypeAlias
---> 10 import dask.distributed as dd
     11 import numpy as np
     12 import pandas as pd

ModuleNotFoundError: No module named 'dask'"
}

distributed

ModuleNotFoundError                       Traceback (most recent call last)
File c:\Users\kikstra\Documents\GitHub\emissions_harmonization_historical\.venv\Lib\site-packages\dask\distributed.py:13
     [12](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/dask/distributed.py:12) try:
---> [13](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/dask/distributed.py:13)     from distributed import *
     [14](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/dask/distributed.py:14) except ImportError as e:

ModuleNotFoundError: No module named 'distributed'

The above exception was the direct cause of the following exception:

ImportError                               Traceback (most recent call last)
Cell In[4], [line 1](vscode-notebook-cell:?execution_count=4&line=1)
----> [1](vscode-notebook-cell:?execution_count=4&line=1) from concordia import RegionMapping

File c:\Users\kikstra\Documents\GitHub\emissions_harmonization_historical\.venv\Lib\site-packages\concordia\__init__.py:8
      [5](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/concordia/__init__.py:5) from importlib.metadata import version as _version
      [7](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/concordia/__init__.py:7) from .report import add_hypothesis, add_plotly_header, add_sticky_toc, embed_image
----> [8](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/concordia/__init__.py:8) from .utils import (
      [9](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/concordia/__init__.py:9)     CondordiaMagics,
     [10](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/concordia/__init__.py:10)     RegionMapping,
     [11](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/concordia/__init__.py:11)     VariableDefinitions,
     [12](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/concordia/__init__.py:12) )
     [15](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/concordia/__init__.py:15) try:
     [16](file:///C:/Users/kikstra/Documents/GitHub/emissions_harmonization_historical/.venv/Lib/site-packages/concordia/__init__.py:16)     __version__ = _version("concordia")
...

colorlog

{
    "name": "ModuleNotFoundError",
    "message": "No module named 'colorlog'",
    "stack": "---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[4], line 1
----> 1 from concordia import RegionMapping

File c:\\Users\\kikstra\\Documents\\GitHub\\emissions_harmonization_historical\\.venv\\Lib\\site-packages\\concordia\\__init__.py:8
      5 from importlib.metadata import version as _version
      7 from .report import add_hypothesis, add_plotly_header, add_sticky_toc, embed_image
----> 8 from .utils import (
      9     CondordiaMagics,
     10     RegionMapping,
     11     VariableDefinitions,
     12 )
     15 try:
     16     __version__ = _version(\"concordia\")

File c:\\Users\\kikstra\\Documents\\GitHub\\emissions_harmonization_historical\\.venv\\Lib\\site-packages\\concordia\\utils.py:15
     13 import pycountry
     14 from attrs import define
---> 15 from colorlog import ColoredFormatter
     16 from IPython.core.magic import Magics, cell_magic, magics_class
     17 from pandas import DataFrame, isna

ModuleNotFoundError: No module named 'colorlog'"
}

[coroa]

That sounds like you did not install the conda environment from environment.yml

[jkikstra]

You're right. And you're also right that that's is the way that it is currently supported - according to the readme.

I was doing it using poetry (when I was creating an environment for emissions_harmonization_historical), which I suppose looks at the pyproject.toml, which I think is out of sync?

[coroa]

It does look at that, but it's more by design than mistake. pyproject and pip/poetry and consorts in general have a very bad track record of working with anything that involves geospatial analysis around the gdal and osgeo libraries. It is in my experience safer to always manage environments with conda/mamba for those analyses. If zeb or others say that is no longer the case, i would be open to including them in pyproject.toml, but a few years back this has caused hard to debug headaches.

[znichollscr]

Nah all good @coroa thanks. We'll switch to pixi

[jkikstra]

Okidokes, so then I suppose this issue can be closed.

[coroa]

@znichollscr Thanks. pixi looks very helpful.

[znicholls]

Glad it helps. If you're interested, our default environment managers are now basically:

• anything needing conda dependencies, use pixi (an example repo doing the whole set is here if you're interested: https://github.com/climate-resource/input4mips_validation)
• everything else, use uv (e.g. here: https://github.com/climate-resource/bookshelf)

In both cases, we use pdm as the build backend because it has some features we like, like locked builds (this repo basically demonstrates it https://github.com/openscm/openscm-zenodo). I reckon that uv will have their own version of this pretty soon, at which point we might stop using pdm.