logger.partitioning is accessed before definition

[lgouldsbrough]

logger.partitioning is accessed before definition using Python version: Python 3.8.11; cf-python version 3.11.0 from pypi

Using import cf immediately throws an attribute error: 'Logger' object has no attribute 'partitioning'. The expected behaviour is that logger.partitioning is defined before it is called, allowing for the package to be imported. Following the traceback shows that logger.partitioning is accessed by way of line 194 in cf/.init.py:

  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/X/X_env/lib/python3.8/site-packages/cf/__init__.py", line 194, in <module>
    from .mixin import Coordinate
  File "/home/X/X_env/lib/python3.8/site-packages/cf/mixin/__init__.py", line 2, in <module>
    from .propertiesdata import PropertiesData, Subspace
  File "/home/X/X_env/lib/python3.8/site-packages/cf/mixin/propertiesdata.py", line 24, in <module>
    from ..timeduration import TimeDuration
  File "/home/X/X_env/lib/python3.8/site-packages/cf/timeduration.py", line 28, in <module>
    _one_year = Data(1, "calendar_years")
  File "/home/X/X_env/lib/python3.8/site-packages/cf/data/data.py", line 750, in __init__
    self._set_partition_matrix(
  File "/home/X/X_env/lib/python3.8/site-packages/cf/data/data.py", line 832, in _set_partition_matrix
    self.to_disk()
  File "/home/X/X_env/lib/python3.8/site-packages/cf/data/data.py", line 12125, in to_disk
    partition.close()
  File "/home/X/X_env/lib/python3.8/site-packages/cf/data/partition.py", line 788, in close
    logger.partitioning("Partition.close: original = {}".format(original))
AttributeError: 'Logger' object has no attribute 'partitioning'

logger.partitioning is defined on line 330 in cf/.init.py and is not created before any submodules are imported.

A possible solution is to move the logger definitions to before importing the submodules.

[sadielbartholomew]

Hi @lgouldsbrough, thanks very much for reporting this, and sorry you encountered an immediate issue with cf-python.

I can't reproduce this error, and am dubious as to how such an immediate error would have not shown up in our pre-release testing process, though your logic seems sensible so I am keen to understand why you are getting the error when we didn't.

When I investigated to try to reproduce the issue, I created a clean conda environment with Python 3.8.11 and installed the PyPI version 3.11.0 but everything seems fine:

$ conda list
# packages in environment at /home/sadie/anaconda3/envs/testlatestcf:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
cf-python                 3.11.0                   pypi_0    pypi
cfdm                      1.9.0.1                  pypi_0    pypi
cftime                    1.5.1                    pypi_0    pypi
cfunits                   3.3.4                    pypi_0    pypi
expat                     2.4.1                h9c3ff4c_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 11.2.0              h1d223b6_11    conda-forge
libgomp                   11.2.0              h1d223b6_11    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_11    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
netcdf-flattener          1.2.0                    pypi_0    pypi
netcdf4                   1.5.7                    pypi_0    pypi
numpy                     1.21.3                   pypi_0    pypi
openssl                   1.1.1l               h7f98852_0    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
psutil                    5.8.0                    pypi_0    pypi
python                    3.8.11          h12debd9_0_cpython    defaults
python_abi                3.8                      2_cp38    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                58.2.0           py38h578d9bd_0    conda-forge
sqlite                    3.36.0               h9cd32fc_2    conda-forge
tk                        8.6.11               h27826a3_1    conda-forge
udunits2                  2.2.27.27            hc3e0081_2    conda-forge
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h36c2ea0_1013    conda-forge
$ python
Python 3.8.11 (default, Aug  3 2021, 15:09:35) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cf
>>> cf.__version__
'3.11.0'

Would you mind sharing more information about the environment you are using to help us understand what might be going on resulting in you running into that error? Are you sure you are running version 3.11.0, for example? Can you share a pip list (and a conda list if you use conda channels for package installation as well as PyPI)?

[MirandaLv]

I had the similar error when I tried to install cf-python on my virtual environment. Here is my steps to install:

1. create a new Conda environment with Conda create -n myenv python=3.8
2. install cf packages following the instructions on https://ncas-cms.github.io/cf-python/installation.html#conda: conda install -c ncas -c conda-forge cf-python cf-plot udunits2 conda install -c conda-forge mpich esmpy
3. import cf, received below error message saying AttributeError: 'Logger' object has no attribute 'partitioning'
4. I also tried to install the package from source package, and received the same error.

Any suggestions? Thank you.

[MirandaLv]

In case it is useful, below is my Conda list packages:

[MirandaLv]

@sadielbartholomew I just posted the errors/environments that come with this bug, thank you.

[sadielbartholomew]

Hi @MirandaLv, sorry for the delay in replying (we are very busy with urgent pre-Christmas tasks) and thanks for your further report on this, especially for including the necessary detail on your environment and instructions on how to get the right environment.

I will look into this soon, if not later today then early next week. I am still unsure as to how such a critical error is emerging for some and not others, e.g. not showing up on our local and continuous integration testing - if present, I would have thought the error indicated would emerge for everyone, so there is possibly some subtle packaging issue at play here. Regardless of the cause, I will investigate to find out.

Of course, if by following our installation instructions you and @lgouldsbrough and possibly others immediately run into an error, that is a bad bug that needs to be fixed.

When I have more idea what might be going, I will comment here again. Thanks.

[jocheco]

Hi @sadielbartholomew, I am really new to the package and also to the whole NetCDF area.

I have just found the same problem using python 3.8 and python 3.9. I managed to get a working version with python 3.7 and cf 3.1.

Here some information from my side, if it can be of any help. I am working on a Centos8 headless system and the procedure followed in all cases was in the lines of the following listing. Cheers.

└─ $ > conda create -n rd39 python=3.9
└─ $ > conda activate rd39
└─ $ > conda install -c ncas -c conda-forge cf-python cf-plot udunits2 
└─ $ > python
Python 3.9.12 (main, Apr  5 2022, 06:56:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/envs/rd39/lib/python3.9/site-packages/cf/__init__.py", line 207
, in <module>
    from .mixin import Coordinate
  File "/opt/anaconda3/envs/rd39/lib/python3.9/site-packages/cf/mixin/__init__.py", li
ne 2, in <module>
    from .propertiesdata import PropertiesData, Subspace
  File "/opt/anaconda3/envs/rd39/lib/python3.9/site-packages/cf/mixin/propertiesdata.p
y", line 14, in <module>
    from ..timeduration import TimeDuration
  File "/opt/anaconda3/envs/rd39/lib/python3.9/site-packages/cf/timeduration.py", line
 33, in <module>
    _one_year = Data(1, "calendar_years")
  File "/opt/anaconda3/envs/rd39/lib/python3.9/site-packages/cf/data/data.py", line 77
2, in __init__
    self._set_partition_matrix(
  File "/opt/anaconda3/envs/rd39/lib/python3.9/site-packages/cf/data/data.py", line 85
4, in _set_partition_matrix
    self.to_disk()
  File "/opt/anaconda3/envs/rd39/lib/python3.9/site-packages/cf/data/data.py", line 12
148, in to_disk
    partition.close()
  File "/opt/anaconda3/envs/rd39/lib/python3.9/site-packages/cf/data/partition.py", li
ne 792, in close
    logger.partitioning("Partition.close: original = {}".format(original))
AttributeError: 'Logger' object has no attribute 'partitioning'