ImportError ('funcs' from 'ipart.utils') impeding functionality

[sadielbartholomew]

Towards openjournals/joss-reviews#2407 review, I am trying to confirm a working installation and sufficient claimed functionality but, after a standard install as per the current instructions, I am running immediately into the same exception, ImportError.

Namely, using the following commands in the root directory of my fork of the IPART repo, updated to the current state of the IPART master branch:

$ conda env create -f environment_py3.yml
...
$ conda activate ipartpy3
(ipartpy3) $ pip install -e .
...
(ipartpy3) $ python
...
(ipartpy3) $ python -m unittest discover -s tests
... <full output copied below> 

the unit tests, the scripts and the notebooks all immediately error from various usage of from ipart.utils import funcs giving the ultimate exception of:

ImportError: cannot import name 'funcs' from 'ipart.utils' (unknown location)

(although running import ipart in the Python session is successful and does not output that, or indeed any, error). I have included the full unit test run, and some representative script, outputs below if they may be helpful.

It seems an old location may be referenced for the import in question, or perhaps it is an attempted circular import. Whatever the cause, this issue needs resolving since this error is preventing any access to most of the functionality of IPART. Given the form it should be quick to fix.

Exact outputs

All with the same underlying error, but just to illustrate the extent to which it is emerging...

Some script outputs

$ cd scripts
$ python compute_ivt.py 
Traceback (most recent call last):
  File "compute_ivt.py", line 28, in <module>
    from ipart.utils import funcs
$ python trace_ARs.py 
Traceback (most recent call last):
  File "trace_ARs.py", line 65, in <module>
    from ipart.utils import plot
  File "/home/sadie/IPART/ipart/utils/plot.py", line 11, in <module>
    from . import funcs as functions
ImportError: cannot import name 'funcs' from 'ipart.utils' (unknown location)

Full test output

Running python -m unittest discover -s tests under the root IPART repo directory returns:

EEEEE
======================================================================
ERROR: test_data (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_data
Traceback (most recent call last):
  File "/home/sadie/anaconda3/envs/ipartpy3/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/envs/ipartpy3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/IPART/tests/test_data.py", line 15, in <module>
    from ipart.utils import funcs
ImportError: cannot import name 'funcs' from 'ipart.utils' (unknown location)

======================================================================
ERROR: test_ipart (test_install.TestIPRT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/sadie/IPART/tests/test_install.py", line 13, in test_ipart
    from ipart.thr import THR, rotatingTHR
ModuleNotFoundError: No module named 'ipart.thr'

======================================================================
ERROR: test_river_tracker1 (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_river_tracker1
Traceback (most recent call last):
  File "/home/sadie/anaconda3/envs/ipartpy3/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/envs/ipartpy3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/IPART/tests/test_river_tracker1.py", line 14, in <module>
    from ipart.utils import funcs
ImportError: cannot import name 'funcs' from 'ipart.utils' (unknown location)

======================================================================
ERROR: test_river_tracker2 (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_river_tracker2
Traceback (most recent call last):
  File "/home/sadie/anaconda3/envs/ipartpy3/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/envs/ipartpy3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/IPART/tests/test_river_tracker2.py", line 13, in <module>
    from ipart.AR_tracer import readCSVRecord, trackARs, filterTracks
  File "/home/sadie/IPART/ipart/AR_tracer.py", line 19, in <module>
    from ipart.utils import funcs
ImportError: cannot import name 'funcs' from 'ipart.utils' (unknown location)

======================================================================
ERROR: test_thr (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_thr
Traceback (most recent call last):
  File "/home/sadie/anaconda3/envs/ipartpy3/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/envs/ipartpy3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/IPART/tests/test_thr.py", line 14, in <module>
    from ipart.utils import funcs
ImportError: cannot import name 'funcs' from 'ipart.utils' (unknown location)

----------------------------------------------------------------------
Ran 5 tests in 0.001s

FAILED (errors=5)

[Xunius]

@sadielbartholomew Thanks for the report. I repeated your steps but didn't manage to reproduce the error. I didn't get the same ImportError, and the unit tests ran OK (aside from another unrelated error which I have fixed).

I searched a bit about this unknown location error, I don't think I got anything particularly useful. Some one said it might be a name conflict. Do you happen to have something named funcs that is importable, maybe related to one of the records in the PYTHONPATH env variable?

[sadielbartholomew]

Hi @Xunius, thanks for your quick response and investigations. If this is indeed a mistake on my part, which seems the likely case, then I am sorry. However, I did create a clean conda environment as shown in my opening comment, so I want to make sure that it is not an issue others an run into before I close this. Please bear with me while I investigate.

Some one said it might be a name conflict. Do you happen to have something named funcs that is importable, maybe related to one of the records in the PYTHONPATH env variable?

Good idea and that is probably the case. I tried running a script with python -v to get further details about the imports, and though I cannot find any other funcs mentioned in the verbose output, I do see there is a potential namespace clash on utils as there is also a numpy.lib.utils imported. So I suspect that may be the underlying issue. I am just trying to confirm this or otherwise.

I'll report back by the end of today on my findings. Thanks for your patience.

[sadielbartholomew]

Hi again, I have isolated the point at which the error is introduced. It is the pip install -e . command, which the installation instructions in the README state to use but not the docs. Either way, If I instead install IPART via conda, conda install -c conda-forge ipart, the unit tests run and pass for me. But with the development install with pip install -e . they fail with the ImportError as described above.

So this would imply that codebase in the state uploaded to conda-forge does not have produce issue, but the current state of the codebase as in the master branch does have the issue present. It seems like a change made since the conda-forge upload ~2 months ago something has changed to result in the error.

Given that for the release of the JOSS paper we will need you to create a tagged release of IPART to associate with the paper and to archive, we need the current branch to be working and to provide the necessary functionality. So in conclusion, I do think this is a problem that needs resolving, because there is something broken on master, rather than an issue with my own environment setup.

Does that make sense? I can always ask the editor or my co-reviewer to double check to see if they can see this issue too with pip install -e . if you would like someone to double check. Thanks.

[Xunius]

@sadielbartholomew Hi. But that doesn't explain why I could not reproduce the error. I can install from conda or pip install -e ., both worked for me. The last conda upload was fixing the dependency issue, I removed the basemap module from the dependency list, then nothing significant got changed since then. So I don't think I introduced the error to master that would trigger such a dramatic failure.

[sadielbartholomew]

OK that's fine, if it also works for you with pip install -e . then I am thinking again it is likely to be an error in my environment. I've been creating a clean ipartpy3 env but maybe there is something in my base env or similar that is interacting. Sorry about this, I will try to get to the bottom of it today. Thanks for bearing with me.

[Xunius]

@sadielbartholomew I'm creating an update on the conda version, but my internet has been real slow today. I'll give you a notice when that's done, then you can maybe give it another try and see if the same issue persists in the conda install as well.

[sadielbartholomew]

then you can maybe give it another try and see if the same issue persists in the conda install as well.

The conda install is fine, there is no such issue when I install with conda, the error just appears with pip install -e .. So it is the current codebase that might be the issue, if not something about the way I installed it or my environment.

[Xunius]

@sadielbartholomew I've updated the conda version to the newest 3.0.7 so it is now the same as the master branch. And I did a test using conda and pip install -e ., again both fine. Are you familiar with google colab? Maybe we could try the pip approach from within google colab? I've never used it before, I didn't managed to run the conda env create -f env.yml command, probably something missing. Here is my current attempt: https://colab.research.google.com/drive/1dK5nH9hu9x7UC_tlLq4aHvC73f_lrt35?usp=sharing

[sadielbartholomew]

Sorry @Xunius, your comment above must have fell off my radar. I have been finishing my review and have since worked out why I was seeing those errors. The good news is that it was indeed a user error on my part, namely that some of the modules had disappeared in my forked repo, including the func.py, so naturally the tests were complaining that they couldn't import that module! I have no idea how it happened, perhaps a rm command was passed without my realising.

Apologies for the mistake. The errors reported above are not at all issues with the library, rather that I managed to mess up my repo somehow. I'll close this now :slightly_smiling_face:. FYI I have ensured my repo is in sync with the master branch and with no missing files and all the tests pass:

$ python -m unittest discover -s tests
.......
# <prepareMeta>: Metadata created.
.
# <prepareMeta>: Metadata created.

# <findARsGen>: Processing time: 1984-02-01 00:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-01 06:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-01 12:00
# <findARsGen>: NO. of ARs found =  3

# <findARsGen>: Processing time: 1984-02-01 18:00
# <findARsGen>: NO. of ARs found =  3

# <findARsGen>: Processing time: 1984-02-02 00:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-02 06:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-02 12:00
# <findARsGen>: NO. of ARs found =  1

# <findARsGen>: Processing time: 1984-02-02 18:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-03 00:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-03 06:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-03 12:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-03 18:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-04 00:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-04 06:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-04 12:00
# <findARsGen>: NO. of ARs found =  1

# <findARsGen>: Processing time: 1984-02-04 18:00
# <findARsGen>: NO. of ARs found =  1

# <findARsGen>: Processing time: 1984-02-05 00:00
# <findARsGen>: NO. of ARs found =  1

# <findARsGen>: Processing time: 1984-02-05 06:00
# <findARsGen>: NO. of ARs found =  1

# <findARsGen>: Processing time: 1984-02-05 12:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-05 18:00
# <findARsGen>: NO. of ARs found =  2

# <findARsGen>: Processing time: 1984-02-06 00:00
# <findARsGen>: NO. of ARs found =  3

# <findARsGen>: Processing time: 1984-02-06 06:00
# <findARsGen>: NO. of ARs found =  3

# <findARsGen>: Processing time: 1984-02-06 12:00
# <findARsGen>: NO. of ARs found =  3

# <findARsGen>: Processing time: 1984-02-06 18:00
# <findARsGen>: NO. of ARs found =  3

# <findARsGen>: Processing time: 1984-02-07 00:00
# <findARsGen>: NO. of ARs found =  3
.
# <prepareMeta>: Metadata created.
...
----------------------------------------------------------------------
Ran 12 tests in 9.011s

OK

Or equivalently:

$ pytest
============================= test session starts ==============================
platform linux -- Python 3.7.8, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/sadie/IPART
collected 12 items                                                             

tests/test_data.py ......                                                [ 50%]
tests/test_install.py .                                                  [ 58%]
tests/test_river_tracker1.py ...                                         [ 83%]
tests/test_river_tracker2.py .                                           [ 91%]
tests/test_thr.py .                                                      [100%]

============================= 12 passed in 14.26s ==============================

via pytest.

[sadielbartholomew]

(I should add that once I noticed some modules were missing I thought it could be an issue with the .gitignore specifications meaning some files would not be staged from development, but I checked and it looks fine, so there is no issue there either.)