search

LBL-EESA / TECA

TECA, theToolkit for Extreme Climate Analysis, contains a collection of climate anlysis algorithms targetted at extreme event detection and analysis.
Other
58 stars 21 forks source link

Tests for setup.py install of TECA #256

Open taobrienlbl opened 5 years ago

taobrienlbl commented 5 years ago

Issue #254 and PR #255 appear to suggest that we need tests that exercise the setup.py installation method.

Perhaps this could be done via another docker image -- one that:

  1. downloads miniconda,
  2. installs the necessary packages,
  3. attempts to install via setup.py, and
  4. runs one or more of the python tests.

@elbashandy, @burlen - any thoughts about this?

elbashandy commented 5 years ago

I agree. Another Travis-ci job could be added specifically to only run setup.py. I see it's quite redundant to run python tests as they are being run in the cmake builds. Please correct me if I am mistaken.

Additional option is to create tests that use the generated apps like teca_convert_table (including the non python ones) which will test the functionality and the presence of these applications.

taobrienlbl commented 5 years ago

Glad we are on the same page. I think running at least one python test is critical, since a successful build/install doesn’t guarantee that the code will actually function in python. The test would verify that the code can both be imported and utilized from python after the setup.py install is complete.

On Thu, Jun 13, 2019 at 5:44 PM Abdelrahman Elbashandy < notifications@github.com> wrote:

I agree. Another Travis-ci job could be added specifically to only run setup.py. I see it's quite redundant to run python tests as they are being run in the cmake builds. Please correct me if I am mistaken.

Additional option is to create tests that use the generated apps like teca_convert_table (including the non python ones) which will test the functionality and the presence of these applications.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/LBL-EESA/TECA/issues/256?email_source=notifications&email_token=ACDDUFSA2BXGBG6UA4XE3X3P2LSYHA5CNFSM4HYCGYR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXVM65A#issuecomment-501927796, or mute the thread https://github.com/notifications/unsubscribe-auth/ACDDUFS64R5GJQB5XYVHSBDP2LSYHANCNFSM4HYCGYRQ .

-- Travis A. O'Brien

http://eesa.lbl.gov/profiles/travis-a-obrien/ +1 (510) 495-8047

burlen commented 5 years ago

I like Abdels idea of adding a Travis ci run to do a 'python setup.py install'. And after the last fiasco of testing this by 'from teca import *' I feel like it is critical to run some code in the installed python environment. This would not be our crest suite but rather a snippet that verifies that the install is functional. Perhaps looping over all the classes, constructing each, and printing it's dir. Or something similar that exercises the install

burlen commented 5 years ago

@elbashandy would you please take this one?

elbashandy commented 5 years ago

@burlen Sure!

burlen commented 5 years ago

@elbashandy This is working for me on Ubuntu 18.04 with Python 3. Before continuing with this, please update all of the travis ci runs to use Python 3. This is critical step because on 1/1/2020 Python 2 is being officially retired. See #290. If after updating to Python 3, you continue to have issue please paste the full the output into the issue. Thanks!

elbashandy commented 4 years ago

I tried installing teca using python3 setup.py install on both, fedora:28 & ubuntu:18.04 docker containers with the current master. The installation pass successfully but problems start to happen when I try to import teca.

On Ubuntu importing teca directly gives this error:

>>> import teca
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/teca-2.2.2-py3.6-linux-x86_64.egg/teca/__init__.py", line 3, in <module>
    from teca.teca_py import *
  File "/usr/local/lib/python3.6/dist-packages/teca-2.2.2-py3.6-linux-x86_64.egg/teca/teca_py.py", line 18491, in <module>
    from teca_python_algorithm import *
  File "/usr/local/lib/python3.6/dist-packages/teca-2.2.2-py3.6-linux-x86_64.egg/teca/teca_python_algorithm.py", line 1, in <module>
    import teca_py
  File "/usr/local/lib/python3.6/dist-packages/teca-2.2.2-py3.6-linux-x86_64.egg/teca/teca_py.py", line 18494, in <module>
    from teca_tc_activity import *
  File "/usr/local/lib/python3.6/dist-packages/teca-2.2.2-py3.6-linux-x86_64.egg/teca/teca_tc_activity.py", line 5, in <module>
    class teca_tc_activity(teca_py.teca_python_algorithm):
AttributeError: module 'teca_py' has no attribute 'teca_python_algorithm'

However, import teca_py works, and interestingly following it with import teca succeeds as well.

>>> import teca_py
>>> import teca
>>>

On fedora, I get this error:

>>> import teca
Traceback (most recent call last):
  File "/usr/local/lib64/python3.6/site-packages/teca-2.2.2-py3.6-linux-x86_64.egg/teca/teca_py.py", line 33, in swig_import_helper
    return importlib.import_module(mname)
  File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: libxlsxwriter.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib64/python3.6/site-packages/teca-2.2.2-py3.6-linux-x86_64.egg/teca/__init__.py", line 3, in <module>
    from teca.teca_py import *
  File "/usr/local/lib64/python3.6/site-packages/teca-2.2.2-py3.6-linux-x86_64.egg/teca/teca_py.py", line 36, in <module>
    _teca_py = swig_import_helper()
  File "/usr/local/lib64/python3.6/site-packages/teca-2.2.2-py3.6-linux-x86_64.egg/teca/teca_py.py", line 35, in swig_import_helper
    return importlib.import_module('_teca_py')
  File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: libxlsxwriter.so: cannot open shared object file: No such file or directory

I reinstalled libxlsxwriter but still doesn't work.

I will try to investigate more. We can also look at it in our next workathon.

taobrienlbl commented 4 years ago

Brought this up again in 1/30 meeting - should have a dedicated meeting about this

burlen commented 4 years ago

this is because libxlsxwriter.so is not in LD_LIBRARY_PATH. You can add that path and it should solve.

taobrienlbl commented 4 years ago

we may just want to remove libxlsx from the docker image -- after discussing, it seems that this is not in the test suite anyway.

elbashandy commented 4 years ago

Rebase to make sure it's still happening.

burlen commented 4 years ago 
AttributeError: module 'teca_py' has no attribute 'teca_python_algorithm'

I hit this one as well when I did the PyPi package for 3.0.0 (see 0cdf5cd). This should be fixed now.

elbashandy commented 4 years ago

I did test this on Fedora & ubuntu docker images and both gave me this error when I tried:

>>> from teca import *
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/teca-3.0.0-py3.6-linux-x86_64.egg/teca/teca_py.py", line 33, in swig_import_helper
    return importlib.import_module(mname)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: libxlsxwriter.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/teca-3.0.0-py3.6-linux-x86_64.egg/teca/__init__.py", line 3, in <module>
    from teca.teca_py import *
  File "/usr/local/lib/python3.6/dist-packages/teca-3.0.0-py3.6-linux-x86_64.egg/teca/teca_py.py", line 36, in <module>
    _teca_py = swig_import_helper()
  File "/usr/local/lib/python3.6/dist-packages/teca-3.0.0-py3.6-linux-x86_64.egg/teca/teca_py.py", line 35, in swig_import_helper
    return importlib.import_module('_teca_py')
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: libxlsxwriter.so: cannot open shared object file: No such file or directory

I also set the LD_LIBRARY_PATH & PYTHONPATH to where libxlsxwriter is located, and still getting the same error.

@burlen please let me know if I missed something.

elbashandy commented 4 years ago

Update:

issue #441 is related to this issue, I can't complete an installation with setup.py without fixing the circular dependency. The main issue I am facing is that I am trying to only import teca_deeplabv3p_ar_detect from alg/teca_deeplabv3p_ar_detect.py because there are other objects in that file that we don't want exposed, like the Neural Network ResNet(nn.Module) class for example.

The import part is done in teca_py_alg.i:

/***************************************************************************
 model_segmentation & deeplabv3p_ar_detect
 ***************************************************************************/
%pythoncode %{
try:
    from teca_model_segmentation import *
    from teca_deeplabv3p_ar_detect import teca_deeplabv3p_ar_detect
except ImportError:
    pass
%}

changing from teca_deeplabv3p_ar_detect import teca_deeplabv3p_ar_detect -> from teca_deeplabv3p_ar_detect import * fixes the problem but we don't want that.

I am working on finding an alternative solution, but let me know if anyone has any suggestions.

burlen commented 4 years ago

One improvement would be to rename those classes. For eg. ResNet could be renamed to teca_resnet. then it wouldn't matter if they got pulled in. this cold be done with sed in a matter of seconds from the command line. a further improvement would be to nest all of the supporting classes in an outer class for instance teca_model_segmentation_internals. Then it is obvious that these are not supposed to be user accessible codes.

elbashandy commented 4 years ago

Ah! Nested classes.. great suggestion. Will start working on it right away. Thanks Burlen