search

USGS-R / river-dl

Deep learning model for predicting environmental variables on river systems
Creative Commons Zero v1.0 Universal
21 stars 15 forks source link

dependency issue on Tallgrass #15

Closed jsadler2 closed 4 years ago

jsadler2 commented 4 years ago

I have a dependency issue on Tallgrass. I emailed hpc@usgs.gov about it. Below is the correspondence.

jsadler2 commented 4 years ago

Me:

I am getting a weird dependency issue. I created a new conda environment, "rgcn," to run Tensorflow 2. I also installed a few other libraries my script needs including pyarrow. However, when I run my script via slurm, I get this error:

image

I'm confused b/c if I go into the analytics container and run python in my "rgcn" environment I can run the "read_feather" operation which is what is throwing the error when I do the slurm script. Also if do an ls on the ".../rgcn/../site-packages/," I see the "pyarrow" directory. So it seems like "pyarrow" is there... Any ideas why I'm getting the import error?

Tue 3/10/2020 10:45 AM

jsadler2 commented 4 years ago

Davis, Matthew (Contractor) J commented:

Hi Jeff,

This is indeed very confusing. I've been able to reproduce everything mentioned, but have had no luck solving it yet. I'll let you know what I can find.

-Matt

Thu 3/12/2020 4:58 PM

jsadler2 commented 4 years ago

Davis, Matthew (Contractor) J commented:

Hi Jeff,

I was able to get past this error, but I'm still not exactly sure of why it's having issues and am therefore not very happy with the solution. On line 10 of RGCN_tf2.py, I added "import pyarrow". From what I've tested, this line needs to happen before tensorflow is imported. My best guess is that for some reason when tensorflow is loaded it messes with library paths for the python environment. Pyarrow depends on a library which can then not be found anymore. By importing pyarrow before tensorflow, that library is held in memory so that future imports (which are made by pandas later) can be made successfully after tensorflow is loaded.

Here's an example: (rgcn) bash-4.4$ python Python 3.7.6 | packaged by conda-forge | (default, Mar 5 2020, 15:27:18) [GCC 7.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pyarrow >>> import tensorflow as tf

(rgcn) bash-4.4$ python Python 3.7.6 | packaged by conda-forge | (default, Mar 5 2020, 15:27:18) [GCC 7.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import tensorflow as tf >>> import pyarrow Traceback (most recent call last): File "", line 1, in File "/home/mdavis/.conda/envs/rgcn/lib/python3.7/site-packages/pyarrow/init.py", line 49, in from pyarrow.lib import cpu_count, set_cpu_count ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /home/mdavis/.conda/envs/rgcn/lib/python3.7/site-packages/pyarrow/../../../libarrow.so.16)

-Matt

Fri 3/13/2020 4:55 PM

jsadler2 commented 4 years ago

So it seems like I need to explicitly import pyarrow above tensorflow