Trouble setting up PyOD together with Keras and Tensorflow

[rwgsmits]

Hello,

I am very keen to use the PyOD models including the AutoEncoders. For this, I created a new conda environment with Python 3.7 and used the newest pip version to install pyod. This gives me a working environment. However, I am also interested in the AutoEncoders models, so used conda to install keras and tensorflow. After running the commands, as stated in the installation guide, it messes up the environment. Importing the pyod package then throws me the following error:

import pyod Traceback (most recent call last): File "", line 1, in File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\pyod__init.py", line 4, in from . import utils File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\pyod\utils__init.py", line 2, in from .utility import check_parameter File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\pyod\utils\utility.py", line 14, in import sklearn File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\sklearn__init.py", line 80, in from .base import clone File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\sklearn\base.py", line 21, in from .utils import _IS_32BIT File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\sklearn\utils__init__.py", line 23, in from .class_weight import compute_class_weight, compute_sample_weight File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\sklearn\utils\class_weight.py", line 7, in from .validation import _deprecate_positional_args File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\sklearn\utils\validation.py", line 25, in from .fixes import _object_dtype_isnan, parse_version File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\sklearn\utils\fixes.py", line 18, in import scipy.stats File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\scipy\stats__init__.py", line 388, in from .stats import * File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\scipy\stats\stats.py", line 180, in from . import distributions File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\scipy\stats\distributions.py", line 8, in from ._distn_infrastructure import (entropy, rv_discrete, rv_continuous, File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\scipy\stats_distn_infrastructure.py", line 23, in from scipy import optimize File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\scipy\optimize\init.py", line 387, in from .optimize import * File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\scipy\optimize\optimize.py", line 36, in from ._numdiff import approx_derivative File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\scipy\optimize_numdiff.py", line 6, in from scipy.sparse.linalg import LinearOperator File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\scipy\sparse\linalg\init__.py", line 114, in from .eigen import * File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\scipy\sparse\linalg\eigen\init__.py", line 9, in from .arpack import * File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\scipy\sparse\linalg\eigen\arpack\init__.py", line 20, in from .arpack import * File "C:\Users\rsmits004\Anaconda3\envs\anomaly_test\lib\site-packages\scipy\sparse\linalg\eigen\arpack\arpack.py", line 43, in from . import _arpack ImportError: DLL load failed: The specified procedure could not be found.

I found that conda downgrades my scipy from 1.5.2. (pre-installation keras and tensorflow) to 1.5.0. (post-installation). I included the list of packages pre and post installation.

Packages before and after conda install keras tensorflow.docx

Uninstalling and installing the scipy package to 1.5.2. again seems to get importing pyod and tensorflow working again, but the follow message in the Anaconda Prompt makes me a bit nervous:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

tensorflow 2.3.0 requires numpy<1.19.0,>=1.16.0, but you'll have numpy 1.19.1 which is incompatible. tensorflow 2.3.0 requires scipy==1.4.1, but you'll have scipy 1.5.2 which is incompatible.

Any recommendations on setting up the environment appropriately?

[andrewm4894]

iirc i got it working with Python 3.7 and:

Keras==2.3.1 
tensorflow==1.14.0

[yzhao062]

Yeah. We are leaving the setup of keras to the user. I think the latest keras+TF works with my clound env.

My local env works with tf=1.14.0 and keras==2.2.4