Python 3.6.13 Error - Githubissues

RossPitman commented 3 years ago

Hi @marco-willi. We've recently had to upgrade python within our docker image, to run the latest version of PantheraIDS. We're still using the same version of Tensorflow, but now getting this error when trying to run the Southern Africa model:

Traceback (most recent call last): File "/srv/shiny-server/custom/machine_classifier/main_prediction/predict.py", line 105, in main() File "/srv/shiny-server/custom/machine_classifier/main_prediction/predict.py", line 85, in main aggregation_mode=args['aggregation_mode']) File "/srv/shiny-server/custom/machine_classifier/main_prediction/camera_trap_classifier/predicting/predictor.py", line 73, in init self.model = load_model_from_disk(self.model_path, compile=False) File "/srv/shiny-server/custom/machine_classifier/main_prediction/camera_trap_classifier/training/prepare_model.py", line 31, in load_model_from_disk build_masked_loss(K.sparse_categorical_crossentropy)}) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/keras/engine/saving.py", line 229, in load_model model_config = json.loads(model_config.decode('utf-8')) AttributeError: 'str' object has no attribute 'decode'

RossPitman commented 3 years ago

Any ideas where I should start looking to fix this?

RossPitman commented 3 years ago

pip freeze

absl-py==0.12.0 astor==0.8.1 awscli==1.19.25 botocore==1.20.25 cached-property==1.5.2 colorama==0.4.3 docutils==0.15.2 gast==0.4.0 grpcio==1.36.1 h5py==3.1.0 importlib-metadata==3.7.2 jmespath==0.10.0 Keras==2.4.3 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.2 Markdown==3.3.4 numpy==1.19.5 opencv-contrib-python==3.3.1.11 Pillow==8.1.2 protobuf==3.15.5 pyasn1==0.4.8 python-dateutil==2.8.1 PyYAML==5.4.1 rsa==4.7.2 s3transfer==0.3.4 scipy==1.5.4 six==1.15.0 tensorboard==1.12.2 tensorflow==1.12.0 termcolor==1.1.0 typing-extensions==3.7.4.3 urllib3==1.26.3 Werkzeug==1.0.1 zipp==3.4.1

RossPitman commented 3 years ago

FYI – we aren't using Anaconda anymore, in an attempt to save space, so installing everything manually now.

RossPitman commented 3 years ago

I decided to try downgrading the version of h5py:

pip install 'h5py==2.10.0' --force-reinstall

Which seems to overcome the above error, but resulted in another one:

WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:87: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with distribution=normal is deprecated and will be removed in a future version. Instructions for updating: normal is a deprecated alias for truncated_normal WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:87: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with distribution=normal is deprecated and will be removed in a future version. Instructions for updating: normal is a deprecated alias for truncated_normal Found 1651 images in /ids_volume/data/images/S18_20160902_20160923/images/ Traceback (most recent call last): File "/srv/shiny-server/custom/machine_classifier/main_prediction/predict.py", line 105, in main() File "/srv/shiny-server/custom/machine_classifier/main_prediction/predict.py", line 92, in main batch_size=args['batch_size']) File "/srv/shiny-server/custom/machine_classifier/main_prediction/camera_trap_classifier/predicting/predictor.py", line 103, in predict_from_image_dir batch_size, export_type) File "/srv/shiny-server/custom/machine_classifier/main_prediction/camera_trap_classifier/predicting/predictor.py", line 294, in _predict_inventory sub_preds = self._iterate_inventory_dataset(dataset, sub_inventory) File "/srv/shiny-server/custom/machine_classifier/main_prediction/camera_trap_classifier/predicting/predictor.py", line 391, in _iterate_inventory_dataset img_id = [i for i, x in enumerate(inventory[_id]['images']) KeyError: b'/ids_volume/data/images/S18_20160902_20160923/images/S18Station1Camera1CAM637912016-09-0211-33-401__1.JPG'

marco-willi commented 3 years ago

Hi Ross, Good to hear from you!

I think downgrading h5py was the right action according to issues that I found here: https://github.com/tensorflow/tensorflow/issues/44467

The error you report later is rather odd. It seems like it occurs here: https://github.com/marco-willi/camera-trap-classifier/blob/master/camera_trap_classifier/predicting/predictor.py#L389

Where the following key is not found in a dictionary: b'/ids_volume/data/images/S18_20160902_20160923/images/S18__Station1__Camera1__CAM63791__2016-09-02__11-33-40__1__1.JPG'

Strange is, that the key is a byte string b' instead of a normal str.

However, the following line ensures that the conversion from b' to str is being made: https://github.com/marco-willi/camera-trap-classifier/blob/master/camera_trap_classifier/predicting/predictor.py#L372.

Can you confirm that this line was not changed in your code?

RossPitman commented 3 years ago

Hi Marco. Thanks for your response! Yep, I can confirm that this line was not changed.

marco-willi commented 3 years ago

Here is the list of python packages installed in the Docker-Image of the classifier (see https://github.com/marco-willi/camera-trap-classifier/blob/master/Dockerfile.cpu).

Maybe this helps to find the issue (although not all the packages are actually needed). Note, that the 'camera-trap-classifier' is itself a package that you can use by invoking ctc.predict.
Another option may be to use separate python versions for IDS and the classifier. You could put everything in a venv and activate it before running the classifier.
You could also try to debug the issue and check the types of the key that causes the error. It is still a mystery to me how this happens based on the error log you sent.

absl-py==0.6.1 astor==0.7.1 backcall==0.1.0 bleach==3.0.2 camera-trap-classifier==2.1.0 cycler==0.10.0 decorator==4.3.0 defusedxml==0.5.0 entrypoints==0.2.3 gast==0.2.0 grpcio==1.16.0 h5py==2.8.0 ipykernel==5.1.0 ipython==7.1.1 ipython-genutils==0.2.0 ipywidgets==7.4.2 jedi==0.13.1 Jinja2==2.10 jsonschema==2.6.0 jupyter==1.0.0 jupyter-client==5.2.3 jupyter-console==6.0.0 jupyter-core==4.4.0 Keras-Applications==1.0.6 Keras-Preprocessing==1.0.5 kiwisolver==1.0.1 Markdown==3.0.1 MarkupSafe==1.1.0 matplotlib==3.0.1 mistune==0.8.4 nbconvert==5.4.0 nbformat==4.4.0 notebook==5.7.0 numpy==1.15.4 pandas==0.23.4 pandocfilters==1.4.2 parso==0.3.1 pexpect==4.6.0 pickleshare==0.7.5 Pillow==5.3.0 prometheus-client==0.4.2 prompt-toolkit==2.0.7 protobuf==3.6.1 ptyprocess==0.6.0 pycurl==7.43.0 Pygments==2.2.0 pygobject==3.20.0 pyparsing==2.3.0 python-apt==1.1.0b1+ubuntu0.16.4.2 python-dateutil==2.7.5 pytz==2018.7 PyYAML==5.3.1 pyzmq==17.1.2 qtconsole==4.4.2 scikit-learn==0.20.0 scipy==1.1.0 Send2Trash==1.5.0 six==1.11.0 sklearn==0.0 tensorboard==1.12.0 tensorflow==1.12.0 termcolor==1.1.0 terminado==0.8.1 testpath==0.4.2 tornado==5.1.1 traitlets==4.3.2 wcwidth==0.1.7 webencodings==0.5.1 Werkzeug==0.14.1 widgetsnbextension==3.4.2

RossPitman commented 3 years ago

Hi Marco. Strangely, after downgrading h5py, and rebooting the system, I cannot recreate the error. Seems to be working now. Sorry for the confusion!

marco-willi commented 3 years ago

Glad to hear!

marco-willi / camera-trap-classifier

Python 3.6.13 Error #26