ybkscht / EfficientPose

Other
247 stars 68 forks source link

evaluation phase for object 8 of linemod -- raise ValueError('Unexpectedly found an instance of type `' + str(type(x)) + ValueError: Unexpectedly found an instance of type `<class 'keras.src.engine.keras_tensor.KerasTensor'>`. Expected a symbolic tensor instance. #71

Closed monajalal closed 7 months ago

monajalal commented 10 months ago

I am getting this error in TensorFlow 2.0 for evaluation

(effpose) mona@ada:~/effpose_new/EfficientPose$ python evaluate.py --phi 0 --weights weights/Weights/Linemod/object_8/phi_0_linemod_best_ADD.h5 --validation-image-save-path val_imgs linemod  ./data/Linemod_preprocessed/ --object-id 8
2023-11-28 15:55:47.220295: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-11-28 15:55:47.257482: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-28 15:55:47.257531: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-28 15:55:47.257560: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-28 15:55:47.264950: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-28 15:55:48.840055: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1886] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 45076 MB memory:  -> device: 0, name: NVIDIA RTX 6000 Ada Generation, pci bus id: 0000:52:00.0, compute capability: 8.9
{'dataset_type': 'linemod', 'rotation_representation': 'axis_angle', 'weights': 'weights/Weights/Linemod/object_8/phi_0_linemod_best_ADD.h5', 'batch_size': 1, 'phi': 0, 'gpu': None, 'score_threshold': 0.5, 'validation_image_save_path': 'val_imgs', 'linemod_path': './data/Linemod_preprocessed/', 'object_id': 8}

Creating the Generators...
Done!

Building the Model...
input shape: (512, 512, 3)
type(image_input): <class 'keras.src.engine.keras_tensor.KerasTensor'>
image_input shape:   (None, 512, 512, 3)
input_tensor shape is (None, 512, 512, 3)
type input_tensor:  <class 'keras.src.engine.keras_tensor.KerasTensor'>
Traceback (most recent call last):
  File "/home/mona/effpose_new/EfficientPose/evaluate.py", line 370, in <module>
    main()
  File "/home/mona/effpose_new/EfficientPose/evaluate.py", line 111, in main
    _, prediction_model, _ = build_EfficientPose(args.phi,
  File "/home/mona/effpose_new/EfficientPose/model.py", line 105, in build_EfficientPose
    backbone_feature_maps = backbone_class(input_tensor = image_input, freeze_bn = freeze_bn)
  File "/home/mona/effpose_new/EfficientPose/utils/__init__.py", line 61, in wrapper
    return func(*args, **kwargs)
  File "/home/mona/effpose_new/EfficientPose/efficientnet.py", line 459, in EfficientNetB0
    return EfficientNet(1.0, 1.0, 224, 0.2,
  File "/home/mona/effpose_new/EfficientPose/efficientnet.py", line 386, in EfficientNet
    if not is_keras_tensor(input_tensor):
  File "/home/mona/anaconda3/envs/effpose/lib/python3.10/site-packages/tensorflow/python/keras/backend.py", line 1248, in is_keras_tensor
    raise ValueError('Unexpectedly found an instance of type `' + str(type(x)) +
ValueError: Unexpectedly found an instance of type `<class 'keras.src.engine.keras_tensor.KerasTensor'>`. Expected a symbolic tensor instance.
(effpose) mona@ada:~/effpose_new/EfficientPose$ pip list
Package                              Version
------------------------------------ ------------
absl-py                              2.0.0
action-msgs                          1.2.1
action-tutorials-interfaces          0.20.3
action-tutorials-py                  0.20.3
actionlib-msgs                       4.2.3
ament-cmake-test                     1.3.5
ament-copyright                      0.12.8
ament-cppcheck                       0.12.8
ament-cpplint                        0.12.8
ament-flake8                         0.12.8
ament-index-python                   1.4.0
ament-lint                           0.12.8
ament-lint-cmake                     0.12.8
ament-package                        0.14.0
ament-pep257                         0.12.8
ament-uncrustify                     0.12.8
ament-xmllint                        0.12.8
angles                               1.15.0
asttokens                            2.4.1
astunparse                           1.6.3
builtin-interfaces                   1.2.1
cachetools                           5.3.2
certifi                              2023.11.17
charset-normalizer                   3.3.2
composition-interfaces               1.2.1
contourpy                            1.2.0
cv-bridge                            3.2.1
cycler                               0.12.1
Cython                               3.0.5
demo-nodes-py                        0.20.3
diagnostic-msgs                      4.2.3
diagnostic-updater                   3.1.2
domain-coordinator                   0.10.0
example-interfaces                   0.9.3
examples-rclpy-executors             0.15.1
examples-rclpy-minimal-action-client 0.15.1
examples-rclpy-minimal-action-server 0.15.1
examples-rclpy-minimal-client        0.15.1
examples-rclpy-minimal-publisher     0.15.1
examples-rclpy-minimal-service       0.15.1
examples-rclpy-minimal-subscriber    0.15.1
exceptiongroup                       1.1.3
executing                            2.0.1
filelock                             3.9.0
flatbuffers                          23.5.26
fonttools                            4.45.1
fsspec                               2023.4.0
gast                                 0.5.4
geometry-msgs                        4.2.3
google-auth                          2.23.4
google-auth-oauthlib                 1.0.0
google-pasta                         0.2.0
grpcio                               1.59.3
h5py                                 3.10.0
idna                                 3.6
image-geometry                       3.2.1
imageio                              2.32.0
imgaug                               0.4.0
interactive-markers                  2.3.2
ipdb                                 0.13.13
ipython                              8.17.2
jedi                                 0.19.1
Jinja2                               3.1.2
joblib                               1.3.2
keras                                2.14.0
Keras-Applications                   1.0.8
kiwisolver                           1.4.5
laser-geometry                       2.4.0
launch                               1.0.4
launch-ros                           0.19.6
launch-testing                       1.0.4
launch-testing-ros                   0.19.6
launch-xml                           1.0.4
launch-yaml                          1.0.4
lazy_loader                          0.3
libclang                             16.0.6
lifecycle-msgs                       1.2.1
logging-demo                         0.20.3
map-msgs                             2.1.0
Markdown                             3.5.1
MarkupSafe                           2.1.3
matplotlib                           3.8.2
matplotlib-inline                    0.1.6
message-filters                      4.3.3
ml-dtypes                            0.2.0
nav-msgs                             4.2.3
networkx                             3.0
numpy                                1.26.2
nvidia-cublas-cu11                   11.11.3.6
nvidia-cuda-cupti-cu11               11.8.87
nvidia-cuda-nvcc-cu11                11.8.89
nvidia-cuda-runtime-cu11             11.8.89
nvidia-cudnn-cu11                    8.7.0.84
nvidia-cufft-cu11                    10.9.0.58
nvidia-curand-cu11                   10.3.0.86
nvidia-cusolver-cu11                 11.4.1.48
nvidia-cusparse-cu11                 11.7.5.86
nvidia-nccl-cu11                     2.16.5
oauthlib                             3.2.2
opencv-python                        4.8.1.78
opt-einsum                           3.3.0
osrf-pycommon                        2.0.2
packaging                            23.2
parso                                0.8.3
pcl-msgs                             1.0.0
pendulum-msgs                        0.20.3
Pillow                               10.0.1
pip                                  23.3.1
plyfile                              1.0.2
progressbar2                         4.2.0
prompt-toolkit                       3.0.41
protobuf                             4.25.1
pure-eval                            0.2.2
pyasn1                               0.5.1
pyasn1-modules                       0.3.0
pycocotools                          2.0.7
PyOpenGL                             3.1.7
pyparsing                            3.1.1
python-dateutil                      2.8.2
python-qt-binding                    1.1.1
python-utils                         3.8.1
PyYAML                               6.0.1
qt-dotgraph                          2.2.2
qt-gui                               2.2.2
qt-gui-cpp                           2.2.2
qt-gui-py-common                     2.2.2
quality-of-service-demo-py           0.20.3
rcl-interfaces                       1.2.1
rclpy                                3.3.10
rcutils                              5.1.3
realsense2-camera-msgs               4.54.1
requests                             2.31.0
requests-oauthlib                    1.3.1
resource-retriever                   3.1.1
rmw-dds-common                       1.6.0
ros2action                           0.18.7
ros2bag                              0.15.8
ros2cli                              0.18.7
ros2component                        0.18.7
ros2doctor                           0.18.7
ros2interface                        0.18.7
ros2launch                           0.19.6
ros2lifecycle                        0.18.7
ros2multicast                        0.18.7
ros2node                             0.18.7
ros2param                            0.18.7
ros2pkg                              0.18.7
ros2run                              0.18.7
ros2service                          0.18.7
ros2topic                            0.18.7
rosbag2-interfaces                   0.15.8
rosbag2-py                           0.15.8
rosbags                              0.9.16
rosgraph-msgs                        1.2.1
rosidl-adapter                       3.1.5
rosidl-cli                           3.1.5
rosidl-cmake                         3.1.5
rosidl-generator-c                   3.1.5
rosidl-generator-cpp                 3.1.5
rosidl-generator-py                  0.14.4
rosidl-parser                        3.1.5
rosidl-runtime-py                    0.9.3
rosidl-typesupport-c                 2.0.1
rosidl-typesupport-cpp               2.0.1
rosidl-typesupport-fastrtps-c        2.2.1
rosidl-typesupport-fastrtps-cpp      2.2.1
rosidl-typesupport-introspection-c   3.1.5
rosidl-typesupport-introspection-cpp 3.1.5
rpyutils                             0.2.1
rqt-action                           2.0.1
rqt-bag                              1.1.4
rqt-bag-plugins                      1.1.4
rqt-console                          2.0.2
rqt-graph                            1.3.0
rqt-gui                              1.1.5
rqt-gui-py                           1.1.5
rqt-msg                              1.2.0
rqt-plot                             1.1.2
rqt-publisher                        1.5.0
rqt-py-common                        1.1.5
rqt-py-console                       1.0.2
rqt-reconfigure                      1.1.1
rqt-service-caller                   1.0.5
rqt-shell                            1.0.2
rqt-srv                              1.0.3
rqt-topic                            1.5.0
rsa                                  4.9
ruamel.yaml                          0.18.5
ruamel.yaml.clib                     0.2.8
scikit-image                         0.22.0
scikit-learn                         1.3.2
scipy                                1.11.4
sensor-msgs                          4.2.3
sensor-msgs-py                       4.2.3
setuptools                           68.2.2
shape-msgs                           4.2.3
shapely                              2.0.2
six                                  1.16.0
sros2                                0.10.4
stack-data                           0.6.3
statistics-msgs                      1.2.1
std-msgs                             4.2.3
std-srvs                             4.2.3
stereo-msgs                          4.2.3
teleop-twist-keyboard                2.3.2
tensorboard                          2.14.1
tensorboard-data-server              0.7.2
tensorflow                           2.14.1
tensorflow-estimator                 2.14.0
tensorflow-io-gcs-filesystem         0.34.0
tensorrt                             8.5.3.1
termcolor                            2.3.0
tf2-geometry-msgs                    0.25.4
tf2-kdl                              0.25.4
tf2-msgs                             0.25.4
tf2-py                               0.25.4
tf2-ros-py                           0.25.4
tf2-tools                            0.25.4
threadpoolctl                        3.2.0
tifffile                             2023.9.26
tomli                                2.0.1
topic-monitor                        0.20.3
torch                                2.1.1+cu118
torchaudio                           2.1.1+cu118
torchvision                          0.16.1+cu118
tqdm                                 4.66.1
traitlets                            5.13.0
trajectory-msgs                      4.2.3
triton                               2.1.0
turtlesim                            1.4.2
typeguard                            4.1.5
typing_extensions                    4.8.0
unique-identifier-msgs               2.2.1
urllib3                              2.1.0
visualization-msgs                   4.2.3
wcwidth                              0.2.10
Werkzeug                             3.0.1
wheel                                0.41.2
wrapt                                1.14.1
yacs                                 0.1.8
zstandard                            0.22.0

sys info:

(base) mona@ada:~$ lsb_release -a
LSB Version:    core-11.1.0ubuntu4-noarch:security-11.1.0ubuntu4-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:    22.04
Codename:   jammy
(base) mona@ada:~$ uname -a
Linux ada 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov  2 18:01:13 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
(base) mona@ada:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
(base) mona@ada:~$ nvidia-smi 
Tue Nov 28 15:58:14 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.12             Driver Version: 535.104.12   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX 6000 Ada Gene...    On  | 00000000:52:00.0  On |                  Off |
| 30%   43C    P8              27W / 300W |   1597MiB / 49140MiB |     13%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2317      G   /usr/lib/xorg/Xorg                          608MiB |
|    0   N/A  N/A      2519      G   /usr/bin/gnome-shell                        107MiB |
|    0   N/A  N/A      2994      G   ...AAAAAAAACAAAAAAAAAA= --shared-files      100MiB |
|    0   N/A  N/A     25264      G   ...0208189,17325718055376231948,262144       44MiB |
|    0   N/A  N/A    652962      G   ...irefox/3358/usr/lib/firefox/firefox      288MiB |
|    0   N/A  N/A    703622      G   blender                                     227MiB |
|    0   N/A  N/A    810224      G   ...sion,SpareRendererForSitePerProcess      101MiB |
|    0   N/A  N/A    829624      G   /usr/bin/gnome-control-center                 7MiB |
+---------------------------------------------------------------------------------------+

I have

(effpose) mona@ada:~/effpose_new/EfficientPose$ python
Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.