ybkscht / EfficientPose

Other
236 stars 67 forks source link

training using COCO pretrained weights for object 9 of linemod -- ValueError: Unexpectedly found an instance of type `<class 'keras.src.engine.keras_tensor.KerasTensor'>`. Expected a symbolic tensor instance. #72

Closed monajalal closed 4 months ago

monajalal commented 7 months ago

I get this error. Please let me know what you suggest

(effpose) mona@ada:~/effpose_new/EfficientPose$ python train.py --phi 0 --weights weights/Weights/Linemod/object_8/phi_0_linemod_best_ADD.h5 linemod data/Linemod_preprocessed/ --object-id 8
2023-11-28 16:04:48.938173: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-11-28 16:04:48.975203: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-28 16:04:48.975250: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-28 16:04:48.975277: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-28 16:04:48.982670: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-28 16:04:50.568105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1886] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 44806 MB memory:  -> device: 0, name: NVIDIA RTX 6000 Ada Generation, pci bus id: 0000:52:00.0, compute capability: 8.9
{'dataset_type': 'linemod', 'rotation_representation': 'axis_angle', 'weights': 'weights/Weights/Linemod/object_8/phi_0_linemod_best_ADD.h5', 'freeze_backbone': False, 'no_freeze_bn': False, 'batch_size': 1, 'lr': 0.0001, 'no_color_augmentation': False, 'no_6dof_augmentation': False, 'phi': 0, 'gpu': None, 'epochs': 500, 'steps': 1790, 'snapshot_path': 'checkpoints/28_11_2023_16_04_50', 'tensorboard_dir': 'logs/28_11_2023_16_04_50', 'snapshots': True, 'evaluation': True, 'compute_val_loss': False, 'score_threshold': 0.5, 'validation_image_save_path': None, 'multiprocessing': False, 'workers': 4, 'max_queue_size': 10, 'linemod_path': 'data/Linemod_preprocessed/', 'object_id': 8}

Creating the Generators...
Done!

Building the Model...
input shape: (512, 512, 3)
type(image_input): <class 'keras.src.engine.keras_tensor.KerasTensor'>
image_input shape:   (None, 512, 512, 3)
input_tensor shape is (None, 512, 512, 3)
type input_tensor:  <class 'keras.src.engine.keras_tensor.KerasTensor'>
Traceback (most recent call last):
  File "/home/mona/effpose_new/EfficientPose/train.py", line 370, in <module>
    main()
  File "/home/mona/effpose_new/EfficientPose/train.py", line 132, in main
    model, prediction_model, all_layers = build_EfficientPose(args.phi,
  File "/home/mona/effpose_new/EfficientPose/model.py", line 105, in build_EfficientPose
    backbone_feature_maps = backbone_class(input_tensor = image_input, freeze_bn = freeze_bn)
  File "/home/mona/effpose_new/EfficientPose/utils/__init__.py", line 61, in wrapper
    return func(*args, **kwargs)
  File "/home/mona/effpose_new/EfficientPose/efficientnet.py", line 459, in EfficientNetB0
    return EfficientNet(1.0, 1.0, 224, 0.2,
  File "/home/mona/effpose_new/EfficientPose/efficientnet.py", line 386, in EfficientNet
    if not is_keras_tensor(input_tensor):
  File "/home/mona/anaconda3/envs/effpose/lib/python3.10/site-packages/tensorflow/python/keras/backend.py", line 1248, in is_keras_tensor
    raise ValueError('Unexpectedly found an instance of type `' + str(type(x)) +
ValueError: Unexpectedly found an instance of type `<class 'keras.src.engine.keras_tensor.KerasTensor'>`. Expected a symbolic tensor instance.