yuan0821
commented
1 year ago • (tf23) F:\finnaldannce\demo_video\mouse_4>dannce-predict dannce_mouse_config_4.yaml • 2023-05-04 01:56:56.987771: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll • io_config not found in io.yaml file, falling back to main config • new_n_channels_out not found in io.yaml file, falling back to main config • downfac not found in io.yaml file, falling back to main config • extension not found in io.yaml file, falling back to main config • batch_size not found in io.yaml file, falling back to main config • n_views not found in io.yaml file, falling back to main config • mono not found in io.yaml file, falling back to main config • debug not found in io.yaml file, falling back to main config • epochs not found in io.yaml file, falling back to main config • net_type not found in io.yaml file, falling back to main config • train_mode not found in io.yaml file, falling back to main config • num_validation_per_exp not found in io.yaml file, falling back to main config • vol_size not found in io.yaml file, falling back to main config • nvox not found in io.yaml file, falling back to main config • max_num_samples not found in io.yaml file, falling back to main config • loss not found in io.yaml file, falling back to main config • augment_brightness not found in io.yaml file, falling back to main config • n_rand_views not found in io.yaml file, falling back to main config • dannce_finetune_weights not found in io.yaml file, falling back to main config • predict_mode not found in io.yaml file, falling back to main config • com_train_dir set to: .\COM\train_results\ • com_predict_dir set to: .\COM\predict_results\ • dannce_train_dir set to: .\DANNCE\train_results\AVG\ • dannce_predict_dir set to: .\DANNCE\predict_results\ • exp set to: [{'label3d_file': 'F:\finnaldannce\demo_video\mouse_4\du_cam1_to_cam5_dannce.mat'}] • io_config set to: io.yaml • new_n_channels_out set to: 16 • downfac set to: 4 • extension set to: .avi • batch_size set to: 1 • n_views set to: 5 • mono set to: True • debug set to: True • epochs set to: 10 • net_type set to: AVG • train_mode set to: finetune • num_validation_per_exp set to: 0 • vol_size set to: 240 • nvox set to: 80 • max_num_samples set to: 100 • loss set to: mask_nan_l1_loss • augment_brightness set to: True • n_rand_views set to: None • dannce_finetune_weights set to: F:\finnaldannce\weights\max5\ • predict_mode set to: torch • base_config set to: dannce_mouse_config_4.yaml • viddir set to: videos • crop_height set to: None • crop_width set to: None • camnames set to: None • n_channels_out set to: 20 • sigma set to: 10 • verbose set to: 1 • net set to: None • gpu_id set to: 0 • immode set to: vid • mirror set to: False • start_batch set to: 0 • start_sample set to: None • com_fromlabels set to: False • medfilt_window set to: None • com_file set to: None • new_last_kernel_size set to: [3, 3, 3] • n_layers_locked set to: 2 • vmin set to: None • vmax set to: None • interp set to: nearest • depth set to: False • comthresh set to: 0 • weighted set to: False • com_method set to: median • cthresh set to: None • channel_combo set to: None • dannce_predict_model set to: None • expval set to: None • from_weights set to: None • write_npy set to: None • n_channels_in set to: None • vid_dir_flag set to: None • num_train_per_exp set to: None • chunks set to: None • lockfirst set to: None • load_valid set to: None • augment_hue set to: False • augment_hue_val set to: 0.05 • augment_bright_val set to: 0.05 • augment_rotation_val set to: 5 • drop_landmark set to: None • raw_im_h set to: None • raw_im_w set to: None • n_instances set to: 1 • use_npy set to: False • data_split_seed set to: None • valid_exp set to: None • metric set to: ['euclidean_distance_3D'] • lr set to: 0.001 • rotate set to: True • augment_continuous_rotation set to: False • com_thresh set to: None • cam3_train set to: None • debug_volume_tifdir set to: None • dannce_predict_vol_tifdir set to: None • rand_view_replace set to: True • multi_gpu_train set to: False • heatmap_reg set to: False • heatmap_reg_coeff set to: 0.01 • save_pred_targets set to: False • Using the following dannce.mat files: .\du_cam1_to_cam5_dannce.mat • Setting vid_dir_flag to True. • Setting extension to .avi. • Setting chunks to {'Camera1': array([0]), 'Camera2': array([0]), 'Camera3': array([0]), 'Camera4': array([0]), 'Camera5': array([0])}. • Setting n_channels_in to 3. • Setting raw_im_h to 2560. • Setting raw_im_w to 2560. • Setting expval to True. • Setting net to finetune_AVG. • Setting crop_height to [0, 2560]. • Setting crop_width to [0, 2560]. • Setting maxbatch to 100. • Setting start_batch to 0. • Setting vmin to -120.0. • Setting vmax to 120.0. • Setting n_rand_views to None. • Using the following dannce.mat files: .\du_cam1_to_cam5_dannce.mat • Using torch predict mode • Using camnames: ['Camera1', 'Camera2', 'Camera3', 'Camera4', 'Camera5'] • Experiment 0 using com3d: .\du_cam1_to_cam5_dannce.mat • Removed 0 samples from the dataset because they either had COM positions over cthresh, or did not have matching sampleIDs in the • COM file • Saving 3D COM to .\DANNCE\predict_results\com3d_used.mat • None • 2023-05-04 01:57:01.592851: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI • Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 • To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. • 2023-05-04 01:57:01.810827: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2053bf83360 initialized for platform • Host (this does not guarantee that XLA will be used). Devices: • 2023-05-04 01:57:01.811082: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version • 2023-05-04 01:57:01.815819: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll • 2023-05-04 01:57:01.845859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: • pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 3080 computeCapability: 8.6 • coreClock: 1.8GHz coreCount: 68 deviceMemorySize: 10.00GiB deviceMemoryBandwidth: 707.88GiB/s • 2023-05-04 01:57:01.846193: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll • 2023-05-04 01:57:01.846419: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll • 2023-05-04 01:57:01.846580: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll • 2023-05-04 01:57:01.846729: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll • 2023-05-04 01:57:01.846879: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll • 2023-05-04 01:57:01.847007: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll • 2023-05-04 01:57:01.847140: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll • 2023-05-04 01:57:01.847339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 • 2023-05-04 01:57:02.723605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix: • 2023-05-04 01:57:02.723854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0 • 2023-05-04 01:57:02.724007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N • 2023-05-04 01:57:02.724268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4607 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3080, pci bus id: 0000:41:00.0, compute capability: 8.6) • 2023-05-04 01:57:02.740601: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x205a265c200 initialized for platform • CUDA (this does not guarantee that XLA will be used). Devices: • 2023-05-04 01:57:02.740811: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 3080, Compute Capability 8.6 • Init took 2.1324803829193115 sec. • Initializing Network... • Loading model from .\DANNCE\train_results\AVG\weights.9-7.06966.hdf5
I use the dannce_master branch, and all steps follow the instruction, except I pip install tensorflow==2.3.1.
The information of traceback: `Traceback (most recent call last): File "E:\anaconda\envs\tf23\Scripts\dannce-predict-script.py", line 33, in
sys.exit(load_entry_point('dannce', 'console_scripts', 'dannce-predict')())
File "f:\finnaldannce\dannce_master\dannce\cli.py", line 54, in dannce_predict_cli dannce_predict(params)
File "f:\finnaldannce\dannce_master\dannce\interface.py", line 1534, in dannce_predict "centered_euclidean_distance_3D": losses.centered_euclidean_distance_3D,
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\saving\save.py", line 182, in load_model return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\saving\hdf5_format.py", line 178, in load_model_from_hdf5 custom_objects=custom_objects)
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\saving\model_config.py", line 55, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\layers\serialization.py", line 175, in deserialize
printable_module_name='layer')
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 358, in deserialize_keras_object list(custom_objects.items())))
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\functional.py", line 617, in from_config config, custom_objects)
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\functional.py", line 1204, in reconstruct_from_config process_layer(layer_data)
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\functional.py", line 1186, in process_layer layer = deserialize_layer(layer_data, custom_objects=custom_objects)
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\layers\serialization.py", line 175, in deserialize printable_module_name='layer')
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 358, in deserialize_keras_object list(custom_objects.items())))
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\layers\core.py", line 1006, in from_config config, custom_objects, 'function', 'module', 'function_type')
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\layers\core.py", line 1058, in _parse_function_from_config config[func_attr_name], globs=globs)
File "E:\anaconda\envs\tf23\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 457, in func_load code = marshal.loads(raw_code)
ValueError: bad marshal data (unknown type code)`