Closed MMathisLab closed 3 years ago
Heya, what's the status on this? It seems like it's pretty much ready to go
I made a few changes and can run the testscript in tf2.2 (minus tensorpack, which hangs in my ubuntu docker env). Changes: https://github.com/rat-emotion/DeepLabCut-core/commit/f532ebbb7b9c8d2827ad79ba2a565b6d272d52b9
p.s. I'm interested in contributing but haven't touched open source before, if it's useful I'm happy to PR those changes
The save_format='h5'
option mentioned in the short list warnings seems to refer to Model.save and isn't part of tf.train.Saver so I removed it
Hey! I agree it’s very close! If you have edits that make it fly, please do make a PR :). You can comment on what you changed overall, and we will review it. Thanks for contributing! Much appreciated!!!
few remaining issues after PR #8: see--> https://github.com/DeepLabCut/DeepLabCut-core/pull/8#issuecomment-695217681
(1) warnings w/default loader, but it does train w/default and with imgaug on tensorflow 2.2:
WARNING:tensorflow:From /Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From /Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/nnet/losses.py:38: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
Loading ImageNet-pretrained resnet_50
Training parameter:
{'stride': 8.0, 'weigh_part_predictions': False, 'weigh_negatives': False, 'fg_fraction': 0.25, 'weigh_only_present_joints': False, 'mean_pixel': [123.68, 116.779, 103.939], 'shuffle': True, 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-0/TestcoreSep19-trainset80shuffle1/train/snapshot', 'log_dir': 'log', 'global_scale': 0.8, 'location_refinement': True, 'locref_stdev': 7.2801, 'locref_loss_weight': 0.05, 'locref_huber_loss': True, 'optimizer': 'sgd', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'regularize': False, 'weight_decay': 0.0001, 'mirror': False, 'crop_pad': 0, 'scoremap_dir': 'test', 'batch_size': 1, 'dataset_type': 'default', 'deterministic': False, 'crop': True, 'cropratio': 0.4, 'minsize': 100, 'leftwidth': 400, 'rightwidth': 400, 'topheight': 400, 'bottomheight': 400, 'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle1.mat', 'display_iters': 2, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'max_input_size': 1500, 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle1.pickle', 'min_input_size': 64, 'multi_step': [[0.001, 5]], 'net_type': 'resnet_50', 'num_joints': 4, 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'save_iters': 5, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'output_stride': 16, 'deconvolutionstride': 2}
Starting training....
iteration: 2 loss: 1.1634 lr: 0.001
iteration: 4 loss: 0.5873 lr: 0.001
...
Starting with imgaug pose-dataset loader.
Batch Size is 1
Initializing ResNet
Loading ImageNet-pretrained resnet_50
Training parameter:
{'stride': 8.0, 'weigh_part_predictions': False, 'weigh_negatives': False, 'fg_fraction': 0.25, 'weigh_only_present_joints': False, 'mean_pixel': [123.68, 116.779, 103.939], 'shuffle': True, 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-1/TestcoreSep19-trainset80shuffle1/train/snapshot', 'log_dir': 'log', 'global_scale': 0.8, 'location_refinement': True, 'locref_stdev': 7.2801, 'locref_loss_weight': 0.05, 'locref_huber_loss': True, 'optimizer': 'sgd', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'regularize': False, 'weight_decay': 0.0001, 'mirror': False, 'crop_pad': 0, 'scoremap_dir': 'test', 'batch_size': 1, 'dataset_type': 'imgaug', 'deterministic': False, 'crop': True, 'cropratio': 0.4, 'minsize': 100, 'leftwidth': 400, 'rightwidth': 400, 'topheight': 400, 'bottomheight': 400, 'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'dataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle1.mat', 'display_iters': 1, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'max_input_size': 1500, 'metadataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle1.pickle', 'min_input_size': 64, 'multi_step': [[0.001, 5]], 'net_type': 'resnet_50', 'num_joints': 4, 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'save_iters': 5, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'output_stride': 16, 'deconvolutionstride': 2, 'num_outputs': 1, 'Task': None, 'scorer': None, 'date': None, 'video_sets': None, 'bodyparts': None, 'start': None, 'stop': None, 'numframes2pick': None, 'skeleton': [], 'skeleton_color': 'black', 'pcutoff': None, 'dotsize': None, 'alphavalue': None, 'colormap': None, 'TrainingFraction': None, 'iteration': None, 'resnet': None, 'snapshotindex': None, 'cropping': None, 'x1': None, 'x2': None, 'y1': None, 'y2': None, 'corner2move2': None, 'move2corner': None}
Starting training....
iteration: 1 loss: 1.5909 lr: 0.001
iteration: 2 loss: 0.7031 lr: 0.001
iteration: 3 loss: 0.5709 lr: 0.001
iteration: 4 loss: 0.5077 lr: 0.001
iteration: 5 loss: 0.3996 lr: 0.001
...
ALL DONE!!! - default cases without Tensorpack loader are functional.
(2)
ANALYZING some individual frames
Traceback (most recent call last):
File "testscript.py", line 225, in <module>
dlc.analyze_time_lapse_frames(path_config_file,os.path.join(cfg['project_path'],'labeled-data/reachingvideo1/'))
File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/predict_videos.py", line 681, in analyze_time_lapse_frames
TF.reset_default_graph()
AttributeError: module 'tensorflow' has no attribute 'reset_default_graph'
Currently default train, evaluate cases are functional with TensorFlow 2.2 and 2.3 with only the following warnings:
[ ] WARNING:tensorflow:From /Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead.
[ ] WARNING:tensorflow:From /Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers.
[ ] WARNING:tensorflow:From /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/nnet/losses.py:38: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead.
outstanding issue that would need resolved before merging:
export_model
:
Initializing ResNet
WARNING:tensorflow:From /Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
File "testscript_cli.py", line 162, in <module>
dlc.export_model(path_config_file,shuffle=1,make_tar=False)
File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/export.py", line 327, in export_model
tf_to_pb(sess, ckpt, output, output_dir=full_export_dir)
File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/export.py", line 214, in tf_to_pb
initializer_nodes='')
File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/tools/freeze_graph.py", line 361, in freeze_graph
checkpoint_version=checkpoint_version)
File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/tools/freeze_graph.py", line 190, in freeze_graph_with_def_protos
var_list=var_list, write_version=checkpoint_version)
File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 836, in __init__
self.build()
File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 848, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 886, in _build
build_restore=build_restore)
File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 490, in _build_internal
names_to_saveables)
File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 349, in validate_and_slice_inputs
for converted_saveable_object in saveable_objects_for_op(op, name):
File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 210, in saveable_objects_for_op
variable, "", name)
File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 84, in __init__
self.handle_op = var.op.inputs[0]
IndexError: tuple index out of range
Awesome! The export_model issues seems to be related to freeze_graph which a lot of people have been having troubles with in tf2
Relevant: https://leimao.github.io/blog/Save-Load-Inference-From-TF2-Frozen-Graph/
awe, that's too bad! It's a rather core feature of DLC now ... I don't have more time today, but I'll look into it (or please feel free! :) Thanks again!)
This seems to be a great resource: https://github.com/leimao/Frozen_Graph_TensorFlow/tree/master/TensorFlow_v2
@gkane26 do you want to have a look at the export?
I took a quick stab at the model export with tf 2.3 on my macbook -- it took some minimal changes -- the freeze_graph function was throwing an error that I didn't see a way around (see https://github.com/tensorflow/tensorflow/issues/24591). What did work was using the convert_variables_to_constants function (see the question here https://stackoverflow.com/questions/55299995/exporting-a-frozen-graph-pb-file-in-tensorflow-2). In addition to that change, there were some package imports that needed to be changed (tf_slim which seems to be required for mobilenets is now its own package), but otherwise things look good!
@gkane26 - could you make the required changes here; https://github.com/DeepLabCut/DeepLabCut-core/pull/9
Would there be any interest in a tf2 keras model + loader?
I had some trouble extending the current tf.compat.v1 version, so I made a tf2/keras implementation and wrote a function to load weights from pose-tensorflow/deeplabcut models
Hi @altear is there something in PR #9 that does not work for you currently? But happy to discuss another loader.
I started a branch called TF2.2alpha that is a start at migrating to TF2.
I followed the guide here: https://www.tensorflow.org/guide/migrate and am utilizing pip install tf_ slim also see issue https://github.com/DeepLabCut/DeepLabCut/issues/601
Here is the log of the outstanding issues, some of which are resolved, and some need more work. Zero rush. just did this for a bit fo fun.
short list:
report.txt
remaining issues (a like):
(1) testscript is failing due to no video to load into project
[x]
CREATING PROJECT Created "/Users/mwmathis/Documents/DeepLabCut-core_v3/Testcore-Alex-2020-06-24/videos" Created "/Users/mwmathis/Documents/DeepLabCut-core_v3/Testcore-Alex-2020-06-24/labeled-data" Created "/Users/mwmathis/Documents/DeepLabCut-core_v3/Testcore-Alex-2020-06-24/training-datasets" Created "/Users/mwmathis/Documents/DeepLabCut-core_v3/Testcore-Alex-2020-06-24/dlc-models" Copying the videos WARNING: No valid videos were found. The project was not created ... Verify the video files and re-create the project. Traceback (most recent call last): File "testscript.py", line 57, in <module> cfg=dlc.auxiliaryfunctions.read_config(path_config_file) File "/Users/mwmathis/Documents/DeepLabCut-core_v3/deeplabcutcore/utils/auxiliaryfunctions.py", line 132, in read_config "Config file is not found. Please make sure that the file exists and/or that you passed the path of the config file correctly!") FileNotFoundError: Config file is not found. Please make sure that the file exists and/or that you passed the path of the config file correctly!
[x]
dlccore.train_network( ) pose_estimation_tensorflow/nnet/pose_net.py", line 69, in prediction_layers with tf.variable_scope('pose', reuse=reuse): AttributeError: module 'tensorflow' has no attribute 'variable_scope'