search

DeepLabCut / DeepLabCut-core

Headless DeepLabCut (no GUI support)
http://deeplabcut.org
GNU Lesser General Public License v3.0
30 stars 17 forks source link

Quick fixes for tf2.2 #8

Closed MMathisLab closed 3 years ago

MMathisLab commented 3 years ago

https://github.com/DeepLabCut/DeepLabCut-core/issues/6#issuecomment-694903268

MMathisLab commented 3 years ago

getting closer :)

TL;DR

The remaining issues w/time_lapse_image function only:

Firstly, I ran testscript.py that is in the tf2.2alpha branch (on the Reaching-Mackenzie .. data folder from the main DLC repo):

TRACEBACK

```python Successfully installed tensorboard-2.2.2 tensorflow-2.2.0 tensorflow-estimator-2.2.0 (DLC-CPU) mwmathis@mackenzies-macbook-pro DeepLabCut-core % pythonw testscript.py Imported DLC! ['/Users/mwmathis/Documents/DLCreleases/DeepLabCut/examples/Reaching-Mackenzie-2018-08-30/videos/reachingvideo1.avi'] On Windows/OSX tensorpack is not tested by default. CREATING PROJECT Created "/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos" Created "/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/labeled-data" Created "/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/training-datasets" Created "/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models" Copying the videos WARNING: No valid videos were found. The project was not created ... Verify the video files and re-create the project. Traceback (most recent call last): File "testscript.py", line 57, in cfg=dlc.auxiliaryfunctions.read_config(path_config_file) File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/utils/auxiliaryfunctions.py", line 132, in read_config "Config file is not found. Please make sure that the file exists and/or that you passed the path of the config file correctly!") FileNotFoundError: Config file is not found. Please make sure that the file exists and/or that you passed the path of the config file correctly! (DLC-CPU) mwmathis@mackenzies-macbook-pro DeepLabCut-core % pythonw testscript.py Imported DLC! ['/Users/mwmathis/Documents/DeepLabCut-core/Reaching-Mackenzie-2018-08-30/videos/reachingvideo1.avi'] On Windows/OSX tensorpack is not tested by default. CREATING PROJECT Created "/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos" Created "/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/labeled-data" Created "/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/training-datasets" Created "/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models" Copying the videos /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1.avi Generated "/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/config.yaml" A new project with name Testcore-Alex-2020-09-19 is created at /Users/mwmathis/Documents/DeepLabCut-core and a configurable file (config.yaml) is stored there. Change the parameters in this file to adapt to your project's needs. Once you have changed the configuration file, use the function 'extract_frames' to select frames for labeling. . [OPTIONAL] Use the function 'add_new_videos' to add new videos to your project (at any stage). EXTRACTING FRAMES Config file read successfully. Extracting frames based on kmeans ... Kmeans-quantization based extracting of frames from 0.0 seconds to 8.53 seconds. Extracting and downsampling... 256 frames from the video. 256it [00:00, 274.48it/s] Kmeans clustering ... (this might take a while) Frames were successfully extracted. You can now label the frames using the function 'label_frames' (if you extracted enough frames for all videos). CREATING-SOME LABELS FOR THE FRAMES Plot labels... Creating images with labels by Alex. They are stored in the following folder: /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/labeled-data/reachingvideo1_labeled. If all the labels are ok, then use the function 'create_training_dataset' to create the training dataset! CREATING TRAININGSET Downloading a ImageNet-pretrained model from http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz.... The training dataset is successfully created. Use the function 'train_network' to start training. Happy training! CHANGING training parameters to end quickly! TRAIN Config: {'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'batch_size': 1, 'bottomheight': 400, 'crop': True, 'crop_pad': 0, 'cropratio': 0.4, 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle1.mat', 'dataset_type': 'default', 'deterministic': False, 'display_iters': 2, 'fg_fraction': 0.25, 'global_scale': 0.8, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'leftwidth': 400, 'location_refinement': True, 'locref_huber_loss': True, 'locref_loss_weight': 0.05, 'locref_stdev': 7.2801, 'log_dir': 'log', 'max_input_size': 1500, 'mean_pixel': [123.68, 116.779, 103.939], 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle1.pickle', 'min_input_size': 64, 'minsize': 100, 'mirror': False, 'multi_step': [[0.001, 5]], 'net_type': 'resnet_50', 'num_joints': 4, 'optimizer': 'sgd', 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'regularize': False, 'rightwidth': 400, 'save_iters': 5, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'scoremap_dir': 'test', 'shuffle': True, 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-0/TestcoreSep19-trainset80shuffle1/train/snapshot', 'stride': 8.0, 'topheight': 400, 'weigh_negatives': False, 'weigh_only_present_joints': False, 'weigh_part_predictions': False, 'weight_decay': 0.0001} Switching batchsize to 1, as default/tensorpack/deterministic loaders do not support batches >1. Use imgaug loader. Starting with standard pose-dataset loader. Initializing ResNet WARNING:tensorflow:From /Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. WARNING:tensorflow:From /Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers. WARNING:tensorflow:From /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/nnet/losses.py:38: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Loading ImageNet-pretrained resnet_50 Training parameter: {'stride': 8.0, 'weigh_part_predictions': False, 'weigh_negatives': False, 'fg_fraction': 0.25, 'weigh_only_present_joints': False, 'mean_pixel': [123.68, 116.779, 103.939], 'shuffle': True, 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-0/TestcoreSep19-trainset80shuffle1/train/snapshot', 'log_dir': 'log', 'global_scale': 0.8, 'location_refinement': True, 'locref_stdev': 7.2801, 'locref_loss_weight': 0.05, 'locref_huber_loss': True, 'optimizer': 'sgd', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'regularize': False, 'weight_decay': 0.0001, 'mirror': False, 'crop_pad': 0, 'scoremap_dir': 'test', 'batch_size': 1, 'dataset_type': 'default', 'deterministic': False, 'crop': True, 'cropratio': 0.4, 'minsize': 100, 'leftwidth': 400, 'rightwidth': 400, 'topheight': 400, 'bottomheight': 400, 'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle1.mat', 'display_iters': 2, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'max_input_size': 1500, 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle1.pickle', 'min_input_size': 64, 'multi_step': [[0.001, 5]], 'net_type': 'resnet_50', 'num_joints': 4, 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'save_iters': 5, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'output_stride': 16, 'deconvolutionstride': 2} Starting training.... iteration: 2 loss: 1.1634 lr: 0.001 iteration: 4 loss: 0.5873 lr: 0.001 Exception in thread Thread-2: Traceback (most recent call last): File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call return fn(*args) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled [[{{node fifo_queue_enqueue}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py", line 83, in load_and_enqueue sess.run(enqueue_op, feed_dict=food) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 958, in run run_metadata_ptr) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1181, in _run feed_dict_tensor, options, run_metadata) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run run_metadata) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled [[node fifo_queue_enqueue (defined at /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py:69) ]] Errors may have originated from an input operation. Input Source operations connected to node fifo_queue_enqueue: Placeholder_3 (defined at /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py:59) fifo_queue (defined at /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py:68) Original stack trace for 'fifo_queue_enqueue': File "testscript.py", line 105, in dlc.train_network(path_config_file) File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/training.py", line 132, in train_network train(str(poseconfigfile),displayiters,saveiters,maxiters,max_to_keep=max_snapshots_to_keep,keepdeconvweights=keepdeconvweights,allow_growth=allow_growth) #pass on path and file name for pose_cfg.yaml! File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py", line 120, in train batch, enqueue_op, placeholders = setup_preloading(batch_spec) File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py", line 69, in setup_preloading enqueue_op = q.enqueue(placeholders_list) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 345, in enqueue self._queue_ref, vals, name=scope) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 4028, in queue_enqueue_v2 timeout_ms=timeout_ms, name=name) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper attrs=attr_protos, op_def=op_def) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3327, in _create_op_internal op_def=op_def) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1791, in __init__ self._traceback = tf_stack.extract_stack() The network is now trained and ready to evaluate. Use the function 'evaluate_network' to evaluate the network. EVALUATE Config: {'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'batch_size': 1, 'bottomheight': 400, 'crop': True, 'crop_pad': 0, 'cropratio': 0.4, 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle1.mat', 'dataset_type': 'default', 'deconvolutionstride': 2, 'deterministic': False, 'display_iters': 2, 'fg_fraction': 0.25, 'global_scale': 0.8, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'leftwidth': 400, 'location_refinement': True, 'locref_huber_loss': True, 'locref_loss_weight': 0.05, 'locref_stdev': 7.2801, 'log_dir': 'log', 'max_input_size': 1500, 'mean_pixel': [123.68, 116.779, 103.939], 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle1.pickle', 'min_input_size': 64, 'minsize': 100, 'mirror': False, 'multi_step': [[0.001, 5]], 'net_type': 'resnet_50', 'num_joints': 4, 'optimizer': 'sgd', 'output_stride': 16, 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'regularize': False, 'rightwidth': 400, 'save_iters': 5, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'scoremap_dir': 'test', 'shuffle': True, 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-0/TestcoreSep19-trainset80shuffle1/test/snapshot', 'stride': 8.0, 'topheight': 400, 'weigh_negatives': False, 'weigh_only_present_joints': False, 'weigh_part_predictions': False, 'weight_decay': 0.0001} Running DLC_resnet50_TestcoreSep19shuffle1_5 with # of trainingiterations: 5 Initializing ResNet Analyzing data... 5it [00:03, 1.30it/s] Done and results stored for snapshot: snapshot-5 Results for 5 training iterations: 80 1 train error: 450.15 pixels. Test error: 516.71 pixels. With pcutoff of 0.01 train error: 450.15 pixels. Test error: 516.71 pixels Thereby, the errors are given by the average distances between the labels by DLC and the scorer. Plotting... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:02<00:00, 1.81it/s] The network is evaluated and the results are stored in the subdirectory 'evaluation_results'. If it generalizes well, choose the best model for prediction and update the config file with the appropriate index for the 'snapshotindex'. Use the function 'analyze_video' to make predictions on new videos. Otherwise consider retraining the network (see DeepLabCut workflow Fig 2) CUT SHORT VIDEO AND ANALYZE (with dynamic cropping!) Slicing and saving to name /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers built with clang version 4.0.1 (tags/RELEASE_401/final) configuration: --prefix=/Users/mwmathis/opt/anaconda3/envs/DLC-CPU --cc=x86_64-apple-darwin13.4.0-clang --disable-doc --enable-gpl --enable-avresample --disable-gnutls --enable-hardcoded-tables --enable-libfreetype --enable-openssl --enable-libvpx --enable-pthreads --enable-libopus --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame --enable-nonfree --enable-libopenh264 --enable-libx264 libavutil 56. 31.100 / 56. 31.100 libavcodec 58. 54.100 / 58. 54.100 libavformat 58. 29.100 / 58. 29.100 libavdevice 58. 8.100 / 58. 8.100 libavfilter 7. 57.100 / 7. 57.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 5.100 / 5. 5.100 libswresample 3. 5.100 / 3. 5.100 libpostproc 55. 5.100 / 55. 5.100 Input #0, avi, from '/Users/mwmathis/Documents/DeepLabCut-core/Reaching-Mackenzie-2018-08-30/videos/reachingvideo1.avi': Duration: 00:00:08.53, start: 0.000000, bitrate: 12642 kb/s Stream #0:0: Video: mjpeg (Baseline) (MJPG / 0x47504A4D), yuvj420p(pc, bt470bg/unknown/unknown), 832x747 [SAR 1:1 DAR 832:747], 12682 kb/s, 30 fps, 30 tbr, 30 tbn, 30 tbc Metadata: title : ImageJ AVI Stream mapping: Stream #0:0 -> #0:0 (mjpeg (native) -> mpeg4 (native)) Press [q] to stop, [?] for help [swscaler @ 0x7f971db5b000] deprecated pixel format used, make sure you did set range correctly Output #0, avi, to '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi': Metadata: ISFT : Lavf58.29.100 Stream #0:0: Video: mpeg4 (FMP4 / 0x34504D46), yuv420p, 832x747 [SAR 1:1 DAR 832:747], q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc Metadata: title : ImageJ AVI encoder : Lavc58.54.100 mpeg4 Side data: cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1 frame= 12 fps=0.0 q=24.4 Lsize= 199kB time=00:00:00.40 bitrate=4077.2kbits/s speed=7.28x video:193kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 3.075196% Config: {'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'batch_size': 1, 'bottomheight': 400, 'crop': True, 'crop_pad': 0, 'cropratio': 0.4, 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle1.mat', 'dataset_type': 'default', 'deconvolutionstride': 2, 'deterministic': False, 'display_iters': 2, 'fg_fraction': 0.25, 'global_scale': 0.8, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'leftwidth': 400, 'location_refinement': True, 'locref_huber_loss': True, 'locref_loss_weight': 0.05, 'locref_stdev': 7.2801, 'log_dir': 'log', 'max_input_size': 1500, 'mean_pixel': [123.68, 116.779, 103.939], 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle1.pickle', 'min_input_size': 64, 'minsize': 100, 'mirror': False, 'multi_step': [[0.001, 5]], 'net_type': 'resnet_50', 'num_joints': 4, 'optimizer': 'sgd', 'output_stride': 16, 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'regularize': False, 'rightwidth': 400, 'save_iters': 5, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'scoremap_dir': 'test', 'shuffle': True, 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-0/TestcoreSep19-trainset80shuffle1/test/snapshot', 'stride': 8.0, 'topheight': 400, 'weigh_negatives': False, 'weigh_only_present_joints': False, 'weigh_part_predictions': False, 'weight_decay': 0.0001} Using snapshot-5 for model /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-0/TestcoreSep19-trainset80shuffle1 Starting analysis in dynamic cropping mode with parameters: (True, 0.1, 5) Switching batchsize to 1, num_outputs (per animal) to 1 and TFGPUinference to False (all these features are not supported in this mode). Initializing ResNet Starting to analyze % /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi Loading /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi Duration of video [s]: 0.4 , recorded with 30.0 fps! Overall # of frames: 12 found with (before cropping) frame dimensions: 832 747 Starting to extract posture 20it [00:01, 12.96it/s] Saving results in /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos... Saving csv poses! The videos are analyzed. Now your research can truly start! You can create labeled videos with 'create_labeled_video'. If the tracking is not satisfactory for some videos, consider expanding the training set. You can use the function 'extract_outlier_frames' to extract any outlier frames! analyze again... Config: {'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'batch_size': 1, 'bottomheight': 400, 'crop': True, 'crop_pad': 0, 'cropratio': 0.4, 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle1.mat', 'dataset_type': 'default', 'deconvolutionstride': 2, 'deterministic': False, 'display_iters': 2, 'fg_fraction': 0.25, 'global_scale': 0.8, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'leftwidth': 400, 'location_refinement': True, 'locref_huber_loss': True, 'locref_loss_weight': 0.05, 'locref_stdev': 7.2801, 'log_dir': 'log', 'max_input_size': 1500, 'mean_pixel': [123.68, 116.779, 103.939], 'metadataset': 'training-datasets/iteration-0/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle1.pickle', 'min_input_size': 64, 'minsize': 100, 'mirror': False, 'multi_step': [[0.001, 5]], 'net_type': 'resnet_50', 'num_joints': 4, 'num_outputs': 1, 'optimizer': 'sgd', 'output_stride': 16, 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'regularize': False, 'rightwidth': 400, 'save_iters': 5, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'scoremap_dir': 'test', 'shuffle': True, 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-0/TestcoreSep19-trainset80shuffle1/test/snapshot', 'stride': 8.0, 'topheight': 400, 'weigh_negatives': False, 'weigh_only_present_joints': False, 'weigh_part_predictions': False, 'weight_decay': 0.0001} Using snapshot-5 for model /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-0/TestcoreSep19-trainset80shuffle1 Initializing ResNet Starting to analyze % /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi Video already analyzed! /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1shortDLC_resnet50_TestcoreSep19shuffle1_5.h5 The videos are analyzed. Now your research can truly start! You can create labeled videos with 'create_labeled_video'. If the tracking is not satisfactory for some videos, consider expanding the training set. You can use the function 'extract_outlier_frames' to extract any outlier frames! CREATE VIDEO Starting % /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos ['/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi'] Loading /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi and data. 12 Duration of video [s]: 0.4 , recorded with 30.0 fps! Overall # of frames: 12 with cropped frame dimensions: 832 747 Generating frames and creating video. 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:04<00:00, 2.56it/s] All labeled frames were created, now generating video... ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers built with clang version 4.0.1 (tags/RELEASE_401/final) configuration: --prefix=/Users/mwmathis/opt/anaconda3/envs/DLC-CPU --cc=x86_64-apple-darwin13.4.0-clang --disable-doc --enable-gpl --enable-avresample --disable-gnutls --enable-hardcoded-tables --enable-libfreetype --enable-openssl --enable-libvpx --enable-pthreads --enable-libopus --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame --enable-nonfree --enable-libopenh264 --enable-libx264 libavutil 56. 31.100 / 56. 31.100 libavcodec 58. 54.100 / 58. 54.100 libavformat 58. 29.100 / 58. 29.100 libavdevice 58. 8.100 / 58. 8.100 libavfilter 7. 57.100 / 7. 57.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 5.100 / 5. 5.100 libswresample 3. 5.100 / 3. 5.100 libpostproc 55. 5.100 / 55. 5.100 Input #0, image2, from 'file%02d.png': Duration: 00:00:00.40, start: 0.000000, bitrate: N/A Stream #0:0: Video: png, rgba(pc), 832x747 [SAR 3937:3937 DAR 832:747], 30 fps, 30 tbr, 30 tbn, 30 tbc Stream mapping: Stream #0:0 -> #0:0 (png (native) -> h264 (libx264)) Press [q] to stop, [?] for help [libx264 @ 0x7f9b9200c200] using SAR=1/1 [libx264 @ 0x7f9b9200c200] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 [libx264 @ 0x7f9b9200c200] profile High 4:4:4 Predictive, level 3.1, 4:4:4 8-bit [libx264 @ 0x7f9b9200c200] 264 - core 152 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x1:0x111 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00 Output #0, mp4, to '../reachingvideo1shortDLC_resnet50_TestcoreSep19shuffle1_5_labeled.mp4': Metadata: encoder : Lavf58.29.100 Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv444p, 832x747 [SAR 1:1 DAR 832:747], q=-1--1, 30 fps, 15360 tbn, 30 tbc Metadata: encoder : Lavc58.54.100 libx264 Side data: cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1 frame= 12 fps=0.0 q=-1.0 Lsize= 76kB time=00:00:00.30 bitrate=2069.2kbits/s speed=1.59x video:75kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.216761% [libx264 @ 0x7f9b9200c200] frame I:1 Avg QP:21.51 size: 18783 [libx264 @ 0x7f9b9200c200] frame P:9 Avg QP:24.21 size: 5624 [libx264 @ 0x7f9b9200c200] frame B:2 Avg QP:26.40 size: 3302 [libx264 @ 0x7f9b9200c200] consecutive B-frames: 75.0% 0.0% 25.0% 0.0% [libx264 @ 0x7f9b9200c200] mb I I16..4: 66.1% 0.0% 33.9% [libx264 @ 0x7f9b9200c200] mb P I16..4: 7.1% 0.0% 4.0% P16..4: 27.4% 5.4% 2.7% 0.0% 0.0% skip:53.4% [libx264 @ 0x7f9b9200c200] mb B I16..4: 0.3% 0.0% 1.2% B16..8: 31.7% 3.0% 0.9% direct: 2.8% skip:60.1% L0:34.5% L1:62.1% BI: 3.3% [libx264 @ 0x7f9b9200c200] coded y,u,v intra: 32.5% 0.9% 0.8% inter: 8.7% 0.1% 0.1% [libx264 @ 0x7f9b9200c200] i16 v,h,dc,p: 53% 17% 15% 15% [libx264 @ 0x7f9b9200c200] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 30% 16% 23% 4% 6% 8% 5% 4% 3% [libx264 @ 0x7f9b9200c200] Weighted P-Frames: Y:11.1% UV:11.1% [libx264 @ 0x7f9b9200c200] ref P L0: 75.2% 10.2% 11.3% 3.2% [libx264 @ 0x7f9b9200c200] ref B L0: 99.3% 0.7% [libx264 @ 0x7f9b9200c200] kb/s:1520.08 Making plots /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi Starting % /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi Loading /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi and data. /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos already exists! Plots created! Please check the directory "plot-poses" within the video directory EXTRACT OUTLIERS Method jump found 11 putative outlier frames. Do you want to proceed with extracting 5 of those? If this list is very large, perhaps consider changing the paramters (start, stop, epsilon, comparisonbodyparts) or use a different method. Loading video... Duration of video [s]: 0.4 , recorded @ 30.0 fps! Overall # of frames: 12 with (cropped) frame dimensions: Kmeans-quantization based extracting of frames from 0.0 seconds to 0.4 seconds. Extracting and downsampling... 11 frames from the video. 11it [00:00, 276.86it/s] Kmeans clustering ... (this might take a while) Let's select frames indices: [7, 1, 6, 10, 3] New video was added to the project! Use the function 'extract_frames' to select frames for labeling. The outlier frames are extracted. They are stored in the subdirectory labeled-data\reachingvideo1short. Once you extracted frames for all videos, use 'refine_labels' to manually correct the labels. Method Fitting found 0 putative outlier frames. Do you want to proceed with extracting 5 of those? Frames from video reachingvideo1short already extracted (more will be added)! Loading video... Duration of video [s]: 0.4 , recorded @ 30.0 fps! Overall # of frames: 12 with (cropped) frame dimensions: Kmeans-quantization based extracting of frames from 0.0 seconds to 0.4 seconds. Let's select frames indices: [] No frames were extracted. RELABELING MERGING Merged data sets and updated refinement iteration to 1. Now you can create a new training set for the expanded annotated images (use create_training_dataset). CREATING TRAININGSET The training dataset is successfully created. Use the function 'train_network' to start training. Happy training! CHANGING training parameters to end quickly! TRAIN Config: {'Task': None, 'TrainingFraction': None, 'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'alphavalue': None, 'batch_size': 1, 'bodyparts': None, 'bottomheight': 400, 'colormap': None, 'corner2move2': None, 'crop': True, 'crop_pad': 0, 'cropping': None, 'cropratio': 0.4, 'dataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle1.mat', 'dataset_type': 'imgaug', 'date': None, 'deconvolutionstride': 2, 'deterministic': False, 'display_iters': 1, 'dotsize': None, 'fg_fraction': 0.25, 'global_scale': 0.8, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'iteration': None, 'leftwidth': 400, 'location_refinement': True, 'locref_huber_loss': True, 'locref_loss_weight': 0.05, 'locref_stdev': 7.2801, 'log_dir': 'log', 'max_input_size': 1500, 'mean_pixel': [123.68, 116.779, 103.939], 'metadataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle1.pickle', 'min_input_size': 64, 'minsize': 100, 'mirror': False, 'move2corner': None, 'multi_step': [[0.001, 5]], 'net_type': 'resnet_50', 'num_joints': 4, 'num_outputs': 1, 'numframes2pick': None, 'optimizer': 'sgd', 'output_stride': 16, 'pcutoff': None, 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'regularize': False, 'resnet': None, 'rightwidth': 400, 'save_iters': 5, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'scoremap_dir': 'test', 'scorer': None, 'shuffle': True, 'skeleton': [], 'skeleton_color': 'black', 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-1/TestcoreSep19-trainset80shuffle1/train/snapshot', 'snapshotindex': None, 'start': None, 'stop': None, 'stride': 8.0, 'topheight': 400, 'video_sets': None, 'weigh_negatives': False, 'weigh_only_present_joints': False, 'weigh_part_predictions': False, 'weight_decay': 0.0001, 'x1': None, 'x2': None, 'y1': None, 'y2': None} Starting with imgaug pose-dataset loader. Batch Size is 1 Initializing ResNet Loading ImageNet-pretrained resnet_50 Training parameter: {'stride': 8.0, 'weigh_part_predictions': False, 'weigh_negatives': False, 'fg_fraction': 0.25, 'weigh_only_present_joints': False, 'mean_pixel': [123.68, 116.779, 103.939], 'shuffle': True, 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-1/TestcoreSep19-trainset80shuffle1/train/snapshot', 'log_dir': 'log', 'global_scale': 0.8, 'location_refinement': True, 'locref_stdev': 7.2801, 'locref_loss_weight': 0.05, 'locref_huber_loss': True, 'optimizer': 'sgd', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'regularize': False, 'weight_decay': 0.0001, 'mirror': False, 'crop_pad': 0, 'scoremap_dir': 'test', 'batch_size': 1, 'dataset_type': 'imgaug', 'deterministic': False, 'crop': True, 'cropratio': 0.4, 'minsize': 100, 'leftwidth': 400, 'rightwidth': 400, 'topheight': 400, 'bottomheight': 400, 'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'dataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle1.mat', 'display_iters': 1, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'max_input_size': 1500, 'metadataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle1.pickle', 'min_input_size': 64, 'multi_step': [[0.001, 5]], 'net_type': 'resnet_50', 'num_joints': 4, 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'save_iters': 5, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'output_stride': 16, 'deconvolutionstride': 2, 'num_outputs': 1, 'Task': None, 'scorer': None, 'date': None, 'video_sets': None, 'bodyparts': None, 'start': None, 'stop': None, 'numframes2pick': None, 'skeleton': [], 'skeleton_color': 'black', 'pcutoff': None, 'dotsize': None, 'alphavalue': None, 'colormap': None, 'TrainingFraction': None, 'iteration': None, 'resnet': None, 'snapshotindex': None, 'cropping': None, 'x1': None, 'x2': None, 'y1': None, 'y2': None, 'corner2move2': None, 'move2corner': None} Starting training.... iteration: 1 loss: 1.5909 lr: 0.001 iteration: 2 loss: 0.7031 lr: 0.001 iteration: 3 loss: 0.5709 lr: 0.001 iteration: 4 loss: 0.5077 lr: 0.001 iteration: 5 loss: 0.3996 lr: 0.001 Exception in thread Thread-8: Traceback (most recent call last): File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call return fn(*args) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled [[{{node fifo_queue_enqueue}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py", line 83, in load_and_enqueue sess.run(enqueue_op, feed_dict=food) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 958, in run run_metadata_ptr) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1181, in _run feed_dict_tensor, options, run_metadata) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run run_metadata) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled [[node fifo_queue_enqueue (defined at /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py:69) ]] Errors may have originated from an input operation. Input Source operations connected to node fifo_queue_enqueue: fifo_queue (defined at /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py:68) Placeholder (defined at /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py:59) Original stack trace for 'fifo_queue_enqueue': File "testscript.py", line 173, in dlc.train_network(path_config_file) File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/training.py", line 132, in train_network train(str(poseconfigfile),displayiters,saveiters,maxiters,max_to_keep=max_snapshots_to_keep,keepdeconvweights=keepdeconvweights,allow_growth=allow_growth) #pass on path and file name for pose_cfg.yaml! File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py", line 120, in train batch, enqueue_op, placeholders = setup_preloading(batch_spec) File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py", line 69, in setup_preloading enqueue_op = q.enqueue(placeholders_list) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 345, in enqueue self._queue_ref, vals, name=scope) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 4028, in queue_enqueue_v2 timeout_ms=timeout_ms, name=name) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper attrs=attr_protos, op_def=op_def) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3327, in _create_op_internal op_def=op_def) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1791, in __init__ self._traceback = tf_stack.extract_stack() The network is now trained and ready to evaluate. Use the function 'evaluate_network' to evaluate the network. Slicing and saving to name /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short2.avi ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers built with clang version 4.0.1 (tags/RELEASE_401/final) configuration: --prefix=/Users/mwmathis/opt/anaconda3/envs/DLC-CPU --cc=x86_64-apple-darwin13.4.0-clang --disable-doc --enable-gpl --enable-avresample --disable-gnutls --enable-hardcoded-tables --enable-libfreetype --enable-openssl --enable-libvpx --enable-pthreads --enable-libopus --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame --enable-nonfree --enable-libopenh264 --enable-libx264 libavutil 56. 31.100 / 56. 31.100 libavcodec 58. 54.100 / 58. 54.100 libavformat 58. 29.100 / 58. 29.100 libavdevice 58. 8.100 / 58. 8.100 libavfilter 7. 57.100 / 7. 57.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 5.100 / 5. 5.100 libswresample 3. 5.100 / 3. 5.100 libpostproc 55. 5.100 / 55. 5.100 Input #0, avi, from '/Users/mwmathis/Documents/DeepLabCut-core/Reaching-Mackenzie-2018-08-30/videos/reachingvideo1.avi': Duration: 00:00:08.53, start: 0.000000, bitrate: 12642 kb/s Stream #0:0: Video: mjpeg (Baseline) (MJPG / 0x47504A4D), yuvj420p(pc, bt470bg/unknown/unknown), 832x747 [SAR 1:1 DAR 832:747], 12682 kb/s, 30 fps, 30 tbr, 30 tbn, 30 tbc Metadata: title : ImageJ AVI Stream mapping: Stream #0:0 -> #0:0 (mjpeg (native) -> mpeg4 (native)) Press [q] to stop, [?] for help [swscaler @ 0x7fefd5e5b000] deprecated pixel format used, make sure you did set range correctly Output #0, avi, to '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short2.avi': Metadata: ISFT : Lavf58.29.100 Stream #0:0: Video: mpeg4 (FMP4 / 0x34504D46), yuv420p, 832x747 [SAR 1:1 DAR 832:747], q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc Metadata: title : ImageJ AVI encoder : Lavc58.54.100 mpeg4 Side data: cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1 frame= 12 fps=0.0 q=24.4 Lsize= 199kB time=00:00:00.40 bitrate=4077.2kbits/s speed=4.46x video:193kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 3.075196% Inference with direct cropping Overwriting cropping parameters: [0, 50, 0, 50] These are used for all videos, but won't be save to the cfg file. Config: {'Task': None, 'TrainingFraction': None, 'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'alphavalue': None, 'batch_size': 1, 'bodyparts': None, 'bottomheight': 400, 'colormap': None, 'corner2move2': None, 'crop': True, 'crop_pad': 0, 'cropping': None, 'cropratio': 0.4, 'dataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle1.mat', 'dataset_type': 'imgaug', 'date': None, 'deconvolutionstride': 2, 'deterministic': False, 'display_iters': 1, 'dotsize': None, 'fg_fraction': 0.25, 'global_scale': 0.8, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'iteration': None, 'leftwidth': 400, 'location_refinement': True, 'locref_huber_loss': True, 'locref_loss_weight': 0.05, 'locref_stdev': 7.2801, 'log_dir': 'log', 'max_input_size': 1500, 'mean_pixel': [123.68, 116.779, 103.939], 'metadataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle1.pickle', 'min_input_size': 64, 'minsize': 100, 'mirror': False, 'move2corner': None, 'multi_step': [[0.001, 5]], 'net_type': 'resnet_50', 'num_joints': 4, 'num_outputs': 1, 'numframes2pick': None, 'optimizer': 'sgd', 'output_stride': 16, 'pcutoff': None, 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'regularize': False, 'resnet': None, 'rightwidth': 400, 'save_iters': 5, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'scoremap_dir': 'test', 'scorer': None, 'shuffle': True, 'skeleton': [], 'skeleton_color': 'black', 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-1/TestcoreSep19-trainset80shuffle1/test/snapshot', 'snapshotindex': None, 'start': None, 'stop': None, 'stride': 8.0, 'topheight': 400, 'video_sets': None, 'weigh_negatives': False, 'weigh_only_present_joints': False, 'weigh_part_predictions': False, 'weight_decay': 0.0001, 'x1': None, 'x2': None, 'y1': None, 'y2': None} Using snapshot-5 for model /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-1/TestcoreSep19-trainset80shuffle1 Initializing ResNet Starting to analyze % /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short2.avi Loading /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short2.avi Duration of video [s]: 0.4 , recorded with 30.0 fps! Overall # of frames: 12 found with (before cropping) frame dimensions: 832 747 Starting to extract posture Cropping based on the x1 = 0 x2 = 50 y1 = 0 y2 = 50. You can adjust the cropping coordinates in the config.yaml file. 20it [00:00, 42.36it/s] Detected frames: 12 20it [00:00, 37.92it/s] Saving results in /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos... Saving csv poses! The videos are analyzed. Now your research can truly start! You can create labeled videos with 'create_labeled_video'. If the tracking is not satisfactory for some videos, consider expanding the training set. You can use the function 'extract_outlier_frames' to extract any outlier frames! Extracting skeleton distances, filter and plot filtered output Processing /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi Filtering with median model /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short.avi 4it [00:00, 121.05it/s] Saving filtered csv poses! Starting % /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos ['/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short2.avi'] Loading /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short2.avi and data. No filtered predictions found, using frame-by-frame output instead. 12 Duration of video [s]: 0.4 , recorded with 30.0 fps! Overall # of frames: 12 with cropped frame dimensions: 50 50 Generating frames and creating video. 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 230.05it/s] /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short2.avi Starting % /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short2.avi Loading /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/reachingvideo1short2.avi and data. No filtered predictions found, using frame-by-frame output instead. /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos already exists! /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/videos/plot-poses already exists! Plots created! Please check the directory "plot-poses" within the video directory ALL DONE!!! - default cases without Tensorpack loader are functional. CREATING TRAININGSET for shuffle 2 will be used for 3D testscript... /Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19 already exists! The training dataset is successfully created. Use the function 'train_network' to start training. Happy training! CHANGING training parameters to end quickly! TRAINING shuffle 2, with smaller allocated memory Config: {'Task': None, 'TrainingFraction': None, 'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'alphavalue': None, 'batch_size': 1, 'bodyparts': None, 'bottomheight': 400, 'colormap': None, 'corner2move2': None, 'crop': True, 'crop_pad': 0, 'cropping': None, 'cropratio': 0.4, 'dataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle2.mat', 'dataset_type': 'imgaug', 'date': None, 'deconvolutionstride': 2, 'deterministic': False, 'display_iters': 2, 'dotsize': None, 'fg_fraction': 0.25, 'global_scale': 0.8, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'iteration': None, 'leftwidth': 400, 'location_refinement': True, 'locref_huber_loss': True, 'locref_loss_weight': 0.05, 'locref_stdev': 7.2801, 'log_dir': 'log', 'max_input_size': 1500, 'mean_pixel': [123.68, 116.779, 103.939], 'metadataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle2.pickle', 'min_input_size': 64, 'minsize': 100, 'mirror': False, 'move2corner': None, 'multi_step': [[0.001, 10]], 'net_type': 'resnet_50', 'num_joints': 4, 'num_outputs': 1, 'numframes2pick': None, 'optimizer': 'sgd', 'output_stride': 16, 'pcutoff': None, 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'regularize': False, 'resnet': None, 'rightwidth': 400, 'save_iters': 10, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'scoremap_dir': 'test', 'scorer': None, 'shuffle': True, 'skeleton': [], 'skeleton_color': 'black', 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-1/TestcoreSep19-trainset80shuffle2/train/snapshot', 'snapshotindex': None, 'start': None, 'stop': None, 'stride': 8.0, 'topheight': 400, 'video_sets': None, 'weigh_negatives': False, 'weigh_only_present_joints': False, 'weigh_part_predictions': False, 'weight_decay': 0.0001, 'x1': None, 'x2': None, 'y1': None, 'y2': None} Starting with imgaug pose-dataset loader. Batch Size is 1 Initializing ResNet Loading ImageNet-pretrained resnet_50 Training parameter: {'stride': 8.0, 'weigh_part_predictions': False, 'weigh_negatives': False, 'fg_fraction': 0.25, 'weigh_only_present_joints': False, 'mean_pixel': [123.68, 116.779, 103.939], 'shuffle': True, 'snapshot_prefix': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19/dlc-models/iteration-1/TestcoreSep19-trainset80shuffle2/train/snapshot', 'log_dir': 'log', 'global_scale': 0.8, 'location_refinement': True, 'locref_stdev': 7.2801, 'locref_loss_weight': 0.05, 'locref_huber_loss': True, 'optimizer': 'sgd', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'regularize': False, 'weight_decay': 0.0001, 'mirror': False, 'crop_pad': 0, 'scoremap_dir': 'test', 'batch_size': 1, 'dataset_type': 'imgaug', 'deterministic': False, 'crop': True, 'cropratio': 0.4, 'minsize': 100, 'leftwidth': 400, 'rightwidth': 400, 'topheight': 400, 'bottomheight': 400, 'all_joints': [[0], [1], [2], [3]], 'all_joints_names': ['bodypart1', 'bodypart2', 'bodypart3', 'objectA'], 'dataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Testcore_Alex80shuffle2.mat', 'display_iters': 2, 'init_weights': '/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt', 'max_input_size': 1500, 'metadataset': 'training-datasets/iteration-1/UnaugmentedDataSet_TestcoreSep19/Documentation_data-Testcore_80shuffle2.pickle', 'min_input_size': 64, 'multi_step': [[0.001, 10]], 'net_type': 'resnet_50', 'num_joints': 4, 'pos_dist_thresh': 17, 'project_path': '/Users/mwmathis/Documents/DeepLabCut-core/Testcore-Alex-2020-09-19', 'save_iters': 10, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'output_stride': 16, 'deconvolutionstride': 2, 'num_outputs': 1, 'Task': None, 'scorer': None, 'date': None, 'video_sets': None, 'bodyparts': None, 'start': None, 'stop': None, 'numframes2pick': None, 'skeleton': [], 'skeleton_color': 'black', 'pcutoff': None, 'dotsize': None, 'alphavalue': None, 'colormap': None, 'TrainingFraction': None, 'iteration': None, 'resnet': None, 'snapshotindex': None, 'cropping': None, 'x1': None, 'x2': None, 'y1': None, 'y2': None, 'corner2move2': None, 'move2corner': None} Starting training.... iteration: 2 loss: 1.2354 lr: 0.001 iteration: 4 loss: 0.5974 lr: 0.001 iteration: 6 loss: 0.4802 lr: 0.001 iteration: 8 loss: 0.3143 lr: 0.001 iteration: 10 loss: 0.1562 lr: 0.001 Exception in thread Thread-12: Traceback (most recent call last): File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call return fn(*args) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled [[{{node fifo_queue_enqueue}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py", line 83, in load_and_enqueue sess.run(enqueue_op, feed_dict=food) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 958, in run run_metadata_ptr) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1181, in _run feed_dict_tensor, options, run_metadata) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run run_metadata) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled [[node fifo_queue_enqueue (defined at /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py:69) ]] Errors may have originated from an input operation. Input Source operations connected to node fifo_queue_enqueue: Placeholder_2 (defined at /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py:59) fifo_queue (defined at /Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py:68) Original stack trace for 'fifo_queue_enqueue': File "testscript.py", line 222, in dlc.train_network(path_config_file,shuffle=2,allow_growth=True) File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/training.py", line 132, in train_network train(str(poseconfigfile),displayiters,saveiters,maxiters,max_to_keep=max_snapshots_to_keep,keepdeconvweights=keepdeconvweights,allow_growth=allow_growth) #pass on path and file name for pose_cfg.yaml! File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py", line 120, in train batch, enqueue_op, placeholders = setup_preloading(batch_spec) File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/train.py", line 69, in setup_preloading enqueue_op = q.enqueue(placeholders_list) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 345, in enqueue self._queue_ref, vals, name=scope) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 4028, in queue_enqueue_v2 timeout_ms=timeout_ms, name=name) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper attrs=attr_protos, op_def=op_def) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3327, in _create_op_internal op_def=op_def) File "/Users/mwmathis/opt/anaconda3/envs/DLC-CPU/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1791, in __init__ self._traceback = tf_stack.extract_stack() The network is now trained and ready to evaluate. Use the function 'evaluate_network' to evaluate the network. ANALYZING some individual frames Traceback (most recent call last): File "testscript.py", line 225, in dlc.analyze_time_lapse_frames(path_config_file,os.path.join(cfg['project_path'],'labeled-data/reachingvideo1/')) File "/Users/mwmathis/Documents/DeepLabCut-core/deeplabcutcore/pose_estimation_tensorflow/predict_videos.py", line 681, in analyze_time_lapse_frames TF.reset_default_graph() AttributeError: module 'tensorflow' has no attribute 'reset_default_graph' (DLC-CPU) mwmathis@mackenzies-macbook-pro DeepLabCut-core % ```