DLC model not getting trained

emreybroyles commented 3 days ago

Describe the bug sgp.DLCModelTraining.populate(model_training_key) did not result in my DLC model getting trained.

Expected behavior I expected this populate call to take many hours, sometimes up to 24 hours, and result in the 'snapshot_id' (see below; how many iterations the model was trained) to be in the hundred thousands, maximum 1,003,000. Here, it outputted that the model was trained after 2 iterations and just a few minutes of running.

Code block

video_list = ['/stelmo/sc4712/rats/SC1001/raw/20230909/20230909_SC1001_09_s5.2.h264',
              '/stelmo/sc4712/rats/SC1000/raw/20230911/20230911_SC1000_11_s6.2.h264',

               #SC1002 known seizure videos
              '/stelmo/sc4712/rats/SC1002/raw/20231005/20231005_SC1002_07_s4.2.h264', #stage 5 seizure here 30 min in
              '/stelmo/sc4712/rats/SC1002/raw/20231006/20231006_SC1002_05_s3.2.h264', #stage 5 seizure here 16.5 min in
              '/stelmo/sc4712/rats/SC1002/raw/20231007/20231007_SC1002_05_s3.2.h264', #stage 4 seizure here 19.5 min in

              '/stelmo/sc4712/rats/SC57/raw/20240515/20240515_SC57_03_s2.2.h264', #likely stage 5 seizure at the end
              '/stelmo/sc4712/rats/SC59/raw/20240609/20240609_SC59_01_s1.2.h264', 
              '/stelmo/sc4712/rats/SC65/raw/20240701/20240701_SC65_07_s4.2.h264',
              '/stelmo/sc4712/rats/SC66/raw/20240704/20240704_SC66_09_s5.2.h264',
              '/stelmo/sc4712/rats/SC64/raw/20240911/20240911_SC64_05_s3.2.h264', #move from wind to sc4712 in stelmo

]
team_name = "LorenLab" #this was a mistake- meant to be sc_eb
project_name = "sideSleep_noLED" #this was a mistake- meant to be _LED
frames_per_video = 30
bodyparts = ["nose", "greenLED", "redLED_C", "earL", "earR", "spine1", "spine2", "spine3",
             "spine4", "spine5", "tailBase", "tailMid", "tailTip", "forelimbL", "forelimbR"]
project_key = sgp.DLCProject.insert_new_project(
    project_name=project_name,
    bodyparts=bodyparts,
    lab_team=team_name,
    frames_per_video=frames_per_video,
    video_list=video_list,
    skip_duplicates=True,
)
training_params_name = "tutorial"
sgp.DLCModelTrainingSelection().insert1(
    {
        **project_key,
        "dlc_training_params_name": training_params_name,
        "training_id": 0,
        "model_prefix": "",
    }
)
model_training_key = (
    sgp.DLCModelTrainingSelection
    & {
        **project_key,
        "dlc_training_params_name": training_params_name,
    }
).fetch1("KEY")
sgp.DLCModelTraining.populate(model_training_key)

To Reproduce Steps to reproduce the behavior:

This error is `/home/ebroyles/Src/spyglass/notebooks/eb_notebooks/DLCNotebooks/21_DLC_sideSleep_LED.ipynb
Scroll down to cell 14
See error

Error Stack

```python 2024-09-14 15:24:16.845020: I tensorflow[/core/platform/cpu_feature_guard.cc:193](http://localhost:8886/core/platform/cpu_feature_guard.cc#line=192)] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-09-14 15:24:17.114925: I tensorflow[/core/util/util.cc:169](http://localhost:8886/core/util/util.cc#line=168)] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2024-09-14 15:24:17.152488: E tensorflow[/stream_executor/cuda/cuda_blas.cc:2981](http://localhost:8886/stream_executor/cuda/cuda_blas.cc#line=2980)] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-09-14 15:24:20.021459: W tensorflow[/stream_executor/platform/default/dso_loader.cc:64](http://localhost:8886/stream_executor/platform/default/dso_loader.cc#line=63)] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2024-09-14 15:24:20.021624: W tensorflow[/stream_executor/platform/default/dso_loader.cc:64](http://localhost:8886/stream_executor/platform/default/dso_loader.cc#line=63)] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2024-09-14 15:24:20.021635: W tensorflow[/compiler/tf2tensorrt/utils/py_utils.cc:38](http://localhost:8886/compiler/tf2tensorrt/utils/py_utils.cc#line=37)] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. Loading DLC 2.3.10... DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI) [14-Sep-24 15:24:34] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/position_dlc_training.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/position_dlc_training.py), line 188: creating training dataset [14-Sep-24 15:24:36] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py), line 277: Downloading a ImageNet-pretrained model from http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz.... [14-Sep-24 15:24:48] in /home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py, line 277: The training dataset is successfully created. Use the function 'train_network' to start training. Happy training! [14-Sep-24 15:24:48] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py), line 277: Selecting single-animal trainer Config: {'all_joints': [[0], [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14]], 'all_joints_names': ['nose', 'greenLED', 'redLED_C', 'earL', 'earR', 'spine1', 'spine2', 'spine3', 'spine4', 'spine5', 'tailBase', 'tailMid', 'tailTip', 'forelimbL', 'forelimbR'], 'alpha_r': 0.02, 'apply_prob': 0.5, 'batch_size': 1, 'contrast': {'clahe': True, 'claheratio': 0.1, 'histeq': True, 'histeqratio': 0.1}, 'convolution': {'edge': False, 'emboss': {'alpha': [0.0, 1.0], 'strength': [0.5, 1.5]}, 'embossratio': 0.1, 'sharpen': False, 'sharpenratio': 0.3}, 'crop_pad': 0, 'cropratio': 0.4, 'dataset': 'training-datasets[/iteration-0/UnaugmentedDataSet_sideSleep_noLEDSep13/sideSleep_noLED_LorenLab95shuffle1.mat](http://localhost:8886/iteration-0/UnaugmentedDataSet_sideSleep_noLEDSep13/sideSleep_noLED_LorenLab95shuffle1.mat)', 'dataset_type': 'imgaug', 'decay_steps': 30000, 'deterministic': False, 'display_iters': 1000, 'fg_fraction': 0.25, 'global_scale': 0.8, 'init_weights': '[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt)', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'location_refinement': True, 'locref_huber_loss': True, 'locref_loss_weight': 0.05, 'locref_stdev': 7.2801, 'log_dir': 'log', 'lr_init': 0.0005, 'max_input_size': 1500, 'mean_pixel': [123.68, 116.779, 103.939], 'metadataset': 'training-datasets[/iteration-0/UnaugmentedDataSet_sideSleep_noLEDSep13/Documentation_data-sideSleep_noLED_95shuffle1.pickle](http://localhost:8886/iteration-0/UnaugmentedDataSet_sideSleep_noLEDSep13/Documentation_data-sideSleep_noLED_95shuffle1.pickle)', 'min_input_size': 64, 'mirror': False, 'multi_stage': False, 'multi_step': [[0.005, 10000], [0.02, 430000], [0.002, 730000], [0.001, 1030000]], 'net_type': 'resnet_50', 'num_joints': 15, 'optimizer': 'sgd', 'pairwise_huber_loss': False, 'pairwise_predict': False, 'partaffinityfield_predict': False, 'pos_dist_thresh': 17, 'project_path': '[/nimbus/deeplabcut/projects/sideSleep_noLED-LorenLab-2024-09-13](http://localhost:8886/nimbus/deeplabcut/projects/sideSleep_noLED-LorenLab-2024-09-13)', 'regularize': False, 'rotation': 25, 'rotratio': 0.4, 'save_iters': 50000, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'scoremap_dir': 'test', 'shuffle': True, 'snapshot_prefix': '[/nimbus/deeplabcut/projects/sideSleep_noLED-LorenLab-2024-09-13/dlc-models/iteration-0/sideSleep_noLEDSep13-trainset95shuffle1/train/snapshot](http://localhost:8886/nimbus/deeplabcut/projects/sideSleep_noLED-LorenLab-2024-09-13/dlc-models/iteration-0/sideSleep_noLEDSep13-trainset95shuffle1/train/snapshot)', 'stride': 8.0, 'weigh_negatives': False, 'weigh_only_present_joints': False, 'weigh_part_predictions': False, 'weight_decay': 0.0001} [14-Sep-24 15:24:48] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py), line 277: Batch Size is 1 [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1694](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer_v1.py#line=1693): UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead. warnings.warn('`layer.apply` is deprecated and ' 2024-09-14 15:24:51.763493: I tensorflow[/core/platform/cpu_feature_guard.cc:193](http://localhost:8886/core/platform/cpu_feature_guard.cc#line=192)] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-09-14 15:24:52.602775: W tensorflow[/core/common_runtime/gpu/gpu_bfc_allocator.cc:42](http://localhost:8886/core/common_runtime/gpu/gpu_bfc_allocator.cc#line=41)] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. 2024-09-14 15:24:52.602855: I tensorflow[/core/common_runtime/gpu/gpu_device.cc:1616](http://localhost:8886/core/common_runtime/gpu/gpu_device.cc#line=1615)] Created device [/job](http://localhost:8886/job):localhost[/replica:0](http://localhost:8886/replica#line=-1)[/task:0](http://localhost:8886/task#line=-1)[/device](http://localhost:8886/device):GPU:0 with 78307 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:52:00.0, compute capability: 8.0 [14-Sep-24 15:24:52] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py), line 277: Loading ImageNet-pretrained [14-Sep-24 15:24:52] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py), line 277: resnet_50 2024-09-14 15:24:53.167764: I tensorflow[/core/common_runtime/gpu/gpu_device.cc:1616](http://localhost:8886/core/common_runtime/gpu/gpu_device.cc#line=1615)] Created device [/job](http://localhost:8886/job):localhost[/replica:0](http://localhost:8886/replica#line=-1)[/task:0](http://localhost:8886/task#line=-1)[/device](http://localhost:8886/device):GPU:0 with 78307 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:52:00.0, compute capability: 8.0 2024-09-14 15:24:54.754888: I tensorflow[/compiler/mlir/mlir_graph_optimization_pass.cc:354](http://localhost:8886/compiler/mlir/mlir_graph_optimization_pass.cc#line=353)] MLIR V1 optimization pass is not enabled [14-Sep-24 15:24:58] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py), line 277: Max_iters overwritten as [14-Sep-24 15:24:58] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py), line 277: 2 [14-Sep-24 15:24:58] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py), line 277: Training parameter: [14-Sep-24 15:24:58] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py), line 277: {'stride': 8.0, 'weigh_part_predictions': False, 'weigh_negatives': False, 'fg_fraction': 0.25, 'mean_pixel': [123.68, 116.779, 103.939], 'shuffle': True, 'snapshot_prefix': '[/nimbus/deeplabcut/projects/sideSleep_noLED-LorenLab-2024-09-13/dlc-models/iteration-0/sideSleep_noLEDSep13-trainset95shuffle1/train/snapshot](http://localhost:8886/nimbus/deeplabcut/projects/sideSleep_noLED-LorenLab-2024-09-13/dlc-models/iteration-0/sideSleep_noLEDSep13-trainset95shuffle1/train/snapshot)', 'log_dir': 'log', 'global_scale': 0.8, 'location_refinement': True, 'locref_stdev': 7.2801, 'locref_loss_weight': 0.05, 'locref_huber_loss': True, 'optimizer': 'sgd', 'intermediate_supervision': False, 'intermediate_supervision_layer': 12, 'regularize': False, 'weight_decay': 0.0001, 'crop_pad': 0, 'scoremap_dir': 'test', 'batch_size': 1, 'dataset_type': 'imgaug', 'deterministic': False, 'mirror': False, 'pairwise_huber_loss': False, 'weigh_only_present_joints': False, 'partaffinityfield_predict': False, 'pairwise_predict': False, 'all_joints': [[0], [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14]], 'all_joints_names': ['nose', 'greenLED', 'redLED_C', 'earL', 'earR', 'spine1', 'spine2', 'spine3', 'spine4', 'spine5', 'tailBase', 'tailMid', 'tailTip', 'forelimbL', 'forelimbR'], 'alpha_r': 0.02, 'apply_prob': 0.5, 'contrast': {'clahe': True, 'claheratio': 0.1, 'histeq': True, 'histeqratio': 0.1, 'gamma': False, 'sigmoid': False, 'log': False, 'linear': False}, 'convolution': {'edge': False, 'emboss': {'alpha': [0.0, 1.0], 'strength': [0.5, 1.5]}, 'embossratio': 0.1, 'sharpen': False, 'sharpenratio': 0.3}, 'cropratio': 0.4, 'dataset': 'training-datasets[/iteration-0/UnaugmentedDataSet_sideSleep_noLEDSep13/sideSleep_noLED_LorenLab95shuffle1.mat](http://localhost:8886/iteration-0/UnaugmentedDataSet_sideSleep_noLEDSep13/sideSleep_noLED_LorenLab95shuffle1.mat)', 'decay_steps': 30000, 'display_iters': 1000, 'init_weights': '[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt)', 'lr_init': 0.0005, 'max_input_size': 1500, 'metadataset': 'training-datasets[/iteration-0/UnaugmentedDataSet_sideSleep_noLEDSep13/Documentation_data-sideSleep_noLED_95shuffle1.pickle](http://localhost:8886/iteration-0/UnaugmentedDataSet_sideSleep_noLEDSep13/Documentation_data-sideSleep_noLED_95shuffle1.pickle)', 'min_input_size': 64, 'multi_stage': False, 'multi_step': [[0.005, 10000], [0.02, 430000], [0.002, 730000], [0.001, 1030000]], 'net_type': 'resnet_50', 'num_joints': 15, 'pos_dist_thresh': 17, 'project_path': '[/nimbus/deeplabcut/projects/sideSleep_noLED-LorenLab-2024-09-13](http://localhost:8886/nimbus/deeplabcut/projects/sideSleep_noLED-LorenLab-2024-09-13)', 'rotation': 25, 'rotratio': 0.4, 'save_iters': 50000, 'scale_jitter_lo': 0.5, 'scale_jitter_up': 1.25, 'covering': True, 'elastic_transform': True, 'motion_blur': True, 'motion_blur_params': {'k': 7, 'angle': (-90, 90)}} [14-Sep-24 15:24:58] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py), line 277: Starting training.... 2024-09-14 15:25:00.105475: I tensorflow[/stream_executor/cuda/cuda_dnn.cc:384](http://localhost:8886/stream_executor/cuda/cuda_dnn.cc#line=383)] Loaded cuDNN version 8907 2024-09-14 15:25:00.224542: I tensorflow[/core/platform/default/subprocess.cc:304](http://localhost:8886/core/platform/default/subprocess.cc#line=303)] Start cannot spawn child process: No such file or directory 2024-09-14 15:25:01.489704: I tensorflow[/stream_executor/cuda/cuda_blas.cc:1614](http://localhost:8886/stream_executor/cuda/cuda_blas.cc#line=1613)] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 2024-09-14 15:25:06.655888: W tensorflow[/core/kernels/queue_base.cc:277](http://localhost:8886/core/kernels/queue_base.cc#line=276)] _0_fifo_queue: Skipping cancelled enqueue attempt with queue not closed Exception in thread Thread-5: Traceback (most recent call last): File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1378](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py#line=1377), in _do_call return fn(*args) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1361](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py#line=1360), in _run_fn return self._call_tf_sessionrun(options, feed_dict, fetch_list, File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1454](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py#line=1453), in _call_tf_sessionrun return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict, tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled [[{{node fifo_queue_enqueue}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/threading.py", line 980](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/threading.py#line=979), in _bootstrap_inner self.run() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 766](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/ipkernel.py#line=765), in run_closure _threading_Thread_run(self) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/threading.py", line 917](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/threading.py#line=916), in run self._target(*self._args, **self._kwargs) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py", line 86](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py#line=85), in load_and_enqueue sess.run(enqueue_op, feed_dict=food) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 968](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py#line=967), in run result = self._run(None, fetches, feed_dict, options_ptr, File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1191](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py#line=1190), in _run results = self._do_run(handle, final_targets, final_fetches, File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1371](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py#line=1370), in _do_run return self._do_call(_run_fn, feeds, fetches, targets, options, File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1397](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/client/session.py#line=1396), in _do_call raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter tensorflow.python.framework.errors_impl.CancelledError: Graph execution error: Detected at node 'fifo_queue_enqueue' defined at (most recent call last): File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/runpy.py", line 197](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/runpy.py#line=196), in _run_module_as_main return _run_code(code, main_globals, None, File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/runpy.py", line 87](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/runpy.py#line=86), in _run_code exec(code, run_globals) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel_launcher.py", line 18](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel_launcher.py#line=17), in app.launch_new_instance() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/traitlets/config/application.py", line 1075](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/traitlets/config/application.py#line=1074), in launch_instance app.start() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 739](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelapp.py#line=738), in start self.io_loop.start() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 205](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tornado/platform/asyncio.py#line=204), in start self.asyncio_loop.run_forever() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/base_events.py", line 601](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/base_events.py#line=600), in run_forever self._run_once() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/base_events.py", line 1905](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/base_events.py#line=1904), in _run_once handle._run() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/events.py", line 80](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/events.py#line=79), in _run self._context.run(self._callback, *self._args) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 545](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py#line=544), in dispatch_queue await self.process_one() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 534](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py#line=533), in process_one await dispatch(*args) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 437](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py#line=436), in dispatch_shell await result File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 362](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/ipkernel.py#line=361), in execute_request await super().execute_request(stream, ident, parent) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 778](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py#line=777), in execute_request reply_content = await reply_content File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 449](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/ipkernel.py#line=448), in do_execute res = shell.run_cell( File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/zmqshell.py", line 549](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/zmqshell.py#line=548), in run_cell return super().run_cell(*args, **kwargs) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3048](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py#line=3047), in run_cell result = self._run_cell( File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3103](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py#line=3102), in _run_cell result = runner(coro) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 129](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/async_helpers.py#line=128), in _pseudo_sync_runner coro.send(None) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3308](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py#line=3307), in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3490](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py#line=3489), in run_ast_nodes if await self.run_code(code, result, async_=asy): File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3550](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py#line=3549), in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "[/tmp/ipykernel_476565/1266341244.py", line 16](http://localhost:8886/tmp/ipykernel_476565/1266341244.py#line=15), in sgp.DLCModelTraining.populate(model_training_key) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/utils/dj_mixin.py", line 676](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/utils/dj_mixin.py#line=675), in populate return super().populate(*restrictions, **kwargs) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/datajoint/autopopulate.py", line 241](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/datajoint/autopopulate.py#line=240), in populate error = self._populate1(key, jobs, **populate_kwargs) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/datajoint/autopopulate.py", line 292](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/datajoint/autopopulate.py#line=291), in _populate1 make(dict(key), **(make_kwargs or {})) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/position_dlc_training.py", line 206](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/position_dlc_training.py#line=205), in make train_network(dlc_cfg_filepath, **train_network_kwargs) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/training.py", line 273](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/training.py#line=272), in train_network train( File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py", line 172](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py#line=171), in train batch, enqueue_op, placeholders = setup_preloading(batch_spec) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py", line 72](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py#line=71), in setup_preloading enqueue_op = q.enqueue(placeholders_list) Node: 'fifo_queue_enqueue' Enqueue operation was cancelled [[{{node fifo_queue_enqueue}}]] Original stack trace for 'fifo_queue_enqueue': File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/runpy.py", line 197](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/runpy.py#line=196), in _run_module_as_main return _run_code(code, main_globals, None, File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/runpy.py", line 87](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/runpy.py#line=86), in _run_code exec(code, run_globals) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel_launcher.py", line 18](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel_launcher.py#line=17), in app.launch_new_instance() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/traitlets/config/application.py", line 1075](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/traitlets/config/application.py#line=1074), in launch_instance app.start() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 739](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelapp.py#line=738), in start self.io_loop.start() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 205](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tornado/platform/asyncio.py#line=204), in start self.asyncio_loop.run_forever() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/base_events.py", line 601](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/base_events.py#line=600), in run_forever self._run_once() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/base_events.py", line 1905](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/base_events.py#line=1904), in _run_once handle._run() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/events.py", line 80](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/asyncio/events.py#line=79), in _run self._context.run(self._callback, *self._args) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 545](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py#line=544), in dispatch_queue await self.process_one() File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 534](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py#line=533), in process_one await dispatch(*args) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 437](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py#line=436), in dispatch_shell await result File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 362](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/ipkernel.py#line=361), in execute_request await super().execute_request(stream, ident, parent) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 778](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/kernelbase.py#line=777), in execute_request reply_content = await reply_content File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 449](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/ipkernel.py#line=448), in do_execute res = shell.run_cell( File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/zmqshell.py", line 549](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/ipykernel/zmqshell.py#line=548), in run_cell return super().run_cell(*args, **kwargs) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3048](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py#line=3047), in run_cell result = self._run_cell( File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3103](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py#line=3102), in _run_cell result = runner(coro) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 129](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/async_helpers.py#line=128), in _pseudo_sync_runner coro.send(None) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3308](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py#line=3307), in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3490](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py#line=3489), in run_ast_nodes if await self.run_code(code, result, async_=asy): File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3550](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/IPython/core/interactiveshell.py#line=3549), in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "[/tmp/ipykernel_476565/1266341244.py", line 16](http://localhost:8886/tmp/ipykernel_476565/1266341244.py#line=15), in sgp.DLCModelTraining.populate(model_training_key) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/utils/dj_mixin.py", line 676](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/utils/dj_mixin.py#line=675), in populate return super().populate(*restrictions, **kwargs) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/datajoint/autopopulate.py", line 241](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/datajoint/autopopulate.py#line=240), in populate error = self._populate1(key, jobs, **populate_kwargs) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/datajoint/autopopulate.py", line 292](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/datajoint/autopopulate.py#line=291), in _populate1 make(dict(key), **(make_kwargs or {})) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/position_dlc_training.py", line 206](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/position_dlc_training.py#line=205), in make train_network(dlc_cfg_filepath, **train_network_kwargs) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/training.py", line 273](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/training.py#line=272), in train_network train( File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py", line 172](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py#line=171), in train batch, enqueue_op, placeholders = setup_preloading(batch_spec) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py", line 72](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/core/train.py#line=71), in setup_preloading enqueue_op = q.enqueue(placeholders_list) File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/ops/data_flow_ops.py", line 346](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/ops/data_flow_ops.py#line=345), in enqueue return gen_data_flow_ops.queue_enqueue_v2( File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 4063](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/ops/gen_data_flow_ops.py#line=4062), in queue_enqueue_v2 _, _, _op, _outputs = _op_def_library._apply_op_helper( File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 797](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py#line=796), in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "[/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3800](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/tensorflow/python/framework/ops.py#line=3799), in _create_op_internal ret = Operation( [14-Sep-24 15:25:10] in [/home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py](http://localhost:8886/lab/tree/Src/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py), line 277: The network is now trained and ready to evaluate. Use the function 'evaluate_network' to evaluate the network. Inserted sideSleep_noLED_tutorial_00 from sideSleep_noLED into DLCModelSource ```

CBroz1 commented 2 days ago

Hi @emreybroyles - Please post a code block I can use to replicate this error

samuelbray32 commented 2 days ago

I'm going to guess that your spyglass settings have test_mode = True, which would create that behavior here. You can check with:

from spyglass.settings import test_mode
print(test_mode)

emreybroyles commented 2 days ago

@CBroz1 edited the issue- let me know if that isn't sufficient!

emreybroyles commented 2 days ago

@samuelbray32 actually, it looks like I have 'test_mode': 'false'. So I am not sure why that happened. That is good to know, though- i wasn't aware of the test mode parameter. I can do a PR later to update the notebook that we need to set that parameter/make people aware of it.

samuelbray32 commented 2 days ago

The parameter is False by default so I don't think we need to update the notebooks (more information than necessary can confuse people).

My guess could be wrong, just seemed coincidential. I'll trace through that piece more tomorrow and see if it still may be related

samuelbray32 commented 10 hours ago

Your output is showing that the max_iters is being set to 2 before calling the train function:

[14-Sep-24 15:24:58] in /home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py, line 277: Max_iters overwritten as [14-Sep-24 15:24:58] in /home/ebroyles/anaconda3/envs/spyglass-dlc/lib/python3.9/site-packages/spyglass/position/v1/dlc_utils.py, line 277: 2

This message is generated by DLC here

From what I can tell from the parameter set you're using dosen't have a maxiters value so it's not coming from there

CBroz1 commented 10 hours ago

test_mode sets maxiters for our pytest

samuelbray32 commented 10 hours ago

Looking at the notebook you referenced, it looks like your value for test_mode is the string "false" rather than the bool False. Evaluating bool("false") returns True, which would explain your outcome

My guess is that got in your dj config from here

LorenFrankLab / spyglass

DLC model not getting trained #1111