roomrys commented 1 month ago

Description

This PR skips the attempted previous prerelease of 1.4.1a3 (that does no bumpy the python version). But, we were running into trouble with conflicting h5py packages on the WIndows build manual test. So, here we are - releasing what was intended to be 1.4.1a4 as 1.4.1a3.

Types of changes

[ ] Bugfix
[ ] New feature
[ ] Refactor / Code style update (no logical changes)
[ ] Build / CI changes
[ ] Documentation Update
[x] Other (pre-release)

Does this address any currently open issues?

[list open issues here]

Outside contributors checklist

[ ] Review the guidelines for contributing to this repository
[ ] Read and sign the CLA and add yourself to the authors list
[ ] Make sure you are making a pull request against the develop branch (not main). Also you should start your branch off develop
[ ] Add tests that prove your fix is effective or that your feature works
[ ] Add necessary documentation (if appropriate)

Thank you for contributing to SLEAP!

:heart:

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 75.55%. Comparing base (8252868) to head (7b77b7a).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## elizabeth/update-python-and-dependencies #1989 +/- ## ============================================================================ - Coverage 75.62% 75.55% -0.07% ============================================================================ Files 133 133 Lines 24628 24628 ============================================================================ - Hits 18625 18608 -17 - Misses 6003 6020 +17 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

roomrys commented 1 month ago

Windows (manual test)

Backwards compatibility

#### pip freeze ```bash ``` #### mamba list ```bash ```

Training/Inference via GUI

![image](https://github.com/user-attachments/assets/aea067d4-8369-4714-aadc-0e3dd5585889) GPU Memory usage skyrockets when we start the training loop. ![image](https://github.com/user-attachments/assets/ea545caf-ca88-47ef-b0f4-ff7a720f46c7) ```bash }, "instance_cropping": { "center_on_part": "abdomen", "crop_size": 576, "crop_size_detection_padding": 16 } }, "model": { "backbone": { "leap": null, "unet": { "stem_stride": null, "max_stride": 32, "output_stride": 2, "filters": 24, "filters_rate": 1.5, "middle_block": true, "up_interpolate": true, "stacks": 1 }, "hourglass": null, "resnet": null, "pretrained_encoder": null }, "heads": { "single_instance": null, "centroid": { "anchor_part": "abdomen", "sigma": 2.5, "output_stride": 2, "loss_weight": 1.0, "offset_refinement": false }, "centered_instance": null, "multi_instance": null, "multi_class_bottomup": null, "multi_class_topdown": null }, "base_checkpoint": null }, "optimization": { "preload_data": true, "augmentation_config": { "rotate": true, "rotation_min_angle": -180.0, "rotation_max_angle": 180.0, "translate": false, "translate_min": -5, "translate_max": 5, "scale": false, "scale_min": 0.9, "scale_max": 1.1, "uniform_noise": false, "uniform_noise_min_val": 0.0, "uniform_noise_max_val": 10.0, "gaussian_noise": false, "gaussian_noise_mean": 5.0, "gaussian_noise_stddev": 1.0, "contrast": false, "contrast_min_gamma": 0.5, "contrast_max_gamma": 2.0, "brightness": false, "brightness_min_val": 0.0, "brightness_max_val": 10.0, "random_crop": false, "random_crop_height": 256, "random_crop_width": 256, "random_flip": false, "flip_horizontal": true }, "online_shuffling": true, "shuffle_buffer_size": 128, "prefetch": true, "batch_size": 4, "batches_per_epoch": 200, "min_batches_per_epoch": 200, "val_batches_per_epoch": 10, "min_val_batches_per_epoch": 10, "epochs": 2, "optimizer": "adam", "initial_learning_rate": 0.0001, "learning_rate_schedule": { "reduce_on_plateau": true, "reduction_factor": 0.5, "plateau_min_delta": 1e-06, "plateau_patience": 5, "plateau_cooldown": 3, "min_learning_rate": 1e-08 }, "hard_keypoint_mining": { "online_mining": false, "hard_to_easy_ratio": 2.0, "min_hard_keypoints": 2, "max_hard_keypoints": null, "loss_scale": 5.0 }, "early_stopping": { "stop_training_on_plateau": true, "plateau_min_delta": 1e-08, "plateau_patience": 20 } }, "outputs": { "save_outputs": true, "run_name": "241009_114324.centroid.n=3", "run_name_prefix": "", "run_name_suffix": "", "runs_folder": "D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\\models", "tags": [ "" ], "save_visualizations": true, "keep_viz_images": false, "zip_outputs": false, "log_to_csv": true, "checkpointing": { "initial_model": false, "best_model": true, "every_epoch": false, "latest_model": false, "final_model": false }, "tensorboard": { "write_logs": false, "loss_frequency": "epoch", "architecture_graph": false, "profile_graph": false, "visualizations": true }, "zmq": { "subscribe_to_controller": true, "controller_address": "tcp://127.0.0.1:8998", "controller_polling_timeout": 10, "publish_updates": true, "publish_address": "tcp://127.0.0.1:9001" } }, "name": "", "description": "", "sleap_version": "1.4.1a2", "filename": "C:\\Users\\TalmoLab\\AppData\\Local\\Temp\\tmp4fv5srb5\\241009_114325_training_job.json" } INFO:sleap.nn.training: INFO:sleap.nn.training:Auto-selected GPU 0 with 22982 MiB of free memory. INFO:sleap.nn.training:Using GPU 0 for acceleration. INFO:sleap.nn.training:Disabled GPU memory pre-allocation. INFO:sleap.nn.training:System: GPUs: 1/1 available Device: /physical_device:GPU:0 Available: True Initialized: False Memory growth: True INFO:sleap.nn.training: INFO:sleap.nn.training:Initializing trainer... INFO:sleap.nn.training:Loading training labels from: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/labels.v001.slp INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1 INFO:sleap.nn.training: Splits: Training = 2 / Validation = 1. INFO:sleap.nn.training:Setting up for training... INFO:sleap.nn.training:Setting up pipeline builders... INFO:sleap.nn.training:Setting up model... INFO:sleap.nn.training:Building test pipeline... 2024-10-09 11:43:37.689755: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-09 11:43:38.141710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21631 MB memory: -> device: 0, name: NVIDIA RTX A5000, pci bus id: 0000:01:00.0, compute capability: 8.6 INFO:sleap.nn.training:Loaded test example. [1.944s] INFO:sleap.nn.training: Input shape: (1024, 1024, 1) INFO:sleap.nn.training:Created Keras model. INFO:sleap.nn.training: Backbone: UNet(stacks=1, filters=24, filters_rate=1.5, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=5, middle_block=True, up_blocks=4, up_interpolate=True, block_contraction=False) INFO:sleap.nn.training: Max stride: 32 INFO:sleap.nn.training: Parameters: 1,685,625 INFO:sleap.nn.training: Heads: INFO:sleap.nn.training: [0] = CentroidConfmapsHead(anchor_part='abdomen', sigma=2.5, output_stride=2, loss_weight=1.0) INFO:sleap.nn.training: Outputs: INFO:sleap.nn.training: [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 512, 512, 1), dtype=tf.float32, name=None), name='CentroidConfmapsHead/BiasAdd:0', description="created by layer 'CentroidConfmapsHead'") INFO:sleap.nn.training:Training from scratch INFO:sleap.nn.training:Setting up data pipelines... INFO:sleap.nn.training:Training set: n = 2 INFO:sleap.nn.training:Validation set: n = 1 INFO:sleap.nn.training:Setting up optimization... INFO:sleap.nn.training: Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08) INFO:sleap.nn.training: Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=20) INFO:sleap.nn.training:Setting up outputs... INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:8998 (topic: ) INFO:sleap.nn.training: ZMQ controller subcribed to: tcp://127.0.0.1:8998 INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set INFO:sleap.nn.training: ZMQ progress reporter publish on: tcp://127.0.0.1:9001 INFO:sleap.nn.training:Created run path: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114324.centroid.n=3 INFO:sleap.nn.training:Setting up visualization... 2024-10-09 11:43:41.557756: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -34 } dim { size: -35 } dim { size: -36 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA RTX A5000" frequency: 1695 num_cores: 64 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22681944064 bandwidth: 768096000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -37 } dim { size: -38 } dim { size: 1 } } } 2024-10-09 11:43:42.437965: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -34 } dim { size: -35 } dim { size: -36 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA RTX A5000" frequency: 1695 num_cores: 64 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22681944064 bandwidth: 768096000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -37 } dim { size: -38 } dim { size: 1 } } } INFO:sleap.nn.training:Finished trainer set up. [4.8s] INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation... INFO:sleap.nn.training:Finished creating training datasets. [1.7s] INFO:sleap.nn.training:Starting training loop... Epoch 1/2 2024-10-09 11:43:47.494736: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8201 WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0106s vs `on_train_batch_end` time: 0.1389s). Check your callbacks. 2024-10-09 11:44:42.094369: W tensorflow/core/common_runtime/bfc_allocator.cc:290] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.28GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available. 2024-10-09 11:44:42.095361: W tensorflow/core/common_runtime/bfc_allocator.cc:290] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.28GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available. 2024-10-09 11:44:42.096274: W tensorflow/core/common_runtime/bfc_allocator.cc:290] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.28GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available. 2024-10-09 11:44:42.097218: W tensorflow/core/common_runtime/bfc_allocator.cc:290] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.28GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available. 2024-10-09 11:44:45.250313: I tensorflow/stream_executor/cuda/cuda_blas.cc:1786] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 200/200 - 62s - loss: 1.3333e-04 - val_loss: 6.3918e-05 - lr: 1.0000e-04 - 62s/epoch - 312ms/step Epoch 2/2 Polling: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114324.centroid.n=3\viz\validation.*.png 200/200 - 40s - loss: 4.7547e-05 - val_loss: 7.7009e-05 - lr: 1.0000e-04 - 40s/epoch - 202ms/step Polling: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114324.centroid.n=3\viz\validation.*.png INFO:sleap.nn.training:Finished training loop. [1.7 min] INFO:sleap.nn.training:Deleting visualization directory: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114324.centroid.n=3\viz INFO:sleap.nn.training:Saving evaluation metrics to model folder... Predicting... ---------------------------------------- 0% ETA: -:--:-- ?2024-10-09 11:45:30.200022: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -65 } dim { size: -66 } dim { size: -67 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA RTX A5000" frequency: 1695 num_cores: 64 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22681944064 bandwidth: 768096000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -68 } dim { size: -69 } dim { size: 1 } } } 2024-10-09 11:45:30.201453: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 2 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "103" frequency: 3600 num_cores: 16 environment { key: "cpu_instruction_set" value: "SSE, SSE2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 49152 l2_cache_size: 524288 l3_cache_size: 16777216 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -77 } dim { size: -78 } dim { size: 1 } } } Predicting... ---------------------------------------- 100% ETA: 0:00:00 ? C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\evals.py:539: RuntimeWarning: Mean of empty slice "dist.avg": np.nanmean(dists), C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\evals.py:572: RuntimeWarning: Mean of empty slice. mPCK = mPCK_parts.mean() C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\numpy\core\_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide ret = ret.dtype.type(ret / rcount) C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\evals.py:666: RuntimeWarning: Mean of empty slice. pair_pck = metrics["pck.pcks"].mean(axis=-1).mean(axis=-1) C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\numpy\core\_methods.py:121: RuntimeWarning: invalid value encountered in divide ret = um.true_divide( C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\evals.py:668: RuntimeWarning: Mean of empty slice. metrics["oks.mOKS"] = pair_oks.mean() WARNING:sleap.nn.evals:Failed to compute metrics. INFO:sleap.nn.evals:Saved predictions: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114324.centroid.n=3\labels_pr.train.slp Predicting... ---------------------------------------- 0% ETA: -:--:-- ?2024-10-09 11:45:33.354849: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -65 } dim { size: -66 } dim { size: -67 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA RTX A5000" frequency: 1695 num_cores: 64 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22681944064 bandwidth: 768096000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -68 } dim { size: -69 } dim { size: 1 } } } 2024-10-09 11:45:33.356694: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "103" frequency: 3600 num_cores: 16 environment { key: "cpu_instruction_set" value: "SSE, SSE2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 49152 l2_cache_size: 524288 l3_cache_size: 16777216 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -77 } dim { size: -78 } dim { size: 1 } } } Predicting... ---------------------------------------- 100% ETA: 0:00:00 ? WARNING:sleap.nn.evals:Failed to compute metrics. INFO:sleap.nn.evals:Saved predictions: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114324.centroid.n=3\labels_pr.val.slp INFO:sleap.nn.callbacks:Closing the reporter controller/context. INFO:sleap.nn.callbacks:Closing the training controller socket/context. Run Path: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114324.centroid.n=3 Finished training centroid. Resetting monitor window. Polling: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114536.centered_instance.n=3\viz\validation.*.png Start training centered_instance... ['sleap-train', 'C:\\Users\\TalmoLab\\AppData\\Local\\Temp\\tmp1y6bl0lp\\241009_114536_training_job.json', 'D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/labels.v001.slp', '--zmq', '--controller_port', '8998', '--publish_port', '9001', '--save_viz'] C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\albumentations\__init__.py:13: UserWarning: A new version of Albumentations is available: 1.4.18 (you have 1.4.15). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1. check_for_updates() INFO:sleap.nn.training:Versions: SLEAP: 1.4.1a3 TensorFlow: 2.9.2 Numpy: 1.26.4 Python: 3.10.15 OS: Windows-10-10.0.19044-SP0 INFO:sleap.nn.training:Training labels file: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/labels.v001.slp INFO:sleap.nn.training:Training profile: C:\Users\TalmoLab\AppData\Local\Temp\tmp1y6bl0lp\241009_114536_training_job.json INFO:sleap.nn.training: INFO:sleap.nn.training:Arguments: INFO:sleap.nn.training:{ "training_job_path": "C:\\Users\\TalmoLab\\AppData\\Local\\Temp\\tmp1y6bl0lp\\241009_114536_training_job.json", "labels_path": "D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/labels.v001.slp", "video_paths": [ "" ], "val_labels": null, "test_labels": null, "base_checkpoint": null, "tensorboard": false, "save_viz": true, "keep_viz": false, "zmq": true, "publish_port": 9001, "controller_port": 8998, "run_name": "", "prefix": "", "suffix": "", "cpu": false, "first_gpu": false, "last_gpu": false, "gpu": "auto" } INFO:sleap.nn.training: INFO:sleap.nn.training:Training job: INFO:sleap.nn.training:{ "data": { "labels": { "training_labels": "D:\\social-leap-estimates-animal-poses\\datasets\\drosophila-melanogaster-courtship\\drosophila-melanogaster-courtship\\courtship_labels.slp", "validation_labels": null, "validation_fraction": 0.1, "test_labels": null, "split_by_inds": false, "training_inds": [ 38, 35, 63, 11, 13, 42, 78, 25, 61, 57, 79, 24, 85, 12, 89, 64, 18, 96, 32, 72, 26, 20, 46, 68, 84, 6, 59, 73, 17, 75, 29, 66, 56, 7, 9, 77, 31, 41, 80, 94, 76, 27, 15, 60, 39, 45, 49, 69, 92, 65, 55, 34, 48, 16, 33, 14, 2, 8, 44, 28, 47, 21, 54, 87, 3, 37, 99, 98, 58, 4, 10, 0, 95, 91, 50, 22, 67, 74, 40, 82, 62, 19, 86, 36, 88, 51, 30, 71, 23, 83, 52 ], "validation_inds": [ 53, 43, 100, 90, 1, 70, 97, 5, 93, 81 ], "test_inds": null, "search_path_hints": [ "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "" ], "skeletons": [] }, "preprocessing": { "ensure_rgb": false, "ensure_grayscale": false, "imagenet_mode": null, "input_scaling": 1.0, "pad_to_stride": 1, "resize_and_pad_to_target": true, "target_height": 1280, "target_width": 1280 }, "instance_cropping": { "center_on_part": "abdomen", "crop_size": 256, "crop_size_detection_padding": 16 } }, "model": { "backbone": { "leap": null, "unet": { "stem_stride": null, "max_stride": 128, "output_stride": 4, "filters": 24, "filters_rate": 2.0, "middle_block": true, "up_interpolate": true, "stacks": 1 }, "hourglass": null, "resnet": null, "pretrained_encoder": null }, "heads": { "single_instance": null, "centroid": null, "centered_instance": { "anchor_part": "abdomen", "part_names": [ "head", "thorax", "abdomen", "wingL", "wingR", "forelegL4", "forelegR4", "midlegL4", "midlegR4", "hindlegL4", "hindlegR4", "eyeL", "eyeR" ], "sigma": 2.5, "output_stride": 4, "loss_weight": 1.0, "offset_refinement": false }, "multi_instance": null, "multi_class_bottomup": null, "multi_class_topdown": null }, "base_checkpoint": null }, "optimization": { "preload_data": true, "augmentation_config": { "rotate": true, "rotation_min_angle": -180.0, "rotation_max_angle": 180.0, "translate": false, "translate_min": -5, "translate_max": 5, "scale": false, "scale_min": 0.9, "scale_max": 1.1, "uniform_noise": false, "uniform_noise_min_val": 0.0, "uniform_noise_max_val": 10.0, "gaussian_noise": false, "gaussian_noise_mean": 5.0, "gaussian_noise_stddev": 1.0, "contrast": false, "contrast_min_gamma": 0.5, "contrast_max_gamma": 2.0, "brightness": false, "brightness_min_val": 0.0, "brightness_max_val": 10.0, "random_crop": false, "random_crop_height": 256, "random_crop_width": 256, "random_flip": false, "flip_horizontal": false }, "online_shuffling": true, "shuffle_buffer_size": 128, "prefetch": true, "batch_size": 4, "batches_per_epoch": 200, "min_batches_per_epoch": 200, "val_batches_per_epoch": 10, "min_val_batches_per_epoch": 10, "epochs": 2, "optimizer": "adam", "initial_learning_rate": 0.0001, "learning_rate_schedule": { "reduce_on_plateau": true, "reduction_factor": 0.5, "plateau_min_delta": 1e-06, "plateau_patience": 5, "plateau_cooldown": 3, "min_learning_rate": 1e-08 }, "hard_keypoint_mining": { "online_mining": false, "hard_to_easy_ratio": 2.0, "min_hard_keypoints": 2, "max_hard_keypoints": null, "loss_scale": 5.0 }, "early_stopping": { "stop_training_on_plateau": true, "plateau_min_delta": 1e-08, "plateau_patience": 10 } }, "outputs": { "save_outputs": true, "run_name": "241009_114536.centered_instance.n=3", "run_name_prefix": "", "run_name_suffix": "", "runs_folder": "D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\\models", "tags": [ "" ], "save_visualizations": true, "keep_viz_images": false, "zip_outputs": false, "log_to_csv": true, "checkpointing": { "initial_model": false, "best_model": true, "every_epoch": false, "latest_model": false, "final_model": false }, "tensorboard": { "write_logs": false, "loss_frequency": "epoch", "architecture_graph": false, "profile_graph": false, "visualizations": true }, "zmq": { "subscribe_to_controller": true, "controller_address": "tcp://127.0.0.1:8998", "controller_polling_timeout": 10, "publish_updates": true, "publish_address": "tcp://127.0.0.1:9001" } }, "name": "", "description": "", "sleap_version": "1.4.1a2", "filename": "C:\\Users\\TalmoLab\\AppData\\Local\\Temp\\tmp1y6bl0lp\\241009_114536_training_job.json" } INFO:sleap.nn.training: INFO:sleap.nn.training:Auto-selected GPU 0 with 23459 MiB of free memory. INFO:sleap.nn.training:Using GPU 0 for acceleration. INFO:sleap.nn.training:Disabled GPU memory pre-allocation. INFO:sleap.nn.training:System: GPUs: 1/1 available Device: /physical_device:GPU:0 Available: True Initialized: False Memory growth: True INFO:sleap.nn.training: INFO:sleap.nn.training:Initializing trainer... INFO:sleap.nn.training:Loading training labels from: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/labels.v001.slp INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1 INFO:sleap.nn.training: Splits: Training = 2 / Validation = 1. INFO:sleap.nn.training:Setting up for training... INFO:sleap.nn.training:Setting up pipeline builders... INFO:sleap.nn.training:Setting up model... INFO:sleap.nn.training:Building test pipeline... 2024-10-09 11:45:42.918245: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-09 11:45:43.267306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21631 MB memory: -> device: 0, name: NVIDIA RTX A5000, pci bus id: 0000:01:00.0, compute capability: 8.6 2024-10-09 11:45:45.138164: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1280 } dim { size: 1280 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "103" frequency: 3600 num_cores: 16 environment { key: "cpu_instruction_set" value: "SSE, SSE2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 49152 l2_cache_size: 524288 l3_cache_size: 16777216 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 256 } dim { size: 256 } dim { size: 1 } } } INFO:sleap.nn.training:Loaded test example. [2.358s] INFO:sleap.nn.training: Input shape: (256, 256, 1) INFO:sleap.nn.training:Created Keras model. INFO:sleap.nn.training: Backbone: UNet(stacks=1, filters=24, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=7, middle_block=True, up_blocks=5, up_interpolate=True, block_contraction=False) INFO:sleap.nn.training: Max stride: 128 INFO:sleap.nn.training: Parameters: 283,019,413 INFO:sleap.nn.training: Heads: INFO:sleap.nn.training: [0] = CenteredInstanceConfmapsHead(part_names=['head', 'thorax', 'abdomen', 'wingL', 'wingR', 'forelegL4', 'forelegR4', 'midlegL4', 'midlegR4', 'hindlegL4', 'hindlegR4', 'eyeL', 'eyeR'], anchor_part='abdomen', sigma=2.5, output_stride=4, loss_weight=1.0) INFO:sleap.nn.training: Outputs: INFO:sleap.nn.training: [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 64, 64, 13), dtype=tf.float32, name=None), name='CenteredInstanceConfmapsHead/BiasAdd:0', description="created by layer 'CenteredInstanceConfmapsHead'") INFO:sleap.nn.training:Training from scratch INFO:sleap.nn.training:Setting up data pipelines... INFO:sleap.nn.training:Training set: n = 2 INFO:sleap.nn.training:Validation set: n = 1 INFO:sleap.nn.training:Setting up optimization... INFO:sleap.nn.training: Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08) INFO:sleap.nn.training: Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=10) INFO:sleap.nn.training:Setting up outputs... INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:8998 (topic: ) INFO:sleap.nn.training: ZMQ controller subcribed to: tcp://127.0.0.1:8998 INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set INFO:sleap.nn.training: ZMQ progress reporter publish on: tcp://127.0.0.1:9001 INFO:sleap.nn.training:Created run path: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114536.centered_instance.n=3 INFO:sleap.nn.training:Setting up visualization... 2024-10-09 11:45:46.311081: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1280 } dim { size: 1280 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "103" frequency: 3600 num_cores: 16 environment { key: "cpu_instruction_set" value: "SSE, SSE2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 49152 l2_cache_size: 524288 l3_cache_size: 16777216 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 256 } dim { size: 256 } dim { size: 1 } } } 2024-10-09 11:45:47.120562: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1280 } dim { size: 1280 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "103" frequency: 3600 num_cores: 16 environment { key: "cpu_instruction_set" value: "SSE, SSE2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 49152 l2_cache_size: 524288 l3_cache_size: 16777216 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 256 } dim { size: 256 } dim { size: 1 } } } INFO:sleap.nn.training:Finished trainer set up. [4.3s] INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation... 2024-10-09 11:45:48.033762: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1280 } dim { size: 1280 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "103" frequency: 3600 num_cores: 16 environment { key: "cpu_instruction_set" value: "SSE, SSE2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 49152 l2_cache_size: 524288 l3_cache_size: 16777216 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 256 } dim { size: 256 } dim { size: 1 } } } 2024-10-09 11:45:49.597663: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1280 } dim { size: 1280 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "103" frequency: 3600 num_cores: 16 environment { key: "cpu_instruction_set" value: "SSE, SSE2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 49152 l2_cache_size: 524288 l3_cache_size: 16777216 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 256 } dim { size: 256 } dim { size: 1 } } } INFO:sleap.nn.training:Finished creating training datasets. [2.7s] INFO:sleap.nn.training:Starting training loop... 2024-10-09 11:45:50.090577: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1280 } dim { size: 1280 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "103" frequency: 3600 num_cores: 16 environment { key: "cpu_instruction_set" value: "SSE, SSE2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 49152 l2_cache_size: 524288 l3_cache_size: 16777216 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 256 } dim { size: 256 } dim { size: 1 } } } Epoch 1/2 2024-10-09 11:45:52.328777: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8201 WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0130s vs `on_train_batch_end` time: 0.1220s). Check your callbacks. 2024-10-09 11:46:29.998998: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1280 } dim { size: 1280 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "103" frequency: 3600 num_cores: 16 environment { key: "cpu_instruction_set" value: "SSE, SSE2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 49152 l2_cache_size: 524288 l3_cache_size: 16777216 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 256 } dim { size: 256 } dim { size: 1 } } } 2024-10-09 11:46:31.824231: I tensorflow/stream_executor/cuda/cuda_blas.cc:1786] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once. 2024-10-09 11:46:32.432660: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 13 } dim { size: 64 } dim { size: 64 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA RTX A5000" frequency: 1695 num_cores: 64 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22681944064 bandwidth: 768096000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -5 } dim { size: -6 } dim { size: 1 } } } 200/200 - 48s - loss: 0.0024 - head: 0.0018 - thorax: 0.0021 - abdomen: 0.0027 - wingL: 0.0030 - wingR: 0.0031 - forelegL4: 0.0014 - forelegR4: 9.9911e-05 - midlegL4: 0.0027 - midlegR4: 0.0035 - hindlegL4: 0.0016 - hindlegR4: 0.0034 - eyeL: 0.0028 - eyeR: 0.0027 - val_loss: 0.0027 - val_head: 0.0036 - val_thorax: 0.0019 - val_abdomen: 0.0013 - val_wingL: 0.0028 - val_wingR: 0.0032 - val_forelegL4: 0.0013 - val_forelegR4: 2.5745e-04 - val_midlegL4: 0.0033 - val_midlegR4: 0.0033 - val_hindlegL4: 0.0023 - val_hindlegR4: 0.0037 - val_eyeL: 0.0039 - val_eyeR: 0.0037 - lr: 1.0000e-04 - 48s/epoch - 240ms/step Epoch 2/2 Polling: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114324.centroid.n=3\viz\validation.*.png Polling: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114536.centered_instance.n=3\viz\validation.*.png 2024-10-09 11:47:12.089059: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1280 } dim { size: 1280 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "103" frequency: 3600 num_cores: 16 environment { key: "cpu_instruction_set" value: "SSE, SSE2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 49152 l2_cache_size: 524288 l3_cache_size: 16777216 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 256 } dim { size: 256 } dim { size: 1 } } } 200/200 - 40s - loss: 8.4480e-04 - head: 6.9070e-04 - thorax: 6.0177e-04 - abdomen: 7.6382e-04 - wingL: 7.1380e-04 - wingR: 7.3655e-04 - forelegL4: 0.0011 - forelegR4: 8.6102e-05 - midlegL4: 0.0012 - midlegR4: 9.9908e-04 - hindlegL4: 0.0013 - hindlegR4: 0.0014 - eyeL: 7.9498e-04 - eyeR: 7.0381e-04 - val_loss: 0.0025 - val_head: 0.0040 - val_thorax: 0.0015 - val_abdomen: 0.0012 - val_wingL: 0.0025 - val_wingR: 0.0025 - val_forelegL4: 0.0012 - val_forelegR4: 9.1834e-05 - val_midlegL4: 0.0028 - val_midlegR4: 0.0033 - val_hindlegL4: 0.0020 - val_hindlegR4: 0.0047 - val_eyeL: 0.0030 - val_eyeR: 0.0030 - lr: 1.0000e-04 - 40s/epoch - 202ms/step INFO:sleap.nn.training:Finished training loop. [1.5 min] INFO:sleap.nn.training:Deleting visualization directory: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114536.centered_instance.n=3\viz INFO:sleap.nn.training:Saving evaluation metrics to model folder... Predicting... ---------------------------------------- 0% ETA: -:--:-- ?Polling: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114324.centroid.n=3\viz\validation.*.png Polling: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114536.centered_instance.n=3\viz\validation.*.png 2024-10-09 11:47:20.567626: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 2 } dim { size: 1280 } dim { size: 1280 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "103" frequency: 3600 num_cores: 16 environment { key: "cpu_instruction_set" value: "SSE, SSE2" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 49152 l2_cache_size: 524288 l3_cache_size: 16777216 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -24 } dim { size: -25 } dim { size: 1 } } } 2024-10-09 11:47:20.579069: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -65 } dim { size: -66 } dim { size: -67 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -9 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA RTX A5000" frequency: 1695 num_cores: 64 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 22681944064 bandwidth: 768096000 } outputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: -70 } dim { size: -71 } dim { size: 1 } } } error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice 2024-10-09 11:47:21.848472: W tensorflow/core/framework/op_kernel.cc:1733] UNKNOWN: JIT compilation failed. Predicting... ---------------------------------------- 0% ETA: -:--:-- ? Traceback (most recent call last): File "\\?\C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\Scripts\sleap-train-script.py", line 33, in sys.exit(load_entry_point('sleap==1.4.1a3', 'console_scripts', 'sleap-train')()) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\training.py", line 2039, in main trainer.train() File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\training.py", line 953, in train self.evaluate() File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\training.py", line 961, in evaluate sleap.nn.evals.evaluate_model( File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\evals.py", line 744, in evaluate_model labels_pr: Labels = predictor.predict(labels_gt, make_labels=True) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 527, in predict self._make_labeled_frames_from_generator(generator, data) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 2645, in _make_labeled_frames_from_generator for ex in generator: File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 437, in _predict_generator ex = process_batch(ex) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 400, in process_batch preds = self.inference_model.predict_on_batch(ex, numpy=True) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 1070, in predict_on_batch outs = super().predict_on_batch(data, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 2230, in predict_on_batch outputs = self.predict_function(iterator) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\tensorflow\python\util\traceback_utils.py", line 153, in error_handler raise e.with_traceback(filtered_tb) from None File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\tensorflow\python\eager\execute.py", line 54, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.UnknownError: Graph execution error: Detected at node 'FloorMod' defined at (most recent call last): File "\\?\C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\Scripts\sleap-train-script.py", line 33, in sys.exit(load_entry_point('sleap==1.4.1a3', 'console_scripts', 'sleap-train')()) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\training.py", line 2039, in main trainer.train() File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\training.py", line 953, in train self.evaluate() File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\training.py", line 961, in evaluate sleap.nn.evals.evaluate_model( File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\evals.py", line 744, in evaluate_model labels_pr: Labels = predictor.predict(labels_gt, make_labels=True) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 527, in predict self._make_labeled_frames_from_generator(generator, data) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 2645, in _make_labeled_frames_from_generator for ex in generator: File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 437, in _predict_generator ex = process_batch(ex) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 400, in process_batch preds = self.inference_model.predict_on_batch(ex, numpy=True) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 1070, in predict_on_batch outs = super().predict_on_batch(data, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 2230, in predict_on_batch outputs = self.predict_function(iterator) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 1845, in predict_function return step_function(self, iterator) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 1834, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 1823, in run_step outputs = model.predict_step(data) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 1791, in predict_step return self(x, training=False) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 490, in __call__ return super().__call__(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 2267, in call if isinstance(self.instance_peaks, FindInstancePeaksGroundTruth): File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 2276, in call peaks_output = self.instance_peaks(crop_output) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 2112, in call if self.offsets_ind is None: File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 2114, in call peak_points, peak_vals = sleap.nn.peak_finding.find_global_peaks( File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\peak_finding.py", line 366, in find_global_peaks rough_peaks, peak_vals = find_global_peaks_rough( File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\peak_finding.py", line 224, in find_global_peaks_rough channel_subs = tf.math.mod(tf.range(total_peaks, dtype=tf.int64), channels) Node: 'FloorMod' Detected at node 'FloorMod' defined at (most recent call last): File "\\?\C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\Scripts\sleap-train-script.py", line 33, in sys.exit(load_entry_point('sleap==1.4.1a3', 'console_scripts', 'sleap-train')()) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\training.py", line 2039, in main trainer.train() File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\training.py", line 953, in train self.evaluate() File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\training.py", line 961, in evaluate sleap.nn.evals.evaluate_model( File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\evals.py", line 744, in evaluate_model labels_pr: Labels = predictor.predict(labels_gt, make_labels=True) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 527, in predict self._make_labeled_frames_from_generator(generator, data) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 2645, in _make_labeled_frames_from_generator for ex in generator: File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 437, in _predict_generator ex = process_batch(ex) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 400, in process_batch preds = self.inference_model.predict_on_batch(ex, numpy=True) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 1070, in predict_on_batch outs = super().predict_on_batch(data, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 2230, in predict_on_batch outputs = self.predict_function(iterator) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 1845, in predict_function return step_function(self, iterator) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 1834, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 1823, in run_step outputs = model.predict_step(data) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 1791, in predict_step return self(x, training=False) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\training.py", line 490, in __call__ return super().__call__(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 2267, in call if isinstance(self.instance_peaks, FindInstancePeaksGroundTruth): File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 2276, in call peaks_output = self.instance_peaks(crop_output) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\engine\base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 2112, in call if self.offsets_ind is None: File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\inference.py", line 2114, in call peak_points, peak_vals = sleap.nn.peak_finding.find_global_peaks( File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\peak_finding.py", line 366, in find_global_peaks rough_peaks, peak_vals = find_global_peaks_rough( File "C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\sleap\nn\peak_finding.py", line 224, in find_global_peaks_rough channel_subs = tf.math.mod(tf.range(total_peaks, dtype=tf.int64), channels) Node: 'FloorMod' 2 root error(s) found. (0) UNKNOWN: JIT compilation failed. [[{{node FloorMod}}]] [[top_down_inference_model/find_instance_peaks_1/RaggedFromValueRowIds_1/RowPartitionFromValueRowIds/assert_less/Assert/AssertGuard/pivot_f/_177/_415]] (1) UNKNOWN: JIT compilation failed. [[{{node FloorMod}}]] 0 successful operations. 0 derived errors ignored. [Op:__inference_predict_function_31992] INFO:sleap.nn.callbacks:Closing the reporter controller/context. INFO:sleap.nn.callbacks:Closing the training controller socket/context. Run Path: D:/social-leap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship\models\241009_114536.centered_instance.n=3 ```

Training Dialog

![image](https://github.com/user-attachments/assets/d7a58e6f-03ee-4903-b212-bc06d7eef934) ![image](https://github.com/user-attachments/assets/c6d060e3-39e8-4c64-adb6-69e54eb0c7c1) ![image](https://github.com/user-attachments/assets/9ef030b2-e365-420f-b54d-84871b6e0bcc) ```bash While structuring TrainingJobConfig (1 sub-exception) While structuring TrainingJobConfig (1 sub-exception) While structuring TrainingJobConfig (1 sub-exception) While structuring TrainingJobConfig (1 sub-exception) While structuring TrainingJobConfig (1 sub-exception) While structuring TrainingJobConfig (1 sub-exception) While structuring TrainingJobConfig (1 sub-exception) While structuring TrainingJobConfig (1 sub-exception) ```

sleap-label

![image](https://github.com/user-attachments/assets/35ec26d9-67bd-4531-8e66-d5cd93d016a8) ```bash (sleap_1.4.1a3_py310) λ sleap-label C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310\lib\site-packages\albumentations\__init__.py:13: UserWarning: A new version of Albumentations is available: 1.4.18 (you have 1.4.15). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1. check_for_updates() Saving config: C:\Users\TalmoLab\.sleap\1.4.1a3\preferences.yaml Software versions: SLEAP: 1.4.1a3 TensorFlow: 2.9.2 Numpy: 1.26.4 Python: 3.10.15 OS: Windows-10-10.0.19044-SP0 Happy SLEAPing! :) ```

Installation

```bash λ mamba create -n sleap_1.4.1a3_py310 -c sleap-deps -c conda-forge -c nvidia -c ./sleap-build-win -c anaconda sleap=1.4.1a3 __ __ __ __ / \ / \ / \ / \ / \/ \/ \/ \ ███████████████/ /██/ /██/ /██/ /████████████████████████ / / \ / \ / \ / \ \____ / / \_/ \_/ \_/ \ o \__, / _/ \_____/ ` |/ ███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗ ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗ ██╔████╔██║███████║██╔████╔██║██████╔╝███████║ ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║ ██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║ ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝ mamba (1.4.1) supported by @QuantStack GitHub: https://github.com/mamba-org/mamba Twitter: https://twitter.com/QuantStack █████████████████████████████████████████████████████████████ Looking for: ['sleap=1.4.1a3'] sleap-deps/win-64 Using cache sleap-deps/noarch Using cache nvidia/win-64 Using cache nvidia/noarch Using cache anaconda/noarch Using cache file://C:/Users/TalmoLab/Downloads/sleap-build-w.. 927.0 B @ 3.3MB/s 0.0s file://C:/Users/TalmoLab/Downloads/sleap-build-w.. 96.0 B @ 564.7kB/s 0.0s anaconda/win-64 3.3MB @ 5.7MB/s 0.7s conda-forge/noarch 19.4MB @ 7.5MB/s 3.3s conda-forge/win-64 29.3MB @ 6.8MB/s 5.4s Transaction Prefix: C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310 Updating specs: - sleap=1.4.1a3 Package Version Build Channel Size ------------------------------------------------------------------------------------------------------------------------------------------------------ Install: ------------------------------------------------------------------------------------------------------------------------------------------------------ + _libavif_api 1.1.1 h57928b3_1 conda-forge/win-64 9kB + albucore 0.0.16 pyhd8ed1ab_0 conda-forge/noarch 15kB + albumentations 1.4.15 pyhd8ed1ab_0 conda-forge/noarch 150kB + annotated-types 0.7.0 pyhd8ed1ab_0 conda-forge/noarch Cached + aom 3.9.1 he0c23c2_0 conda-forge/win-64 Cached + attrs 24.2.0 pyh71513ae_0 conda-forge/noarch Cached + blosc 1.21.6 h85f69ea_0 conda-forge/win-64 Cached + brotli 1.1.0 h2466b09_2 conda-forge/win-64 Cached + brotli-bin 1.1.0 h2466b09_2 conda-forge/win-64 Cached + bzip2 1.0.8 h2466b09_7 conda-forge/win-64 Cached + c-blosc2 2.15.1 hb461149_0 conda-forge/win-64 Cached + ca-certificates 2024.8.30 h56e8100_0 conda-forge/win-64 Cached + cached-property 1.5.2 hd8ed1ab_1 conda-forge/noarch Cached + cached_property 1.5.2 pyha770c72_1 conda-forge/noarch Cached + cairo 1.18.0 h32b962e_3 conda-forge/win-64 Cached + cattrs 24.1.2 pyhd8ed1ab_0 conda-forge/noarch 52kB + certifi 2024.8.30 pyhd8ed1ab_0 conda-forge/noarch Cached + charls 2.4.2 h1537add_0 conda-forge/win-64 Cached + contourpy 1.3.0 py310hc19bc0b_2 conda-forge/win-64 200kB + cuda-nvcc 11.3.58 hb8d16a4_0 nvidia/win-64 Cached + cudatoolkit 11.3.1 hf2f0253_13 conda-forge/win-64 Cached + cudnn 8.2.1.32 h754d62a_0 conda-forge/win-64 Cached + cycler 0.12.1 pyhd8ed1ab_0 conda-forge/noarch Cached + dav1d 1.2.1 hcfcfb64_0 conda-forge/win-64 Cached + double-conversion 3.3.0 h63175ca_0 conda-forge/win-64 Cached + eval-type-backport 0.2.0 pyhd8ed1ab_0 conda-forge/noarch 7kB + eval_type_backport 0.2.0 pyha770c72_0 conda-forge/noarch Cached + exceptiongroup 1.2.2 pyhd8ed1ab_0 conda-forge/noarch Cached + expat 2.6.3 he0c23c2_0 conda-forge/win-64 Cached + ffmpeg 6.1.2 gpl_h0820249_105 conda-forge/win-64 Cached + font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge/noarch Cached + font-ttf-inconsolata 3.000 h77eed37_0 conda-forge/noarch Cached + font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge/noarch Cached + font-ttf-ubuntu 0.83 h77eed37_3 conda-forge/noarch Cached + fontconfig 2.14.2 hbde0cde_0 conda-forge/win-64 Cached + fonts-conda-ecosystem 1 0 conda-forge/noarch Cached + fonts-conda-forge 1 0 conda-forge/noarch Cached + fonttools 4.54.1 py310ha8f682b_0 conda-forge/win-64 2MB + freeglut 3.2.2 he0c23c2_3 conda-forge/win-64 Cached + freetype 2.12.1 hdaf720e_2 conda-forge/win-64 Cached + giflib 5.2.2 h64bf75a_0 conda-forge/win-64 Cached + graphite2 1.3.13 h63175ca_1003 conda-forge/win-64 Cached + h5py 3.11.0 nompi_py310h2b0be38_102 conda-forge/win-64 Cached + harfbuzz 9.0.0 h2bedf89_1 conda-forge/win-64 Cached + hdf5 1.14.3 nompi_h2b43c12_105 conda-forge/win-64 Cached + hdmf 3.14.4 pyh2e8e312_0 conda-forge/noarch Cached + icu 75.1 he0c23c2_0 conda-forge/win-64 Cached + imagecodecs 2024.6.1 py310h0e2d205_5 conda-forge/win-64 2MB + imageio 2.35.1 pyh12aca89_0 conda-forge/noarch Cached + imageio-ffmpeg 0.5.1 pyhd8ed1ab_0 conda-forge/noarch Cached + imath 3.1.12 hbb528cf_0 conda-forge/win-64 Cached + importlib-metadata 8.5.0 pyha770c72_0 conda-forge/noarch Cached + importlib_metadata 8.5.0 hd8ed1ab_0 conda-forge/noarch 9kB + importlib_resources 6.4.5 pyhd8ed1ab_0 conda-forge/noarch Cached + intel-openmp 2024.2.1 h57928b3_1083 conda-forge/win-64 Cached + jasper 4.2.4 hcb1a123_0 conda-forge/win-64 Cached + joblib 1.4.2 pyhd8ed1ab_0 conda-forge/noarch Cached + jsmin 3.0.1 pyhd8ed1ab_0 conda-forge/noarch Cached + jsonpickle 1.4.1 pyh9f0ad1d_0 conda-forge/noarch Cached + jsonschema 4.23.0 pyhd8ed1ab_0 conda-forge/noarch Cached + jsonschema-specifications 2024.10.1 pyhd8ed1ab_0 conda-forge/noarch 16kB + jxrlib 1.1 hcfcfb64_3 conda-forge/win-64 Cached + khronos-opencl-icd-loader 2024.05.08 hc70643c_0 conda-forge/win-64 Cached + kiwisolver 1.4.7 py310hc19bc0b_0 conda-forge/win-64 Cached + krb5 1.21.3 hdf4eb48_0 conda-forge/win-64 Cached + lazy-loader 0.4 pyhd8ed1ab_1 conda-forge/noarch Cached + lazy_loader 0.4 pyhd8ed1ab_1 conda-forge/noarch 7kB + lcms2 2.16 h67d730c_0 conda-forge/win-64 Cached + lerc 4.0.0 h63175ca_0 conda-forge/win-64 Cached + libabseil 20240722.0 cxx17_he0c23c2_1 conda-forge/win-64 Cached + libaec 1.1.3 h63175ca_0 conda-forge/win-64 Cached + libasprintf 0.22.5 h5728263_3 conda-forge/win-64 Cached + libavif16 1.1.1 h4e96d62_1 conda-forge/win-64 Cached + libblas 3.9.0 24_win64_mkl conda-forge/win-64 Cached + libbrotlicommon 1.1.0 h2466b09_2 conda-forge/win-64 Cached + libbrotlidec 1.1.0 h2466b09_2 conda-forge/win-64 Cached + libbrotlienc 1.1.0 h2466b09_2 conda-forge/win-64 Cached + libcblas 3.9.0 24_win64_mkl conda-forge/win-64 Cached + libclang13 19.1.1 default_ha5278ca_0 conda-forge/win-64 27MB + libcurl 8.10.1 h1ee3ff0_0 conda-forge/win-64 Cached + libdeflate 1.22 h2466b09_0 conda-forge/win-64 156kB + libexpat 2.6.3 he0c23c2_0 conda-forge/win-64 Cached + libffi 3.4.2 h8ffe710_5 conda-forge/win-64 Cached + libgettextpo 0.22.5 h5728263_3 conda-forge/win-64 Cached + libglib 2.82.1 h7025463_0 conda-forge/win-64 Cached + libhwloc 2.11.1 default_h8125262_1000 conda-forge/win-64 Cached + libiconv 1.17 hcfcfb64_2 conda-forge/win-64 Cached + libintl 0.22.5 h5728263_3 conda-forge/win-64 Cached + libjpeg-turbo 3.0.0 hcfcfb64_1 conda-forge/win-64 Cached + liblapack 3.9.0 24_win64_mkl conda-forge/win-64 Cached + liblapacke 3.9.0 24_win64_mkl conda-forge/win-64 Cached + libopencv 4.10.0 qt6_py310h00b716a_607 conda-forge/win-64 33MB + libopenvino 2024.4.0 hfe1841e_1 conda-forge/win-64 Cached + libopenvino-auto-batch-plugin 2024.4.0 h04f32e0_1 conda-forge/win-64 Cached + libopenvino-auto-plugin 2024.4.0 h04f32e0_1 conda-forge/win-64 Cached + libopenvino-hetero-plugin 2024.4.0 h372dad0_1 conda-forge/win-64 Cached + libopenvino-intel-cpu-plugin 2024.4.0 hfe1841e_1 conda-forge/win-64 Cached + libopenvino-intel-gpu-plugin 2024.4.0 hfe1841e_1 conda-forge/win-64 Cached + libopenvino-ir-frontend 2024.4.0 h372dad0_1 conda-forge/win-64 Cached + libopenvino-onnx-frontend 2024.4.0 h5707d70_1 conda-forge/win-64 Cached + libopenvino-paddle-frontend 2024.4.0 h5707d70_1 conda-forge/win-64 Cached + libopenvino-pytorch-frontend 2024.4.0 he0c23c2_1 conda-forge/win-64 Cached + libopenvino-tensorflow-frontend 2024.4.0 hf4e5e90_1 conda-forge/win-64 Cached + libopenvino-tensorflow-lite-frontend 2024.4.0 he0c23c2_1 conda-forge/win-64 Cached + libopus 1.3.1 h8ffe710_1 conda-forge/win-64 Cached + libpng 1.6.44 h3ca93ac_0 conda-forge/win-64 Cached + libprotobuf 5.27.5 hcaed137_2 conda-forge/win-64 Cached + libsodium 1.0.20 hc70643c_0 conda-forge/win-64 Cached + libsqlite 3.46.1 h2466b09_0 conda-forge/win-64 Cached + libssh2 1.11.0 h7dfc565_0 conda-forge/win-64 Cached + libtiff 4.7.0 hfc51747_1 conda-forge/win-64 979kB + libwebp-base 1.4.0 hcfcfb64_0 conda-forge/win-64 Cached + libxcb 1.16 h013a479_1 conda-forge/win-64 Cached + libxml2 2.12.7 h0f24e4e_4 conda-forge/win-64 Cached + libxslt 1.1.39 h3df6e99_0 conda-forge/win-64 Cached + libzlib 1.3.1 h2466b09_2 conda-forge/win-64 Cached + libzopfli 1.0.3 h0e60522_0 conda-forge/win-64 Cached + lz4-c 1.9.4 hcfcfb64_0 conda-forge/win-64 Cached + m2w64-gcc-libgfortran 5.3.0 6 conda-forge/win-64 Cached + m2w64-gcc-libs 5.3.0 7 conda-forge/win-64 Cached + m2w64-gcc-libs-core 5.3.0 7 conda-forge/win-64 Cached + m2w64-gmp 6.1.0 2 conda-forge/win-64 Cached + m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge/win-64 Cached + markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge/noarch Cached + matplotlib-base 3.9.2 py310h37e0a56_1 conda-forge/win-64 7MB + mdurl 0.1.2 pyhd8ed1ab_0 conda-forge/noarch Cached + mkl 2024.1.0 h66d3029_694 conda-forge/win-64 Cached + msys2-conda-epoch 20160418 1 conda-forge/win-64 Cached + munkres 1.1.4 pyh9f0ad1d_0 conda-forge/noarch Cached + ndx-pose 0.1.1 pyhd8ed1ab_0 conda-forge/noarch Cached + networkx 3.3 pyhd8ed1ab_1 conda-forge/noarch Cached + numpy 1.26.4 py310hf667824_0 conda-forge/win-64 Cached + opencv 4.10.0 qt6_py310h896e0ad_607 conda-forge/win-64 27kB + openexr 3.2.2 h9aba623_2 conda-forge/win-64 Cached + openh264 2.4.1 h63175ca_0 conda-forge/win-64 Cached + openjpeg 2.5.2 h3d672ee_0 conda-forge/win-64 Cached + openssl 3.3.2 h2466b09_0 conda-forge/win-64 Cached + packaging 24.1 pyhd8ed1ab_0 conda-forge/noarch Cached + pandas 2.2.3 py310hb4db72f_1 conda-forge/win-64 12MB + patsy 0.5.6 pyhd8ed1ab_0 conda-forge/noarch Cached + pcre2 10.44 h3d7b363_2 conda-forge/win-64 Cached + pillow 10.4.0 py310h3e38d90_1 conda-forge/win-64 Cached + pip 24.2 pyh8b19718_1 conda-forge/noarch Cached + pixman 0.43.4 h63175ca_0 conda-forge/win-64 Cached + pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge/noarch Cached + protobuf 5.27.5 py310h9e98ed7_0 conda-forge/win-64 373kB + psutil 6.0.0 py310ha8f682b_1 conda-forge/win-64 Cached + pthread-stubs 0.4 hcd874cb_1001 conda-forge/win-64 Cached + pthreads-win32 2.9.1 h2466b09_4 conda-forge/win-64 Cached + pugixml 1.14 h63175ca_0 conda-forge/win-64 Cached + py-opencv 4.10.0 qt6_py310h5f8bd55_607 conda-forge/win-64 1MB + pydantic 2.9.2 pyhd8ed1ab_0 conda-forge/noarch 301kB + pydantic-core 2.23.4 py310hc226416_0 conda-forge/win-64 2MB + pygments 2.18.0 pyhd8ed1ab_0 conda-forge/noarch Cached + pykalman 0.9.7 pyhd8ed1ab_0 conda-forge/noarch Cached + pynwb 2.8.2 pyh2e8e312_0 conda-forge/noarch Cached + pyparsing 3.1.4 pyhd8ed1ab_0 conda-forge/noarch Cached + pyside6 6.7.3 py310h60c6385_1 conda-forge/win-64 9MB + python 3.10.15 hfaddaf0_1_cpython conda-forge/win-64 Cached + python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge/noarch Cached + python-rapidjson 1.20 py310h9e98ed7_0 conda-forge/win-64 Cached + python-tzdata 2024.2 pyhd8ed1ab_0 conda-forge/noarch Cached + python_abi 3.10 5_cp310 conda-forge/win-64 7kB + pytz 2024.1 pyhd8ed1ab_0 conda-forge/noarch Cached + pywavelets 1.7.0 py310hb0944cc_1 conda-forge/win-64 4MB + pyyaml 6.0.2 py310ha8f682b_1 conda-forge/win-64 Cached + pyzmq 26.2.0 py310h656833d_2 conda-forge/win-64 Cached + qhull 2020.2 hc790b64_5 conda-forge/win-64 Cached + qt6-main 6.7.3 hfb098fa_1 conda-forge/win-64 89MB + qtpy 2.4.1 pyhd8ed1ab_0 conda-forge/noarch Cached + qudida 0.0.4 pyhd8ed1ab_0 conda-forge/noarch Cached + rav1e 0.6.6 h975169c_2 conda-forge/win-64 Cached + referencing 0.35.1 pyhd8ed1ab_0 conda-forge/noarch Cached + rich 13.9.2 pyhd8ed1ab_0 conda-forge/noarch 186kB + rpds-py 0.20.0 py310hc226416_1 conda-forge/win-64 Cached + ruamel.yaml 0.18.6 py310h8d17308_0 conda-forge/win-64 Cached + ruamel.yaml.clib 0.2.8 py310h8d17308_0 conda-forge/win-64 Cached + scikit-image 0.24.0 py310hb4db72f_2 conda-forge/win-64 Cached + scikit-learn 1.5.2 py310hf2a6c47_1 conda-forge/win-64 8MB + scipy 1.14.1 py310h46043a1_0 conda-forge/win-64 Cached + seaborn 0.13.2 hd8ed1ab_2 conda-forge/noarch 7kB + seaborn-base 0.13.2 pyhd8ed1ab_2 conda-forge/noarch Cached + setuptools 75.1.0 pyhd8ed1ab_0 conda-forge/noarch Cached + six 1.16.0 pyh6c4a22f_0 conda-forge/noarch Cached + sleap 1.4.1a3 py310_0 C:/Users/TalmoLab/Downloads/sleap-build-win/win-64 3MB + snappy 1.2.1 h23299a8_0 conda-forge/win-64 Cached + statsmodels 0.14.4 py310hb0944cc_0 conda-forge/win-64 10MB + svt-av1 2.2.1 he0c23c2_0 conda-forge/win-64 Cached + tbb 2021.13.0 hc790b64_0 conda-forge/win-64 Cached + tensorflow 2.9.2 py310h82bb817_1 sleap-deps/win-64 420MB + tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge/noarch Cached + threadpoolctl 3.5.0 pyhc1e730c_0 conda-forge/noarch Cached + tifffile 2024.9.20 pyhd8ed1ab_0 conda-forge/noarch 181kB + tk 8.6.13 h5226925_1 conda-forge/win-64 Cached + tomli 2.0.2 pyhd8ed1ab_0 conda-forge/noarch 18kB + typing-extensions 4.12.2 hd8ed1ab_0 conda-forge/noarch 10kB + typing_extensions 4.12.2 pyha770c72_0 conda-forge/noarch Cached + tzdata 2024b hc8b5060_0 conda-forge/noarch Cached + ucrt 10.0.22621.0 h57928b3_1 conda-forge/win-64 Cached + unicodedata2 15.1.0 py310h8d17308_0 conda-forge/win-64 Cached + vc 14.3 h8a93ad2_22 conda-forge/win-64 Cached + vc14_runtime 14.40.33810 hcc2c482_22 conda-forge/win-64 Cached + vs2015_runtime 14.40.33810 h3bf8584_22 conda-forge/win-64 Cached + wheel 0.44.0 pyhd8ed1ab_0 conda-forge/noarch Cached + x264 1!164.3095 h8ffe710_2 conda-forge/win-64 Cached + x265 3.5 h2d74725_3 conda-forge/win-64 Cached + xorg-libxau 1.0.11 hcd874cb_0 conda-forge/win-64 Cached + xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge/win-64 Cached + xz 5.2.6 h8d14728_0 conda-forge/win-64 Cached + yaml 0.2.5 h8ffe710_2 conda-forge/win-64 Cached + zeromq 4.3.5 ha9f60a1_6 conda-forge/win-64 3MB + zfp 1.0.1 he0c23c2_2 conda-forge/win-64 235kB + zipp 3.20.2 pyhd8ed1ab_0 conda-forge/noarch Cached + zlib 1.3.1 h2466b09_2 conda-forge/win-64 107kB + zlib-ng 2.2.2 he0c23c2_0 conda-forge/win-64 109kB + zstd 1.5.6 h0ea2cb4_0 conda-forge/win-64 Cached Summary: Install: 216 packages Total download: 634MB ```

pip freeze

```bash (sleap_1.4.1a3_py310) λ pip freeze absl-py==2.1.0 albucore @ file:///home/conda/feedstock_root/build_artifacts/albucore_1727019653300/work albumentations @ file:///home/conda/feedstock_root/build_artifacts/albumentations_1727025741991/work annotated-types @ file:///home/conda/feedstock_root/build_artifacts/annotated-types_1716290248287/work astunparse==1.6.3 attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1722977137225/work cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work cachetools==5.5.0 cattrs @ file:///home/conda/feedstock_root/build_artifacts/cattrs_1727088879957/work certifi @ file:///home/conda/feedstock_root/build_artifacts/certifi_1725278078093/work/certifi charset-normalizer==3.3.2 contourpy @ file:///D:/bld/contourpy_1727293672566/work cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1696677705766/work debugpy==1.6.6 efficientnet==1.0.0 eval_type_backport @ file:///home/conda/feedstock_root/build_artifacts/eval_type_backport_1724150075856/work exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1720869315914/work flatbuffers==1.12 fonttools @ file:///D:/bld/fonttools_1727206449729/work gast==0.4.0 google-auth==2.35.0 google-auth-oauthlib==0.4.6 google-pasta==0.2.0 grpcio==1.66.2 h5py @ file:///D:/bld/h5py_1717664858971/work hdmf @ file:///D:/bld/hdmf_1725567417856/work idna==3.10 image-classifiers==1.0.0 imagecodecs @ file:///D:/bld/imagecodecs_1728267409164/work imageio @ file:///home/conda/feedstock_root/build_artifacts/imageio_1724069053555/work imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1717461632069/work imgstore==0.2.9 importlib_metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1726082825846/work importlib_resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1725921340658/work ipykernel==6.21.2 joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1714665484399/work jsmin @ file:///home/conda/feedstock_root/build_artifacts/jsmin_1642532731678/work jsonpickle==1.4.1 jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema_1720529478715/work jsonschema-specifications @ file:///tmp/tmpvslgxhz5/src jupyter_core==5.2.0 keras==2.9.0 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.2 kiwisolver @ file:///D:/bld/kiwisolver_1725459382062/work lazy_loader @ file:///home/conda/feedstock_root/build_artifacts/lazy-loader_1723774329602/work libclang==18.1.1 Markdown==3.7 markdown-it-py @ file:///home/conda/feedstock_root/build_artifacts/markdown-it-py_1686175045316/work MarkupSafe==2.1.5 matplotlib==3.9.2 mdurl @ file:///home/conda/feedstock_root/build_artifacts/mdurl_1704317613764/work munkres==1.1.4 ndx-pose @ file:///home/conda/feedstock_root/build_artifacts/ndx-pose_1706810229855/work nest-asyncio==1.5.6 networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1712540363324/work nixio==1.5.3 numpy @ file:///D:/bld/numpy_1707225570061/work/dist/numpy-1.26.4-cp310-cp310-win_amd64.whl#sha256=6761da75b1528684e6bf4dabdbdded9d1eb4d0e9b299482c7ce152cfb3155106 oauthlib==3.2.2 opencv-python==4.10.0 opencv-python-headless==4.10.0 opt_einsum==3.4.0 packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1718189413536/work pandas @ file:///D:/bld/pandas_1726878561601/work patsy @ file:///home/conda/feedstock_root/build_artifacts/patsy_1704469236901/work pillow @ file:///D:/bld/pillow_1726075253811/work pkgutil_resolve_name @ file:///home/conda/feedstock_root/build_artifacts/pkgutil-resolve-name_1694617248815/work protobuf==3.19.6 psutil==5.9.4 pyasn1==0.6.1 pyasn1_modules==0.4.1 pydantic @ file:///home/conda/feedstock_root/build_artifacts/pydantic_1726601062926/work pydantic_core @ file:///D:/bld/pydantic-core_1726525117142/work Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1714846767233/work pykalman @ file:///home/conda/feedstock_root/build_artifacts/pykalman_1711547707628/work pynwb @ file:///D:/bld/pynwb_1725927547197/work pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1724616129934/work PySide6==6.7.3 python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1709299778482/work python-rapidjson @ file:///D:/bld/python-rapidjson_1722901949842/work pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1706886791323/work PyWavelets==1.7.0 pywin32==305 PyYAML @ file:///D:/bld/pyyaml_1725456311802/work pyzmq==25.0.0 qimage2ndarray==1.10.0 QtPy @ file:///home/conda/feedstock_root/build_artifacts/qtpy_1698112029416/work qudida @ file:///home/conda/feedstock_root/build_artifacts/qudida_1651101164121/work referencing @ file:///home/conda/feedstock_root/build_artifacts/referencing_1714619483868/work requests==2.32.3 requests-oauthlib==2.0.0 rich @ file:///home/conda/feedstock_root/build_artifacts/rich_1728057819683/work/dist rpds-py @ file:///D:/bld/rpds-py_1725327161963/work rsa==4.9 ruamel.yaml @ file:///D:/bld/ruamel.yaml_1707298240950/work ruamel.yaml.clib @ file:///D:/bld/ruamel.yaml.clib_1707314694548/work scikit-image @ file:///D:/bld/scikit-image_1723842305941/work scikit-learn @ file:///D:/bld/scikit-learn_1726082855864/work/dist/scikit_learn-1.5.2-cp310-cp310-win_amd64.whl#sha256=f8a2b0c9a97f54c5c064931b1b27e9443957ccd063bc8c437288596a61b2bd5d scipy @ file:///C:/bld/scipy-split_1724327194933/work/dist/scipy-1.14.1-cp310-cp310-win_amd64.whl#sha256=e6bdc831fca55b320340d9d65555e680654f474b6605e599cfd60b4c00f7d5d6 seaborn @ file:///home/conda/feedstock_root/build_artifacts/seaborn-split_1714494649443/work segmentation-models==1.0.1 shiboken6==6.7.3 six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work sleap==1.4.1a3 statsmodels @ file:///D:/bld/statsmodels_1727986825006/work tensorboard==2.9.1 tensorboard-data-server==0.6.1 tensorboard-plugin-wit==1.8.1 tensorflow==2.9.2 tensorflow-estimator==2.9.0 tensorflow-hub @ file:///home/conda/feedstock_root/build_artifacts/tensorflow-hub_1618768305670/work/wheel_dir/tensorflow_hub-0.12.0-py2.py3-none-any.whl tensorflow-io-gcs-filesystem==0.31.0 termcolor==2.4.0 threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1714400101435/work tifffile @ file:///home/conda/feedstock_root/build_artifacts/tifffile_1727250434425/work tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1727974628237/work tornado==6.2 typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1717802530399/work tzdata @ file:///home/conda/feedstock_root/build_artifacts/python-tzdata_1727140567071/work tzlocal==5.2 unicodedata2 @ file:///D:/bld/unicodedata2_1695848155043/work urllib3==2.2.3 Werkzeug==3.0.4 wrapt==1.16.0 zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1726248574750/work ```

mamba list

```bash (sleap_1.4.1a3_py310) λ mamba list # packages in environment at C:\Users\TalmoLab\mambaforge\envs\sleap_1.4.1a3_py310: # # Name Version Build Channel _libavif_api 1.1.1 h57928b3_1 conda-forge absl-py 2.1.0 pypi_0 pypi albucore 0.0.16 pyhd8ed1ab_0 conda-forge albumentations 1.4.15 pyhd8ed1ab_0 conda-forge annotated-types 0.7.0 pyhd8ed1ab_0 conda-forge aom 3.9.1 he0c23c2_0 conda-forge astunparse 1.6.3 pypi_0 pypi attrs 24.2.0 pyh71513ae_0 conda-forge blosc 1.21.6 h85f69ea_0 conda-forge brotli 1.1.0 h2466b09_2 conda-forge brotli-bin 1.1.0 h2466b09_2 conda-forge bzip2 1.0.8 h2466b09_7 conda-forge c-blosc2 2.15.1 hb461149_0 conda-forge ca-certificates 2024.8.30 h56e8100_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.5.0 pypi_0 pypi cairo 1.18.0 h32b962e_3 conda-forge cattrs 24.1.2 pyhd8ed1ab_0 conda-forge certifi 2024.8.30 pyhd8ed1ab_0 conda-forge charls 2.4.2 h1537add_0 conda-forge charset-normalizer 3.3.2 pypi_0 pypi contourpy 1.3.0 py310hc19bc0b_2 conda-forge cuda-nvcc 11.3.58 hb8d16a4_0 nvidia cudatoolkit 11.3.1 hf2f0253_13 conda-forge cudnn 8.2.1.32 h754d62a_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge dav1d 1.2.1 hcfcfb64_0 conda-forge double-conversion 3.3.0 h63175ca_0 conda-forge efficientnet 1.0.0 pypi_0 pypi eval-type-backport 0.2.0 pyhd8ed1ab_0 conda-forge eval_type_backport 0.2.0 pyha770c72_0 conda-forge exceptiongroup 1.2.2 pyhd8ed1ab_0 conda-forge expat 2.6.3 he0c23c2_0 conda-forge ffmpeg 6.1.2 gpl_h0820249_105 conda-forge flatbuffers 1.12 pypi_0 pypi font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_3 conda-forge fontconfig 2.14.2 hbde0cde_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.54.1 py310ha8f682b_0 conda-forge freeglut 3.2.2 he0c23c2_3 conda-forge freetype 2.12.1 hdaf720e_2 conda-forge gast 0.4.0 pypi_0 pypi giflib 5.2.2 h64bf75a_0 conda-forge google-auth 2.35.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi graphite2 1.3.13 h63175ca_1003 conda-forge grpcio 1.66.2 pypi_0 pypi h5py 3.11.0 nompi_py310h2b0be38_102 conda-forge harfbuzz 9.0.0 h2bedf89_1 conda-forge hdf5 1.14.3 nompi_h2b43c12_105 conda-forge hdmf 3.14.4 pyh2e8e312_0 conda-forge icu 75.1 he0c23c2_0 conda-forge idna 3.10 pypi_0 pypi image-classifiers 1.0.0 pypi_0 pypi imagecodecs 2024.6.1 py310h0e2d205_5 conda-forge imageio 2.35.1 pyh12aca89_0 conda-forge imageio-ffmpeg 0.5.1 pyhd8ed1ab_0 conda-forge imath 3.1.12 hbb528cf_0 conda-forge imgstore 0.2.9 pypi_0 pypi importlib-metadata 8.5.0 pyha770c72_0 conda-forge importlib_metadata 8.5.0 hd8ed1ab_0 conda-forge importlib_resources 6.4.5 pyhd8ed1ab_0 conda-forge intel-openmp 2024.2.1 h57928b3_1083 conda-forge jasper 4.2.4 hcb1a123_0 conda-forge joblib 1.4.2 pyhd8ed1ab_0 conda-forge jsmin 3.0.1 pyhd8ed1ab_0 conda-forge jsonpickle 1.4.1 pyh9f0ad1d_0 conda-forge jsonschema 4.23.0 pyhd8ed1ab_0 conda-forge jsonschema-specifications 2024.10.1 pyhd8ed1ab_0 conda-forge jxrlib 1.1 hcfcfb64_3 conda-forge keras 2.9.0 pypi_0 pypi keras-applications 1.0.8 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi khronos-opencl-icd-loader 2024.05.08 hc70643c_0 conda-forge kiwisolver 1.4.7 py310hc19bc0b_0 conda-forge krb5 1.21.3 hdf4eb48_0 conda-forge lazy-loader 0.4 pyhd8ed1ab_1 conda-forge lazy_loader 0.4 pyhd8ed1ab_1 conda-forge lcms2 2.16 h67d730c_0 conda-forge lerc 4.0.0 h63175ca_0 conda-forge libabseil 20240722.0 cxx17_he0c23c2_1 conda-forge libaec 1.1.3 h63175ca_0 conda-forge libasprintf 0.22.5 h5728263_3 conda-forge libavif16 1.1.1 h4e96d62_1 conda-forge libblas 3.9.0 24_win64_mkl conda-forge libbrotlicommon 1.1.0 h2466b09_2 conda-forge libbrotlidec 1.1.0 h2466b09_2 conda-forge libbrotlienc 1.1.0 h2466b09_2 conda-forge libcblas 3.9.0 24_win64_mkl conda-forge libclang 18.1.1 pypi_0 pypi libclang13 19.1.1 default_ha5278ca_0 conda-forge libcurl 8.10.1 h1ee3ff0_0 conda-forge libdeflate 1.22 h2466b09_0 conda-forge libexpat 2.6.3 he0c23c2_0 conda-forge libffi 3.4.2 h8ffe710_5 conda-forge libgettextpo 0.22.5 h5728263_3 conda-forge libglib 2.82.1 h7025463_0 conda-forge libhwloc 2.11.1 default_h8125262_1000 conda-forge libiconv 1.17 hcfcfb64_2 conda-forge libintl 0.22.5 h5728263_3 conda-forge libjpeg-turbo 3.0.0 hcfcfb64_1 conda-forge liblapack 3.9.0 24_win64_mkl conda-forge liblapacke 3.9.0 24_win64_mkl conda-forge libopencv 4.10.0 qt6_py310h00b716a_607 conda-forge libopenvino 2024.4.0 hfe1841e_1 conda-forge libopenvino-auto-batch-plugin 2024.4.0 h04f32e0_1 conda-forge libopenvino-auto-plugin 2024.4.0 h04f32e0_1 conda-forge libopenvino-hetero-plugin 2024.4.0 h372dad0_1 conda-forge libopenvino-intel-cpu-plugin 2024.4.0 hfe1841e_1 conda-forge libopenvino-intel-gpu-plugin 2024.4.0 hfe1841e_1 conda-forge libopenvino-ir-frontend 2024.4.0 h372dad0_1 conda-forge libopenvino-onnx-frontend 2024.4.0 h5707d70_1 conda-forge libopenvino-paddle-frontend 2024.4.0 h5707d70_1 conda-forge libopenvino-pytorch-frontend 2024.4.0 he0c23c2_1 conda-forge libopenvino-tensorflow-frontend 2024.4.0 hf4e5e90_1 conda-forge libopenvino-tensorflow-lite-frontend 2024.4.0 he0c23c2_1 conda-forge libopus 1.3.1 h8ffe710_1 conda-forge libpng 1.6.44 h3ca93ac_0 conda-forge libprotobuf 5.27.5 hcaed137_2 conda-forge libsodium 1.0.20 hc70643c_0 conda-forge libsqlite 3.46.1 h2466b09_0 conda-forge libssh2 1.11.0 h7dfc565_0 conda-forge libtiff 4.7.0 hfc51747_1 conda-forge libwebp-base 1.4.0 hcfcfb64_0 conda-forge libxcb 1.16 h013a479_1 conda-forge libxml2 2.12.7 h0f24e4e_4 conda-forge libxslt 1.1.39 h3df6e99_0 conda-forge libzlib 1.3.1 h2466b09_2 conda-forge libzopfli 1.0.3 h0e60522_0 conda-forge lz4-c 1.9.4 hcfcfb64_0 conda-forge m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markdown 3.7 pypi_0 pypi markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.5 pypi_0 pypi matplotlib-base 3.9.2 py310h37e0a56_1 conda-forge mdurl 0.1.2 pyhd8ed1ab_0 conda-forge mkl 2024.1.0 h66d3029_694 conda-forge msys2-conda-epoch 20160418 1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge ndx-pose 0.1.1 pyhd8ed1ab_0 conda-forge networkx 3.3 pyhd8ed1ab_1 conda-forge nixio 1.5.3 pypi_0 pypi numpy 1.26.4 py310hf667824_0 conda-forge oauthlib 3.2.2 pypi_0 pypi opencv 4.10.0 qt6_py310h896e0ad_607 conda-forge openexr 3.2.2 h9aba623_2 conda-forge openh264 2.4.1 h63175ca_0 conda-forge openjpeg 2.5.2 h3d672ee_0 conda-forge openssl 3.3.2 h2466b09_0 conda-forge opt-einsum 3.4.0 pypi_0 pypi packaging 24.1 pyhd8ed1ab_0 conda-forge pandas 2.2.3 py310hb4db72f_1 conda-forge patsy 0.5.6 pyhd8ed1ab_0 conda-forge pcre2 10.44 h3d7b363_2 conda-forge pillow 10.4.0 py310h3e38d90_1 conda-forge pip 24.2 pyh8b19718_1 conda-forge pixman 0.43.4 h63175ca_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge protobuf 3.19.6 pypi_0 pypi psutil 6.0.0 py310ha8f682b_1 conda-forge pthread-stubs 0.4 hcd874cb_1001 conda-forge pthreads-win32 2.9.1 h2466b09_4 conda-forge pugixml 1.14 h63175ca_0 conda-forge py-opencv 4.10.0 qt6_py310h5f8bd55_607 conda-forge pyasn1 0.6.1 pypi_0 pypi pyasn1-modules 0.4.1 pypi_0 pypi pydantic 2.9.2 pyhd8ed1ab_0 conda-forge pydantic-core 2.23.4 py310hc226416_0 conda-forge pygments 2.18.0 pyhd8ed1ab_0 conda-forge pykalman 0.9.7 pyhd8ed1ab_0 conda-forge pynwb 2.8.2 pyh2e8e312_0 conda-forge pyparsing 3.1.4 pyhd8ed1ab_0 conda-forge pyside6 6.7.3 py310h60c6385_1 conda-forge python 3.10.15 hfaddaf0_1_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-rapidjson 1.20 py310h9e98ed7_0 conda-forge python-tzdata 2024.2 pyhd8ed1ab_0 conda-forge python_abi 3.10 5_cp310 conda-forge pytz 2024.1 pyhd8ed1ab_0 conda-forge pywavelets 1.7.0 py310hb0944cc_1 conda-forge pyyaml 6.0.2 py310ha8f682b_1 conda-forge pyzmq 26.2.0 py310h656833d_2 conda-forge qhull 2020.2 hc790b64_5 conda-forge qimage2ndarray 1.10.0 pypi_0 pypi qt6-main 6.7.3 hfb098fa_1 conda-forge qtpy 2.4.1 pyhd8ed1ab_0 conda-forge qudida 0.0.4 pyhd8ed1ab_0 conda-forge rav1e 0.6.6 h975169c_2 conda-forge referencing 0.35.1 pyhd8ed1ab_0 conda-forge requests 2.32.3 pypi_0 pypi requests-oauthlib 2.0.0 pypi_0 pypi rich 13.9.2 pyhd8ed1ab_0 conda-forge rpds-py 0.20.0 py310hc226416_1 conda-forge ruamel.yaml 0.18.6 py310h8d17308_0 conda-forge ruamel.yaml.clib 0.2.8 py310h8d17308_0 conda-forge scikit-image 0.24.0 py310hb4db72f_2 conda-forge scikit-learn 1.5.2 py310hf2a6c47_1 conda-forge scipy 1.14.1 py310h46043a1_0 conda-forge seaborn 0.13.2 hd8ed1ab_2 conda-forge seaborn-base 0.13.2 pyhd8ed1ab_2 conda-forge segmentation-models 1.0.1 pypi_0 pypi setuptools 75.1.0 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sleap 1.4.1a3 pypi_0 pypi snappy 1.2.1 h23299a8_0 conda-forge statsmodels 0.14.4 py310hb0944cc_0 conda-forge svt-av1 2.2.1 he0c23c2_0 conda-forge tbb 2021.13.0 hc790b64_0 conda-forge tensorboard 2.9.1 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorflow 2.9.2 pypi_0 pypi tensorflow-estimator 2.9.0 pypi_0 pypi tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge tensorflow-io-gcs-filesystem 0.31.0 pypi_0 pypi termcolor 2.4.0 pypi_0 pypi threadpoolctl 3.5.0 pyhc1e730c_0 conda-forge tifffile 2024.9.20 pyhd8ed1ab_0 conda-forge tk 8.6.13 h5226925_1 conda-forge tomli 2.0.2 pyhd8ed1ab_0 conda-forge typing-extensions 4.12.2 hd8ed1ab_0 conda-forge typing_extensions 4.12.2 pyha770c72_0 conda-forge tzdata 2024b hc8b5060_0 conda-forge tzlocal 5.2 pypi_0 pypi ucrt 10.0.22621.0 h57928b3_1 conda-forge unicodedata2 15.1.0 py310h8d17308_0 conda-forge urllib3 2.2.3 pypi_0 pypi vc 14.3 h8a93ad2_22 conda-forge vc14_runtime 14.40.33810 hcc2c482_22 conda-forge vs2015_runtime 14.40.33810 h3bf8584_22 conda-forge werkzeug 3.0.4 pypi_0 pypi wheel 0.44.0 pyhd8ed1ab_0 conda-forge wrapt 1.16.0 pypi_0 pypi x264 1!164.3095 h8ffe710_2 conda-forge x265 3.5 h2d74725_3 conda-forge xorg-libxau 1.0.11 hcd874cb_0 conda-forge xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge xz 5.2.6 h8d14728_0 conda-forge yaml 0.2.5 h8ffe710_2 conda-forge zeromq 4.3.5 ha9f60a1_6 conda-forge zfp 1.0.1 he0c23c2_2 conda-forge zipp 3.20.2 pyhd8ed1ab_0 conda-forge zlib 1.3.1 h2466b09_2 conda-forge zlib-ng 2.2.2 he0c23c2_0 conda-forge zstd 1.5.6 h0ea2cb4_0 conda-forge ```

roomrys commented 1 month ago

Template (manual test)

Backwards compatibility

#### pip freeze ```bash ``` #### mamba list ```bash ```

Training/Inference via GUI

```bash ```

Training Dialog

```bash ```

sleap-label

```bash ```

Installation

```bash ```

pip freeze

```bash ```

mamba list

```bash ```

roomrys commented 1 month ago

Mac (manual test)

Fails on training with leaked semaphore:

(sleap_1.4) liezlmaree:~$sleap-label
/Users/liezlmaree/micromamba/envs/sleap_1.4.1a3_py310/lib/python3.10/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 1.4.18 (you have 1.4.15). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1.
  check_for_updates()
Saving config: /Users/liezlmaree/.sleap/1.4.1a3/preferences.yaml
qt.qpa.drawing: Layer-backing is always enabled.  QT_MAC_WANTS_LAYER/_q_mac_wantsLayer has no effect.

Software versions:
SLEAP: 1.4.1a3
TensorFlow: 2.12.0
Numpy: 1.26.4
Python: 3.10.15
OS: macOS-13.5-arm64-arm-64bit

Happy SLEAPing! :)
qt.qpa.drawing: Layer-backing is always enabled.  QT_MAC_WANTS_LAYER/_q_mac_wantsLayer has no effect.
2024-10-10 11:24:20.787 python[190:5585487] +[CATransaction synchronize] called within transaction
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
While structuring TrainingJobConfig (1 sub-exception)
qt.qpa.drawing: Layer-backing is always enabled.  QT_MAC_WANTS_LAYER/_q_mac_wantsLayer has no effect.
qt.qpa.drawing: Layer-backing is always enabled.  QT_MAC_WANTS_LAYER/_q_mac_wantsLayer has no effect.
Resetting monitor window.
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/241010_112503.centroid.n=103/viz/validation.*.png
Start training centroid...
['sleap-train', '/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpdgvwrl6b/241010_112503_training_job.json', '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp', '--zmq', '--controller_port', '9000', '--publish_port', '9001', '--save_viz']
/Users/liezlmaree/micromamba/envs/sleap_1.4.1a3_py310/lib/python3.10/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 1.4.18 (you have 1.4.15). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1.
  check_for_updates()
INFO:sleap.nn.training:Versions:
SLEAP: 1.4.1a3
TensorFlow: 2.12.0
Numpy: 1.26.4
Python: 3.10.15
OS: macOS-13.5-arm64-arm-64bit
INFO:sleap.nn.training:Training labels file: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Training profile: /var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpdgvwrl6b/241010_112503_training_job.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
    "training_job_path": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpdgvwrl6b/241010_112503_training_job.json",
    "labels_path": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp",
    "video_paths": [
        ""
    ],
    "val_labels": null,
    "test_labels": null,
    "base_checkpoint": null,
    "tensorboard": false,
    "save_viz": true,
    "keep_viz": false,
    "zmq": true,
    "publish_port": 9001,
    "controller_port": 9000,
    "run_name": "",
    "prefix": "",
    "suffix": "",
    "cpu": false,
    "first_gpu": false,
    "last_gpu": false,
    "gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
    "data": {
        "labels": {
            "training_labels": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp",
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": [
                50,
                95,
                44,
                90,
                20,
                4,
                47,
                6,
                58,
                13,
                33,
                40,
                57,
                8,
                56,
                7,
                80,
                12,
                28,
                91,
                16,
                36,
                11,
                52,
                100,
                76,
                23,
                84,
                5,
                75,
                81,
                87,
                2,
                19,
                51,
                27,
                43,
                39,
                72,
                24,
                89,
                35,
                34,
                25,
                29,
                30,
                102,
                32,
                98,
                26,
                60,
                55,
                65,
                77,
                73,
                97,
                59,
                17,
                15,
                83,
                64,
                62,
                86,
                82,
                74,
                21,
                9,
                61,
                54,
                1,
                31,
                10,
                99,
                22,
                70,
                48,
                14,
                101,
                18,
                92,
                88,
                3,
                42,
                46,
                94,
                67,
                93,
                78,
                37,
                68,
                79,
                45,
                49
            ],
            "validation_inds": [
                69,
                66,
                0,
                71,
                85,
                53,
                38,
                63,
                96,
                41
            ],
            "test_inds": null,
            "search_path_hints": [
                "",
                "",
                "",
                "",
                "",
                ""
            ],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 0.5,
            "pad_to_stride": 16,
            "resize_and_pad_to_target": true,
            "target_height": 1024,
            "target_width": 1024
        },
        "instance_cropping": {
            "center_on_part": "thorax",
            "crop_size": null,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 2,
                "filters": 16,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": {
                "anchor_part": "thorax",
                "sigma": 2.5,
                "output_stride": 2,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "centered_instance": null,
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        },
        "base_checkpoint": null
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -180.0,
            "rotation_max_angle": 180.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": false
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": 200,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": 10,
        "min_val_batches_per_epoch": 10,
        "epochs": 2,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 20
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "241010_112503.centroid.n=103",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "keep_viz_images": false,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.3.4",
    "filename": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpdgvwrl6b/241010_112503_training_job.json"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU.
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
       Initialized: False
     Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 93 / Validation = 10.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
2024-10-10 11:25:11.493519: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
INFO:sleap.nn.training:Loaded test example. [0.964s]
INFO:sleap.nn.training:  Input shape: (512, 512, 1)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=16, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=3, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 16
INFO:sleap.nn.training:  Parameters: 1,953,105
INFO:sleap.nn.training:  Heads: 
INFO:sleap.nn.training:    [0] = CentroidConfmapsHead(anchor_part='thorax', sigma=2.5, output_stride=2, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs: 
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 256, 256, 1), dtype=tf.float32, name=None), name='CentroidConfmapsHead/BiasAdd:0', description="created by layer 'CentroidConfmapsHead'")
INFO:sleap.nn.training:Training from scratch
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 93
INFO:sleap.nn.training:Validation set: n = 10
INFO:sleap.nn.training:Setting up optimization...
WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.Adam` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.Adam`.
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=20)
WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., `tf.keras.optimizers.legacy.Adam`.
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:9000 (topic: )
INFO:sleap.nn.training:  ZMQ controller subcribed to: tcp://127.0.0.1:9000
INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set
INFO:sleap.nn.training:  ZMQ progress reporter publish on: tcp://127.0.0.1:9001
INFO:sleap.nn.training:Created run path: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/241010_112503.centroid.n=103
INFO:sleap.nn.training:Setting up visualization...
/Users/liezlmaree/micromamba/envs/sleap_1.4.1a3_py310/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
Run Path: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/241010_112503.centroid.n=103
qt.qpa.drawing: Layer-backing is always enabled.  QT_MAC_WANTS_LAYER/_q_mac_wantsLayer has no effect.
Resetting monitor window.

talmolab / sleap

Bump to 1.4.1a3 (py310) #1989

Description

Types of changes

Does this address any currently open issues?

Outside contributors checklist

Thank you for contributing to SLEAP!

Codecov Report

Windows (manual test)

Template (manual test)

Mac (manual test)