roomrys
commented
1 year ago Still looking into this, but as a quick update, I ran a test in the GUI by creating a .pkg.slp and then running training/inference on it (mainly to double check the command line call) which seems correct:
Command line call:
sleap-track /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp --only-suggested-frames -m /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149 -m /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149 -o /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/predictions/courtship_labels.pkg.slp.231016_072541.predictions.slp --verbosity json --no-empty-frames
Started inference at: 2023-10-16 07:25:46.154717
Args:
{
│ 'data_path': '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp',
│ 'models': [
│ │ '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149',
│ │ '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149'
│ ],
│ 'frames': '',
│ 'only_labeled_frames': False,
│ 'only_suggested_frames': True,
2023-10-16 07:25:46.723577: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-10-16 07:25:46.723760: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
│ 'output': '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/predictions/courtship_labels.pkg.slp.231016_072541.predictions.slp',
│ 'no_empty_frames': True,
│ 'verbosity': 'json',
│ 'video.dataset': None,
│ 'video.input_format': 'channels_last',
│ 'video.index': '',
│ 'cpu': False,
│ 'first_gpu': False,
│ 'last_gpu': False,
│ 'gpu': 'auto',
│ 'max_edge_length_ratio': 0.25,
│ 'dist_penalty_weight': 1.0,
│ 'batch_size': 4,
│ 'open_in_gui': False,
│ 'peak_threshold': 0.2,
│ 'max_instances': None,
│ 'tracking.tracker': None,
│ 'tracking.max_tracking': None,
│ 'tracking.max_tracks': None,
│ 'tracking.target_instance_count': None,
2023-10-16 07:25:47.767238: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
│ 'tracking.pre_cull_to_target': None,
│ 'tracking.pre_cull_iou_threshold': None,
│ 'tracking.post_connect_single_breaks': None,
│ 'tracking.clean_instance_count': None,
│ 'tracking.clean_iou_threshold': None,
│ 'tracking.similarity': None,
│ 'tracking.match': None,
│ 'tracking.robust': None,
│ 'tracking.track_window': None,
│ 'tracking.min_new_track_points': None,
│ 'tracking.min_match_points': None,
│ 'tracking.img_scale': None,
│ 'tracking.of_window_size': None,
│ 'tracking.of_max_levels': None,
│ 'tracking.save_shifted_instances': None,
│ 'tracking.kf_node_indices': None,
│ 'tracking.kf_init_frame_count': None
}
INFO:sleap.nn.inference:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU.
Metal device set to: Apple M2 Pro
2023-10-16 07:25:49.822852: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-10-16 07:25:49.907230: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -45 } dim { size: -46 } dim { size: -47 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -48 } dim { size: -49 } dim { size: 1 } } }
2023-10-16 07:25:49.907533: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -56 } dim { size: -57 } dim { size: 3 } } }
2023-10-16 07:25:49.911132: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -91 } dim { size: -92 } dim { size: -93 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -20 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -20 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -20 } dim { size: -95 } dim { size: -96 } dim { size: 1 } } }
Versions:
SLEAP: 1.3.3
TensorFlow: 2.9.2
Numpy: 1.22.3
Python: 3.9.15
OS: macOS-13.5-arm64-arm-64bit
System:
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True
Finished inference at: 2023-10-16 07:25:50.748722
Total runtime: 4.5940141677856445 secs
Predicted frames: 20/20
Process return code: 0
skipped 98 redundant instances
olivier-cuttlefish
Hello, First of all, thank you so much for your amazing tool and the support you are providing. I am using v1.3.3. I ran my training on the clusters. Now, I would like to run inference on the labeled frames contained in the training package. The command I am running is
sleap-track -m "models/231013_164017.multi_instance" --only-labeled-frames -o "labels002_predictions.slp" "labels.v002.merged.pkg.slp"I also tried using --only-suggested-frames.
In both cases, it seems that sleap is trying to search for the original videos and pull the frames from there, while it should find them in the training package (as it was able to do so for the training).
Here is the traceback (/home/o/o-xxx/ is on the cluster, while /home/xxx/Documents/ are local paths):