Open sh00kti opened 2 years ago
[2022-09-29 23:45:59,066] INFO [deepethogram.gui.main.project_loaded_buttons:175] Number finalized labels: 3 [2022-09-29 23:46:19,448] INFO [deepethogram.gui.main.get_selected_models:1136] {'flow_generator': {'no pretrained weights': None, '200221_115158_TinyMotionNet': '/home/anuj/Desktop/ananya/test_deepethogram/models/pretrained_models/200221_115158_TinyMotionNet/checkpoint.pt', '220929_172218_flow_generator_train': '/home/anuj/Desktop/ananya/test_deepethogram/models/220929_172218_flow_generator_train/lightning_checkpoints/epoch=2-step=3000.ckpt'}, 'feature_extractor': {'no pretrained weights': None, '200415_125824_hidden_two_stream_kinetics_degf': '/home/anuj/Desktop/ananya/test_deepethogram/models/pretrained_models/200415_125824_hidden_two_stream_kinetics_degf/checkpoint.pt', '220929_194952_feature_extractor_train': '/home/anuj/Desktop/ananya/test_deepethogram/models/220929_194952_feature_extractor_train/lightning_checkpoints/epoch=13-step=14000.ckpt'}, 'sequence': {'': None}} [2022-09-29 23:46:20,986] INFO [deepethogram.projects.convert_config_paths_to_absolute:1135] cwd in absolute: /home/anuj/Desktop/ananya/test_deepethogram/gui_logs/220929_172213 [2022-09-29 23:46:20,986] INFO [deepethogram.projects.convert_config_paths_to_absolute:1178] after absolute: {'class_names': ['background', '"Nose-to-nose', 'nose-to-bosy', 'body-to-body', 'chasing', 'anogenital"'], 'config_file': '/home/anuj/Desktop/ananya/test_deepethogram/project_config.yaml', 'data_path': '/home/anuj/Desktop/ananya/test_deepethogram/DATA', 'labeler': None, 'model_path': '/home/anuj/Desktop/ananya/test_deepethogram/models', 'name': 'test', 'path': '/home/anuj/Desktop/ananya/test_deepethogram', 'pretrained_path': '/home/anuj/Desktop/ananya/test_deepethogram/models/pretrained_models'} [2022-09-29 23:46:20,992] INFO [__main__.sequence_train:43] args: /home/anuj/deepethogram/deepethogram/sequence/train.py project.path=/home/anuj/Desktop/ananya/test_deepethogram [2022-09-29 23:46:20,992] INFO [__main__.sequence_train:49] Configuration used: [2022-09-29 23:46:20,997] INFO [__main__.sequence_train:50] split: reload: true file: null train_val_test: - 0.8 - 0.2 - 0.0 compute: fp16: false num_workers: 8 batch_size: 32 min_batch_size: 8 max_batch_size: 512 distributed: false gpu_id: 0 dali: false metrics_workers: 0 reload: overwrite_cfg: false latest: false notes: null log: level: info feature_extractor: arch: resnet18 fusion: average sampler: null final_bn: false sampling_ratio: null final_activation: sigmoid dropout_p: 0.25 n_flows: 10 n_rgb: 1 curriculum: false inputs: both weights: pretrained train: steps_per_epoch: train: 1000 val: 1000 test: 20 num_epochs: 100 lr: 0.0001 scheduler: plateau min_lr: 5.0e-07 stopping_type: learning_rate milestones: - 50 - 100 - 150 - 200 - 250 - 300 weight_loss: true patience: 5 early_stopping_begins: 0 viz_metrics: true viz_examples: 10 reduction_factor: 0.1 loss_weight_exp: 1.0 loss_gamma: 1.0 label_smoothing: 0.05 oversampling_exp: 0.0 regularization: style: l2 alpha: 0.01 beta: 0.001 sequence: arch: tgmj latent_name: resnet18 output_name: null sequence_length: 180 dropout_p: 0.5 num_layers: 3 hidden_size: 64 nonoverlapping: true filter_length: 15 input_dropout: 0.5 n_filters: 8 tgm_reduction: max c_in: 1 c_out: 8 soft_attn: true num_features: 128 bidirectional: false rnn_style: lstm hidden_dropout: 0.0 weights: null nonlinear_classification: true final_bn: true input_type: features augs: LR: 0.5 UD: 0.0 brightness: 0.25 contrast: 0.1 crop_size: null degrees: 10 grayscale: 0.5 hue: 0.1 normalization: 'N': 185030720 mean: - 0.6841030389917071 - 0.6841030389917071 - 0.6841030389917071 std: - 0.15482315274490271 - 0.15482315274490271 - 0.15482315274490271 pad: null random_resize: false resize: - 224 - 224 saturation: 0.1 cmap: deepethogram control_arrow_jump: 31 label_view_width: 31 postprocessor: min_bout_length: 1 type: min_bout_per_behavior prediction_opacity: 0.2 project: class_names: - background - '"Nose-to-nose' - nose-to-bosy - body-to-body - chasing - anogenital" config_file: /home/anuj/Desktop/ananya/test_deepethogram/project_config.yaml data_path: /home/anuj/Desktop/ananya/test_deepethogram/DATA labeler: null model_path: /home/anuj/Desktop/ananya/test_deepethogram/models name: test path: /home/anuj/Desktop/ananya/test_deepethogram pretrained_path: /home/anuj/Desktop/ananya/test_deepethogram/models/pretrained_models run: type: train model: sequence dir: /home/anuj/Desktop/ananya/test_deepethogram/models/220929_234620_sequence_train unlabeled_alpha: 0.1 vertical_arrow_jump: 3 Traceback (most recent call last): File "/home/anuj/miniconda3/envs/deg/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/anuj/miniconda3/envs/deg/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/anuj/deepethogram/deepethogram/sequence/train.py", line 265, in <module> sequence_train(cfg) File "/home/anuj/deepethogram/deepethogram/sequence/train.py", line 52, in sequence_train datasets, data_info = get_datasets_from_cfg(cfg, 'sequence') File "/home/anuj/deepethogram/deepethogram/data/datasets.py", line 1205, in get_datasets_from_cfg is_keypoint=cfg.sequence.input_type == 'keypoints') File "/home/anuj/deepethogram/deepethogram/data/datasets.py", line 1112, in get_sequence_datasets stack_in_time=stack_in_time) File "/home/anuj/deepethogram/deepethogram/data/datasets.py", line 784, in __init__ dataset = FeatureVectorDataset(datafile, labelfile, *args, **kwargs) File "/home/anuj/deepethogram/deepethogram/data/datasets.py", line 710, in __init__ self.verify_dataset() File "/home/anuj/deepethogram/deepethogram/data/datasets.py", line 726, in verify_dataset assert self.logit_key in f AssertionError [2022-09-29 23:46:21,466] INFO [deepethogram.gui.main.sequence_train:501] Training finished. If you see error messages above, training did not complete successfully. [2022-09-29 23:46:21,466] INFO [deepethogram.gui.main.sequence_train:506] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2022-09-29 23:46:21,495] INFO [deepethogram.gui.main.project_loaded_buttons:175] Number finalized labels: 3 [2022-09-29 23:47:59,878] INFO [deepethogram.gui.main.log_idle:151] User has been idle for 60.0 seconds... [2022-09-29 23:49:33,307] INFO [deepethogram.gui.main.user_did_something:138] User restarted labeling
sh00kti
commented
2 years ago sh00kti
commented
2 years ago