search

IBM / transition-amr-parser

SoTA Abstract Meaning Representation (AMR) parsing with word-node alignments in Pytorch. Includes checkpoints and other tools such as statistical significance Smatch.
Apache License 2.0
231 stars 46 forks source link

Question about "bash tests/minimal_test.sh" #15

Open Dionysusssss opened 2 years ago

Dionysusssss commented 2 years ago

"TypeError: 'NoneType' object is not subscriptable" when I run this instruction.

Here is the detail:

| Wrote preprocessed oracle data to DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24// | Wrote preprocessed embedding data to DATA/wiki25/embeddings/RoBERTa-large-top24 [Training:] [Configuration file:] configs/wiki25.sh | distributed init (rank 0): tcp://localhost:16604 | distributed init (rank 5): tcp://localhost:16604 | initialized host localhost as rank 5 | distributed init (rank 6): tcp://localhost:16604 | initialized host localhost as rank 6 | distributed init (rank 3): tcp://localhost:16604 | initialized host localhost as rank 3 | distributed init (rank 4): tcp://localhost:16604 | initialized host localhost as rank 4 | distributed init (rank 1): tcp://localhost:16604 | initialized host localhost as rank 1 | distributed init (rank 2): tcp://localhost:16604 | initialized host localhost as rank 2 | distributed init (rank 7): tcp://localhost:16604 | initialized host localhost as rank 7 | initialized host localhost as rank 0 Namespace(activation_dropout=0.0, activation_fn='relu', adam_betas='(0.9,0.98)', adam_eps=1e-08, adaptive_input=False, adaptive_softmax_cutoff=None, adaptive_softmax_dropout=0, append_eos_to_target=0, apply_tgt_actnode_masks=0, apply_tgt_input_src=0, apply_tgt_src_align=1, apply_tgt_vocab_masks=1, arch='transformer_tgt_pointer_graphmp', attention_dropout=0.0, bert_backprop=False, best_checkpoint_metric='loss', bpe=None, bucket_cap_mb=25, clip_norm=0.0, collate_tgt_states=1, cpu=False, criterion='label_smoothed_cross_entropy_pointer', curriculum=0, data='DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//', dataset_impl=None, ddp_backend='c10d', decoder_attention_heads=4, decoder_embed_dim=256, decoder_embed_path=None, decoder_ffn_embed_dim=512, decoder_input_dim=256, decoder_layers=6, decoder_learned_pos=False, decoder_normalize_before=False, decoder_output_dim=256, device_id=0, disable_validation=False, distributed_backend='nccl', distributed_init_method='tcp://localhost:16604', distributed_no_spawn=False, distributed_port=-1, distributed_rank=0, distributed_world_size=8, dropout=0.3, emb_dir='DATA/wiki25/embeddings/RoBERTa-large-top24', encode_state_machine=None, encoder_attention_heads=4, encoder_embed_dim=256, encoder_embed_path=None, encoder_ffn_embed_dim=512, encoder_layers=6, encoder_learned_pos=False, encoder_normalize_before=False, find_unused_parameters=False, fix_batches_to_gpus=False, fp16=False, fp16_init_scale=128, fp16_scale_tolerance=0.0, fp16_scale_window=None, keep_interval_updates=-1, keep_last_epochs=6, label_smoothing=0.01, lazy_load=False, left_pad_source='True', left_pad_target='False', log_format='json', log_interval=1000, loss_coef=1.0, lr=[0.0005], lr_scheduler='inverse_sqrt', max_epoch=10, max_sentences=None, max_sentences_valid=None, max_source_positions=1024, max_target_positions=1024, max_tokens=3584, max_tokens_valid=3584, max_update=0, maximize_best_checkpoint_metric=False, memory_efficient_fp16=False, min_loss_scale=0.0001, min_lr=1e-09, no_bert_precompute=False, no_epoch_checkpoints=False, no_last_checkpoints=False, no_progress_bar=False, no_save=False, no_save_optimizer_state=False, no_token_positional_embeddings=False, num_workers=1, optimizer='adam', optimizer_overrides='{}', pointer_dist_decoder_selfattn_avg=0, pointer_dist_decoder_selfattn_heads=1, pointer_dist_decoder_selfattn_infer=5, pointer_dist_decoder_selfattn_layers=[5], pretrained_embed_dim=1024, raw_text=False, required_batch_size_multiple=8, reset_dataloader=False, reset_lr_scheduler=False, reset_meters=False, reset_optimizer=False, restore_file='checkpoint_last.pt', save_dir='DATA/wiki25/models/exp_cofill_o8.3_act-states_RoBERTa-large-top24/_act-pos-grh_vmask1_shiftpos1_ptr-lay6-h1_grh-lay123-h2-allprev_1in1out_cam-layall-h2-abuf/ep10-seed42', save_interval=1, save_interval_updates=0, seed=42, sentence_avg=False, share_all_embeddings=False, share_decoder_input_output_embed=0, shift_pointer_value=1, skip_invalid_size_inputs_valid_test=False, source_lang=None, target_lang=None, task='amr_action_pointer_graphmp', tbmf_wrapper=False, tensorboard_logdir='DATA/wiki25/models/exp_cofill_o8.3_act-states_RoBERTa-large-top24/_act-pos-grh_vmask1_shiftpos1_ptr-lay6-h1_grh-lay123-h2-allprev_1in1out_cam-layall-h2-abuf/ep10-seed42', tgt_factored_emb_out=0, tgt_graph_heads=2, tgt_graph_layers=[0, 1, 2], tgt_graph_mask='allprev_1in1out', tgt_input_src_backprop=1, tgt_input_src_combine='add', tgt_input_src_emb='top', tgt_src_align_focus=['p0c1n0', 'p0c0n*'], tgt_src_align_heads=2, tgt_src_align_layers=[0, 1, 2, 3, 4, 5], threshold_loss_scale=None, tokenizer=None, train_subset='train', update_freq=[1], upsample_primary=1, use_bmuf=False, user_dir='../fairseq_ext', valid_subset='valid', validate_interval=1, warmup_init_lr=1e-07, warmup_updates=4000, weight_decay=0.0) | [en] dictionary: 248 types | [actions_nopos] dictionary: 128 types | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.en | loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/valid.en-actions.en.bert | loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/valid.en-actions.en.wordpieces | loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/valid.en-actions.en.wp2w | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.nopos_in | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.nopos_out | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.pos | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.vocab_masks | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.src_cursors | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actnode_masks | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_masks | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_1stnode_masks | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_cur_node_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_cur_1stnode_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_pre_node_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_directions | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_allpre_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_allpre_pre_node_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//valid.en-actions.actions.actedge_allpre_directions | model transformer_tgt_pointer_graphmp, criterion LabelSmoothedCrossEntropyPointerCriterion | num. model params: 8298496 (num. trained: 8298496) | training on 8 GPUs | max tokens per GPU = 3584 and max sentences per GPU = None | no existing checkpoint found DATA/wiki25/models/exp_cofill_o8.3_act-states_RoBERTa-large-top24/_act-pos-grh_vmask1_shiftpos1_ptr-lay6-h1_grh-lay123-h2-allprev_1in1out_cam-layall-h2-abuf/ep10-seed42/checkpoint_last.pt | loading train data for epoch 0 | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.en | loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/train.en-actions.en.bert | loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/train.en-actions.en.wordpieces | loaded 25 examples from: DATA/wiki25/embeddings/RoBERTa-large-top24/train.en-actions.en.wp2w | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.nopos_in | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.nopos_out | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.pos | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.vocab_masks | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.src_cursors | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actnode_masks | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_masks | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_1stnode_masks | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_cur_node_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_cur_1stnode_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_pre_node_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_directions | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_allpre_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_allpre_pre_node_indexes | loaded 25 examples from: DATA/wiki25/features/cofill_o8.3_act-states_RoBERTa-large-top24//train.en-actions.actions.actedge_allpre_directions /data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache)) /data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache)) /data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache)) /data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache)) /data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache)) /data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache)) /data4/yhchen/miniconda3/envs/AMR/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache)) Traceback (most recent call last): File "fairseq_ext/train.py", line 341, in cli_main() File "fairseq_ext/train.py", line 333, in cli_main nprocs=args.distributed_world_size,
File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn while not spawn_context.join(): File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception:

-- Process 5 terminated with the following error: Traceback (most recent call last): File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, args) File "/data4/yhchen/transition-amr-parser/fairseq_ext/train.py", line 297, in distributed_main main(args, init_distributed=True) # 分布式训练 File "/data4/yhchen/transition-amr-parser/fairseq_ext/train.py", line 103, in main train(args, trainer, task, epoch_itr) File "/data4/yhchen/transition-amr-parser/fairseq_ext/train.py", line 149, in train log_output = trainer.train_step(samples) File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/fairseq/trainer.py", line 264, in train_step ignore_grad File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/fairseq_ext/tasks/amr_action_pointer_graphmp.py", line 462, in train_step loss, sample_size, logging_output = criterion(model, sample) File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(input, kwargs) File "/data4/yhchen/miniconda3/envs/AMR/lib/python3.6/site-packages/fairseq_ext/criterions/label_smoothed_cross_entropy_pointer.py", line 104, in forward net_output = model(sample['net_input']) TypeError: 'NoneType' object is not subscriptable

robby927 commented 2 years ago

I have got the same error

ramon-astudillo commented 2 years ago 
TypeError: 'NoneType' object is not subscriptable

Normally indicates an error in fariseq when loading data. This may happen if e.g. feature extraction was canceled and there is some uncompleted feature extraction file.

Also all the multi-thread code is not ours. Could it be that you are trying to paralelize feature extraction and training? this would trigger that type of error.

ruchika2ar commented 2 years ago

Hi! Were you able to find a fix for this issue? I am facing the same error. Thank you!