Muennighoff
commented
3 years ago Hmm it looks like some of the data is incorrect, can you share the other commands you use & your CUDA version? (i.e. are you installing the correct req's?; Did you extract the features & cp into the folders?)
I just checked & for me it works fine, below is what it should look like:
task: [{'task': 'HM', 'num_choice': 1, 'annotations_jsonpath_train': './data/hm/train.jsonl', 'annotations_jsonpath_dev_seen': './data/hm/dev_seenlong.jsonl', 'annotations_jsonpath_traindev': './data/hm/traindev.jsonl', 'annotations_jsonpath_test_unseen': './data/hm/test_unseenlong.jsonl', 'annotations_jsonpath_test_seen': './data/hm/test_seenlong.jsonl', 'feature_lmdb_path': './data/hm/HM_img.tsv', 'gt_feature_lmdb_path': './data/hm/HM_gt_img.tsv', 'unisex_names_table': './data/vcr/unisex_names_table.csv', 'Proprocessor': 'PreprocessorBasic', 'tokenizer_name': 'FullTokenizer', 'fusion_method': 'mul', 'dropout_rate': 0.1, 'max_seq_len': 128, 'use_gt_fea': True, 'shufflekeep_across_task': True, 'shuffle_every_epoch': True, 'task_weight': 1.0, 'task_prefix': 'hm'}]
/opt/conda/lib/python3.7/site-packages/paddle/fluid/clip.py:779: UserWarning: Caution! 'set_gradient_clip' is not recommended and may be deprecated in future! We recommend a new strategy: set 'grad_clip' when initializing the 'optimizer'. This method can reduce the mistakes, please refer to documention of 'optimizer'.
warnings.warn("Caution! 'set_gradient_clip' is not recommended "
theoretical memory usage:
(18209.21138906479, 19076.31669330597, 'MB')
args.is_distributed: False
W0311 16:59:19.158905 3609 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 60, Driver API Version: 11.0, Runtime API Version: 9.0
W0311 16:59:19.171633 3609 device_context.cc:260] device: 0, cuDNN Version: 7.6.
Load pretraining parameters from ./data/erniesmall/params.
SPLIT: train
Start to load Faster-RCNN detected objects from ./data/hm/HM_img.tsv
Loaded 8596 images in file ./data/hm/HM_img.tsv in 142 seconds.
Start to load Faster-RCNN detected objects from ./data/hm/HM_gt_img.tsv
Loaded 8596 images in file ./data/hm/HM_gt_img.tsv in 104 seconds.
Load 8596 data from split(s) ./data/hm/train.jsonl.
use gt featurre
LEN: 8596
shuffle epoch 0
feed_queue size 30
epoch: 0, progress: 0/0, step: 10, loss: 0.651680, acc: 0.750000
steps: 10
save_steps: 1250
20210311 17:03:40 current learning_rate:0.00000018
used_time: 0.2308497428894043
feed_queue size 30
epoch: 0, progress: 0/0, step: 20, loss: 0.668125, acc: 0.375000
steps: 20
save_steps: 1250
20210311 17:03:42 current learning_rate:0.00000038
used_time: 0.19247126579284668
feed_queue size 30
epoch: 0, progress: 0/0, step: 30, loss: 0.637425, acc: 0.625000
steps: 30
sagarsj42
I'm trying to train ERNIE models by following the specified procedure. On starting off the finetune code by running any of the train commands, I'm getting an error from the py_reader() function. For instance, I'm trying to train ERNIE-Large using command
bash bash/training/ES/hm_ES36.sh. The following is a part of my console output: