search

dbiir / UER-py

Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo
https://github.com/dbiir/UER-py/wiki
Apache License 2.0
3.01k stars 525 forks source link

作者你好，请问small_config.json这个配置文件在哪呢 #357

Closed leiqing110 closed 1 year ago

leiqing110 commented 1 year ago

我想重新进行一下预训练 报错 找不到emb_size 参数设置 是不是config.json 不一致的问题呢 { 2 "architectures": [ 3 "T5ForConditionalGeneration" 4 ], 5 "d_ff": 3072, 6 "d_kv": 64, 7 "d_model": 768, 8 "decoder_start_token_id": 101, 9 "dropout_rate": 0.1, 10 "emb_size": 128, 11 "initializer_factor": 1.0, 12 "is_encoder_decoder": true, 13 "layer_norm_epsilon": 1e-06, 14 "model_type": "t5", 15 "n_positions": 512, 16 "num_heads": 12, 17 "num_layers": 12, 18 "pad_token_id": 0, 19 "relative_attention_num_buckets": 32, 20 "tokenizer_class": "BertTokenizer", 21 "vocab_size": 21228 22 } 我这是采用的t5-base-chinese-cluecorpussmall模型中的配置文件

leiqing110 commented 1 year ago

这是我的执行脚本

python pretrain.py --dataset_path ../t5/data/contrastive_learning_800w_filt32.pt \
                    --pretrained_model_path ../t5/model/t5-base-chinese-cluecorpussmall/pytorch_model.bin \
                    --vocab_path ../t5/model/t5-base-chinese-cluecorpussmall/vocab.txt \
                    --config_path ../t5/model/t5-base-chinese-cluecorpussmall/config.json \
                    --output_model_path ../t5/model/t5-contrastive_learning_800w_filt32_model.bin \
                    --world_size 1 --gpu_ranks 3 \
                    --total_steps 10000 --save_checkpoint_steps 5000 --report_steps 1000 \
                    --learning_rate 5e-4 --batch_size 64 \
                    --span_masking --span_geo_prob 0.3 --span_max_length 5