01-ai / Yi

A series of large language models trained from scratch by developers @01-ai
https://01.ai
Apache License 2.0
7.6k stars 469 forks source link

增量预训练后,效果变得特别差 #406

Closed listwebit closed 6 months ago

listwebit commented 7 months ago

Reminder

Environment

- OS:
- Python:
- PyTorch:
- CUDA:
老师好,我用你们推荐的,https://github.com/hiyouga/LLaMA-Factory 做了增量预训练,但是效果特别差,能帮忙分析一下原因吗?
我用的Yi-34B 基础语言模型做的增量预训练,用了5G的医疗领域数据,5G wudao中的百科数据,用了3台机器做了pt预训练,然后用opencompass 分别对数据集ceval和cmmlu做了PPL测试,结果原始的Yi-34B模型效果很好,预训练之后的效果非常差劲,能帮忙分析一下原因吗?如果说是因为灾难遗忘导致的,但是掉点也不至于这么大吧,而且其中和医疗相关的测评,ceval-basic_medicine ,ceval-clinical_medicine也都掉点很多

这是LLaMA-Factory的启动参加配置:

deepspeed --hostfile=./hostfile --master_port=9901 src/train_bash.py
--deepspeed ./ds_config.json
--stage pt
--do_train
--model_name_or_path ../Yi-34B
--dataset input_test
--finetuning_type full
--lora_target q_proj,v_proj
--output_dir Yi-34B_output
--overwrite_cache
--per_device_train_batch_size 4
--gradient_accumulation_steps 4
--lr_scheduler_type cosine
--logging_steps 5
--save_steps 300
--learning_rate 5e-5
--num_train_epochs 1.0
--plot_loss
--bf16

这是YI-34B 原始的结果

dataset                                         version    metric    mode      yi-34b-hf
----------------------------------------------  ---------  --------  ------  -----------
cmmlu-agronomy                                  18e759     accuracy  ppl           79.88
cmmlu-anatomy                                   e2b12f     accuracy  ppl           91.89
cmmlu-ancient_chinese                           7faca5     accuracy  ppl           52.44
cmmlu-arts                                      cc4476     accuracy  ppl           96.25
cmmlu-astronomy                                 b81d60     accuracy  ppl           61.21
cmmlu-business_ethics                           5a294c     accuracy  ppl           73.68
cmmlu-chinese_civil_service_exam                d2f770     accuracy  ppl           85.62
cmmlu-chinese_driving_rule                      0e8a93     accuracy  ppl           99.24
cmmlu-chinese_food_culture                      8e2055     accuracy  ppl           84.56
cmmlu-chinese_foreign_policy                    484713     accuracy  ppl           85.98
cmmlu-chinese_history                           b336e0     accuracy  ppl           94.74
cmmlu-chinese_literature                        a9a252     accuracy  ppl           78.43
cmmlu-chinese_teacher_qualification             eacc26     accuracy  ppl           93.85
cmmlu-clinical_knowledge                        9993e7     accuracy  ppl           85.23
cmmlu-college_actuarial_science                 1f3eb3     accuracy  ppl           40.57
cmmlu-college_education                         7dbe6b     accuracy  ppl           97.2
cmmlu-college_engineering_hydrology             44f211     accuracy  ppl           84.91
cmmlu-college_law                               524735     accuracy  ppl           86.11
cmmlu-college_mathematics                       b94953     accuracy  ppl           35.24
cmmlu-college_medical_statistics                96aa86     accuracy  ppl           76.42
cmmlu-college_medicine                          5d2d59     accuracy  ppl           91.21
cmmlu-computer_science                          ca1243     accuracy  ppl           89.22
cmmlu-computer_security                         c21394     accuracy  ppl           94.74
cmmlu-conceptual_physics                        6dffab     accuracy  ppl           90.48
cmmlu-construction_project_management           4dd438     accuracy  ppl           76.26
cmmlu-economics                                 2a5014     accuracy  ppl           81.76
cmmlu-education                                 8fb812     accuracy  ppl           82.82
cmmlu-electrical_engineering                    c692a4     accuracy  ppl           84.3
cmmlu-elementary_chinese                        ecace0     accuracy  ppl           83.33
cmmlu-elementary_commonsense                    f0a3cc     accuracy  ppl           86.87
cmmlu-elementary_information_and_technology     edaf57     accuracy  ppl           93.28
cmmlu-elementary_mathematics                    099f4c     accuracy  ppl           56.09
cmmlu-ethnology                                 201b78     accuracy  ppl           90.37
cmmlu-food_science                              ff494e     accuracy  ppl           79.72
cmmlu-genetics                                  aa522b     accuracy  ppl           71.02
cmmlu-global_facts                              0e84cf     accuracy  ppl           92.62
cmmlu-high_school_biology                       4114ed     accuracy  ppl           88.76
cmmlu-high_school_chemistry                     540bcf     accuracy  ppl           78.79
cmmlu-high_school_geography                     70e849     accuracy  ppl           88.98
cmmlu-high_school_mathematics                   04285b     accuracy  ppl           52.44
cmmlu-high_school_physics                       d09967     accuracy  ppl           72.73
cmmlu-high_school_politics                      d047b7     accuracy  ppl           86.71
cmmlu-human_sexuality                           cc75c8     accuracy  ppl           79.37
cmmlu-international_law                         f420cb     accuracy  ppl           74.59
cmmlu-journalism                                5ef649     accuracy  ppl           80.23
cmmlu-jurisprudence                             6a38fc     accuracy  ppl           88.81
cmmlu-legal_and_moral_basis                     6c3e30     accuracy  ppl           97.66
cmmlu-logical                                   56a1ca     accuracy  ppl           74.8
cmmlu-machine_learning                          f26e28     accuracy  ppl           80.33
cmmlu-management                                ed634c     accuracy  ppl           94.29
cmmlu-marketing                                 acb874     accuracy  ppl           90.56
cmmlu-marxist_theory                            1fd3ba     accuracy  ppl           96.83
cmmlu-modern_chinese                            805e5f     accuracy  ppl           66.38
cmmlu-nutrition                                 49ad23     accuracy  ppl           87.59
cmmlu-philosophy                                71f30b     accuracy  ppl           88.57
cmmlu-professional_accounting                   f1c72e     accuracy  ppl           90.86
cmmlu-professional_law                          2651ae     accuracy  ppl           75.36
cmmlu-professional_medicine                     0a6df4     accuracy  ppl           87.77
cmmlu-professional_psychology                   d689f4     accuracy  ppl           90.09
cmmlu-public_relations                          1091db     accuracy  ppl           78.16
cmmlu-security_study                            69d6ff     accuracy  ppl           92.59
cmmlu-sociology                                 d58f6c     accuracy  ppl           81.86
cmmlu-sports_science                            e669b1     accuracy  ppl           84.24
cmmlu-traditional_chinese_medicine              60cd30     accuracy  ppl           86.49
cmmlu-virology                                  4ace32     accuracy  ppl           89.94
cmmlu-world_history                             b012c4     accuracy  ppl           93.17
cmmlu-world_religions                           c59782     accuracy  ppl           88.12
ceval-computer_network                          9b9417     accuracy  ppl           73.68
ceval-operating_system                          b2b8cf     accuracy  ppl           89.47
ceval-computer_architecture                     1bd275     accuracy  ppl           85.71
ceval-college_programming                       2d0833     accuracy  ppl           75.68
ceval-college_physics                           fb7e04     accuracy  ppl           68.42
ceval-college_chemistry                         916b7d     accuracy  ppl           79.17
ceval-advanced_mathematics                      5cad2a     accuracy  ppl           42.11
ceval-probability_and_statistics                a6b30e     accuracy  ppl           33.33
ceval-discrete_mathematics                      68be68     accuracy  ppl           25
ceval-electrical_engineer                       056c2e     accuracy  ppl           59.46
ceval-metrology_engineer                        4a757a     accuracy  ppl           83.33
ceval-high_school_mathematics                   a8ed21     accuracy  ppl           44.44
ceval-high_school_physics                       e1fc86     accuracy  ppl           84.21
ceval-high_school_chemistry                     9021c6     accuracy  ppl           84.21
ceval-high_school_biology                       c7f5a1     accuracy  ppl           89.47
ceval-middle_school_mathematics                 213989     accuracy  ppl           52.63
ceval-middle_school_biology                     ce0420     accuracy  ppl           90.48
ceval-middle_school_physics                     78f3af     accuracy  ppl           94.74
ceval-middle_school_chemistry                   d071d2     accuracy  ppl          100
ceval-veterinary_medicine                       cd3a07     accuracy  ppl           82.61
ceval-college_economics                         a35346     accuracy  ppl           65.45
ceval-business_administration                   69dd6a     accuracy  ppl           87.88
ceval-marxism                                   283ce0     accuracy  ppl           94.74
ceval-mao_zedong_thought                        f38cd1     accuracy  ppl          100
ceval-education_science                         fbd65c     accuracy  ppl           89.66
ceval-teacher_qualification                     c77f1f     accuracy  ppl           93.18
ceval-high_school_politics                      bbac37     accuracy  ppl           94.74
ceval-high_school_geography                     730a30     accuracy  ppl           89.47
ceval-middle_school_politics                    15b2d7     accuracy  ppl           90.48
ceval-middle_school_geography                   b00167     accuracy  ppl           91.67
ceval-modern_chinese_history                    5a04cd     accuracy  ppl           91.3
ceval-ideological_and_moral_cultivation         0829ff     accuracy  ppl          100
ceval-logic                                     c9c394     accuracy  ppl           95.45
ceval-law                                       cbd3c5     accuracy  ppl           79.17
ceval-chinese_language_and_literature           716ab3     accuracy  ppl           65.22
ceval-art_studies                               476114     accuracy  ppl           78.79
ceval-professional_tour_guide                   70f30f     accuracy  ppl           96.55
ceval-legal_professional                        f19cf5     accuracy  ppl           73.91
ceval-high_school_chinese                       931614     accuracy  ppl           78.95
ceval-high_school_history                       4d6364     accuracy  ppl          100
ceval-middle_school_history                     7f6356     accuracy  ppl          100
ceval-civil_servant                             a5dcb8     accuracy  ppl           76.6
ceval-sports_science                            192553     accuracy  ppl           89.47
ceval-plant_protection                          f7ff86     accuracy  ppl           90.91
ceval-basic_medicine                            a95a09     accuracy  ppl           94.74
ceval-clinical_medicine                         664b54     accuracy  ppl           90.91
ceval-urban_and_rural_planner                   fdae6f     accuracy  ppl           82.61
ceval-accountant                                d810a1     accuracy  ppl           85.71
ceval-fire_engineer                             bb924d     accuracy  ppl           80.65
ceval-environmental_impact_assessment_engineer  d59200     accuracy  ppl           80.65
ceval-tax_accountant                            9e16f2     accuracy  ppl           87.76
ceval-physician                                 0e90d5     accuracy  ppl           89.8

这是增量预训练的结果:


dataset                                         version    metric    mode      yi-34b-hf
----------------------------------------------  ---------  --------  ------  -----------
cmmlu-agronomy                                  18e759     accuracy  ppl           68.64
cmmlu-anatomy                                   e2b12f     accuracy  ppl           63.51
cmmlu-ancient_chinese                           7faca5     accuracy  ppl           37.8
cmmlu-arts                                      cc4476     accuracy  ppl           84.38
cmmlu-astronomy                                 b81d60     accuracy  ppl           49.09
cmmlu-business_ethics                           5a294c     accuracy  ppl           69.86
cmmlu-chinese_civil_service_exam                d2f770     accuracy  ppl           59.38
cmmlu-chinese_driving_rule                      0e8a93     accuracy  ppl           87.02
cmmlu-chinese_food_culture                      8e2055     accuracy  ppl           66.18
cmmlu-chinese_foreign_policy                    484713     accuracy  ppl           71.96
cmmlu-chinese_history                           b336e0     accuracy  ppl           77.09
cmmlu-chinese_literature                        a9a252     accuracy  ppl           58.33
cmmlu-chinese_teacher_qualification             eacc26     accuracy  ppl           83.8
cmmlu-clinical_knowledge                        9993e7     accuracy  ppl           62.45
cmmlu-college_actuarial_science                 1f3eb3     accuracy  ppl           26.42
cmmlu-college_education                         7dbe6b     accuracy  ppl           78.5
cmmlu-college_engineering_hydrology             44f211     accuracy  ppl           61.32
cmmlu-college_law                               524735     accuracy  ppl           57.41
cmmlu-college_mathematics                       b94953     accuracy  ppl           30.48
cmmlu-college_medical_statistics                96aa86     accuracy  ppl           61.32
cmmlu-college_medicine                          5d2d59     accuracy  ppl           69.23
cmmlu-computer_science                          ca1243     accuracy  ppl           72.06
cmmlu-computer_security                         c21394     accuracy  ppl           84.8
cmmlu-conceptual_physics                        6dffab     accuracy  ppl           70.07
cmmlu-construction_project_management           4dd438     accuracy  ppl           55.4
cmmlu-economics                                 2a5014     accuracy  ppl           70.44
cmmlu-education                                 8fb812     accuracy  ppl           76.07
cmmlu-electrical_engineering                    c692a4     accuracy  ppl           70.35
cmmlu-elementary_chinese                        ecace0     accuracy  ppl           60.71
cmmlu-elementary_commonsense                    f0a3cc     accuracy  ppl           69.19
cmmlu-elementary_information_and_technology     edaf57     accuracy  ppl           89.08
cmmlu-elementary_mathematics                    099f4c     accuracy  ppl           40.43
cmmlu-ethnology                                 201b78     accuracy  ppl           68.89
cmmlu-food_science                              ff494e     accuracy  ppl           65.03
cmmlu-genetics                                  aa522b     accuracy  ppl           57.95
cmmlu-global_facts                              0e84cf     accuracy  ppl           73.83
cmmlu-high_school_biology                       4114ed     accuracy  ppl           60.36
cmmlu-high_school_chemistry                     540bcf     accuracy  ppl           52.27
cmmlu-high_school_geography                     70e849     accuracy  ppl           79.66
cmmlu-high_school_mathematics                   04285b     accuracy  ppl           38.41
cmmlu-high_school_physics                       d09967     accuracy  ppl           45.45
cmmlu-high_school_politics                      d047b7     accuracy  ppl           66.43
cmmlu-human_sexuality                           cc75c8     accuracy  ppl           68.25
cmmlu-international_law                         f420cb     accuracy  ppl           53.51
cmmlu-journalism                                5ef649     accuracy  ppl           69.19
cmmlu-jurisprudence                             6a38fc     accuracy  ppl           72.26
cmmlu-legal_and_moral_basis                     6c3e30     accuracy  ppl           95.33
cmmlu-logical                                   56a1ca     accuracy  ppl           69.92
cmmlu-machine_learning                          f26e28     accuracy  ppl           64.75
cmmlu-management                                ed634c     accuracy  ppl           83.81
cmmlu-marketing                                 acb874     accuracy  ppl           83.89
cmmlu-marxist_theory                            1fd3ba     accuracy  ppl           85.19
cmmlu-modern_chinese                            805e5f     accuracy  ppl           50
cmmlu-nutrition                                 49ad23     accuracy  ppl           69.66
cmmlu-philosophy                                71f30b     accuracy  ppl           70.48
cmmlu-professional_accounting                   f1c72e     accuracy  ppl           76
cmmlu-professional_law                          2651ae     accuracy  ppl           52.13
cmmlu-professional_medicine                     0a6df4     accuracy  ppl           66.49
cmmlu-professional_psychology                   d689f4     accuracy  ppl           80.6
cmmlu-public_relations                          1091db     accuracy  ppl           67.82
cmmlu-security_study                            69d6ff     accuracy  ppl           77.04
cmmlu-sociology                                 d58f6c     accuracy  ppl           69.03
cmmlu-sports_science                            e669b1     accuracy  ppl           71.52
cmmlu-traditional_chinese_medicine              60cd30     accuracy  ppl           63.24
cmmlu-virology                                  4ace32     accuracy  ppl           76.33
cmmlu-world_history                             b012c4     accuracy  ppl           76.4
cmmlu-world_religions                           c59782     accuracy  ppl           76.88
ceval-computer_network                          9b9417     accuracy  ppl           57.89
ceval-operating_system                          b2b8cf     accuracy  ppl           73.68
ceval-computer_architecture                     1bd275     accuracy  ppl           71.43
ceval-college_programming                       2d0833     accuracy  ppl           78.38
ceval-college_physics                           fb7e04     accuracy  ppl           31.58
ceval-college_chemistry                         916b7d     accuracy  ppl           66.67
ceval-advanced_mathematics                      5cad2a     accuracy  ppl           52.63
ceval-probability_and_statistics                a6b30e     accuracy  ppl           16.67
ceval-discrete_mathematics                      68be68     accuracy  ppl           31.25
ceval-electrical_engineer                       056c2e     accuracy  ppl           48.65
ceval-metrology_engineer                        4a757a     accuracy  ppl           79.17
ceval-high_school_mathematics                   a8ed21     accuracy  ppl           27.78
ceval-high_school_physics                       e1fc86     accuracy  ppl           68.42
ceval-high_school_chemistry                     9021c6     accuracy  ppl           57.89
ceval-high_school_biology                       c7f5a1     accuracy  ppl           63.16
ceval-middle_school_mathematics                 213989     accuracy  ppl           52.63
ceval-middle_school_biology                     ce0420     accuracy  ppl           85.71
ceval-middle_school_physics                     78f3af     accuracy  ppl           78.95
ceval-middle_school_chemistry                   d071d2     accuracy  ppl           85
ceval-veterinary_medicine                       cd3a07     accuracy  ppl           73.91
ceval-college_economics                         a35346     accuracy  ppl           61.82
ceval-business_administration                   69dd6a     accuracy  ppl           66.67
ceval-marxism                                   283ce0     accuracy  ppl           68.42
ceval-mao_zedong_thought                        f38cd1     accuracy  ppl           91.67
ceval-education_science                         fbd65c     accuracy  ppl           72.41
ceval-teacher_qualification                     c77f1f     accuracy  ppl           88.64
ceval-high_school_politics                      bbac37     accuracy  ppl           84.21
ceval-high_school_geography                     730a30     accuracy  ppl           84.21
ceval-middle_school_politics                    15b2d7     accuracy  ppl           85.71
ceval-middle_school_geography                   b00167     accuracy  ppl           66.67
ceval-modern_chinese_history                    5a04cd     accuracy  ppl           82.61
ceval-ideological_and_moral_cultivation         0829ff     accuracy  ppl          100
ceval-logic                                     c9c394     accuracy  ppl           63.64
ceval-law                                       cbd3c5     accuracy  ppl           54.17
ceval-chinese_language_and_literature           716ab3     accuracy  ppl           52.17
ceval-art_studies                               476114     accuracy  ppl           78.79
ceval-professional_tour_guide                   70f30f     accuracy  ppl           82.76
ceval-legal_professional                        f19cf5     accuracy  ppl           60.87
ceval-high_school_chinese                       931614     accuracy  ppl           47.37
ceval-high_school_history                       4d6364     accuracy  ppl           75
ceval-middle_school_history                     7f6356     accuracy  ppl           77.27
ceval-civil_servant                             a5dcb8     accuracy  ppl           46.81
ceval-sports_science                            192553     accuracy  ppl           73.68
ceval-plant_protection                          f7ff86     accuracy  ppl           77.27
ceval-basic_medicine                            a95a09     accuracy  ppl           78.95
ceval-clinical_medicine                         664b54     accuracy  ppl           54.55
ceval-urban_and_rural_planner                   fdae6f     accuracy  ppl           69.57
ceval-accountant                                d810a1     accuracy  ppl           73.47
ceval-fire_engineer                             bb924d     accuracy  ppl           51.61
ceval-environmental_impact_assessment_engineer  d59200     accuracy  ppl           64.52
ceval-tax_accountant                            9e16f2     accuracy  ppl           59.18
ceval-physician                                 0e90d5     accuracy  ppl           75.51

### Current Behavior

是不是学习率最好和你们预训练结束时候的一致,然后继续预训练才能保持比较好的效果呢?能否说一下你们预训练最后的学习率是多少呢?谢谢各位大佬

### Expected Behavior

我用的三台A800 *8的机器,最后的学习率和loss日志如下:

{"current_steps": 5590, "total_steps": 5614, "loss": 1.4313, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 2.2546591544991837e-09, "epoch": 1.0, "percentage": 99.57, "elapsed_time": "5 days, 17:51:46", "remaining_time": "0:35:30"} {"current_steps": 5595, "total_steps": 5614, "loss": 1.4016, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 1.4130842386717025e-09, "epoch": 1.0, "percentage": 99.66, "elapsed_time": "5 days, 17:59:09", "remaining_time": "0:28:06"} {"current_steps": 5600, "total_steps": 5614, "loss": 1.406, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 7.672180148132757e-10, "epoch": 1.0, "percentage": 99.75, "elapsed_time": "5 days, 18:06:27", "remaining_time": "0:20:42"} {"current_steps": 5605, "total_steps": 5614, "loss": 1.4244, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 3.170655392792377e-10, "epoch": 1.0, "percentage": 99.84, "elapsed_time": "5 days, 18:13:48", "remaining_time": "0:13:19"} {"current_steps": 5610, "total_steps": 5614, "loss": 1.4334, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 6.263033621722869e-11, "epoch": 1.0, "percentage": 99.93, "elapsed_time": "5 days, 18:21:07", "remaining_time": "0:05:55"} {"current_steps": 5614, "total_steps": 5614, "loss": null, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.0, "percentage": 100.0, "elapsed_time": "5 days, 18:26:58", "remaining_time": "0:00:00"}



### Steps to Reproduce

环境什么的都是按照官方配置的

### Anything Else?

谢谢老师
listwebit commented 7 months ago

增量数据样例如下,医疗数据和通用数据做了shuff:

{"text": "肾上腺腺瘤的早期症状 通常都是轻到中度肥胖,典型的表现是满月脸、水牛背、颈部短粗等,四肢正常或偏瘦。\n2. 蛋白质代谢异常皮肤比较薄,腰、腹、股、腋窝等处有宽大紫纹,肌肉萎缩无力,>严重骨质疏松。\n3. 糖尿病多饮、多尿、多食,体重减轻。\n4. 性功能障碍男性出现性欲减退、阴茎勃起功能障碍、早泄等;女性出现闭经、月经紊乱或减少、多毛等。\n5. 精神异常轻度患者出现失眠、记忆力
减退、注意力不能集中的表现。中度患者出现忧郁、躁狂等。严重患者出现抑郁症甚至精神分裂症的症状。\n6. 高血压。\n3. 原发性醛固酮增多症有以下表现大部分原发性醛固酮增多症的患者,早期症状不典型>,仅出现高血压。随着病情发展,出现低钾血症。晚期病人的症状发生频率依次如下高血压>低钾血症>肢体麻木和肌肉无力>夜尿增多。\n1. 高血压血压会逐渐升高,可出现头痛、乏力、视物模糊等。\n2. 神经系
统功能障碍早期会出现肌肉麻木和隐隐作痛,之后会出现全身乏力、肌肉酸痛、下肢麻痹等。\n3. 肾脏表现多尿、夜尿增多。\n4. 心脏表现,部分患者会出现阵发性室上性心动过速,严重时会出现心室颤动,甚>至出现心功能衰竭"}
{"text": "编写一份人力资源管理手册,包含员工招聘、培训、福利和考核等方面的制度规定。 人力资源管理手册\n第一章 员工招聘\n1.1 招聘流程\n1.1.1 发布招聘广告\n1.1.2 筛选简历\n1.1.3 安排面试\n1.1.4 面试评分\n1.1.5 决定录用与否\n1.2 招聘标准\n1.2.1 岗位职责与描述\n1.2.2 经验要求\n1.2.3 学历要求\n1.2.4 技能要求\n1.2.5 素质要求\n第二章 培训\n2.1 培训流程\n2.1.1 识别培训需求\n2.1.2 制定培训计划\n2.1.3 筛选培训方式\n2.1.4 培训执行\n2.1.5 培训评估\n2.2 培训类型\n2.2.1 入职培训\n2.2.2 技能培训\n2.2.3 职业发展培训\n2.2.4 领导力培训\n2.2.5 员工福利培训\n第三章 福利\n3.1 薪酬制度\n3.1.1 填写薪酬调查问卷\n3.1.2 提出薪酬方案\n3.1.3 实施薪酬方案\n3.1.4 监测薪酬效果\n3.2 假期制度\n3.2.1 年假\n3.2.2 病假\n3.2.3 婚假\n3.2.4 产假\n3.2.5 陪产假\n3.2.6 哺乳假\n3.2.7 丧假\n3.2.8 探亲假\n3.2.9 调休\n3.3 保险福利\n3.3.1 社会保险\n3.3.2 商业保险\n3.3.3 公积金\n第四章 考核\n4.1 考核流程\n4.1.1 制定考核指标\n4.1.2 员工自我评估\n4.1.3 直属经理评估员工\n4.1.4 员工评估直属经理\n4.1.5 综合评估结果\n4.1.6 告知评估结果\n4.2 考核指标\n4.2.1 工作绩效\n4.2.2 工作态度\n4.2.3 专业技能\n4.2.4 团队意识\n4.2.5 学习能力\n4.2.6 创新能力\n4.3 奖惩措施\n4.3.1 优秀员工奖励\n4.3.2 差评员工警告\n4.3.3 不合格员工处理\n4.3.4 绩效考核带薪调整\n4.3.5 申诉机制\n以上为本公司人力资源管理手册规定,旨在维护企业和员工的权益。违反本手册规定者,将根据
相应的企业规定进行处罚。"}
{"text": "有一年多了,脸上痘坑里都是黑色的,不是很明显,会不会自动消除? 1. 使用含有果酸、维生素C等有效成分的护肤品,可以帮助淡化黑色痘坑。\n2. 定期进行去角质,可以促进新的皮肤细胞生长,>帮助淡化黑色痘坑。\n3. 使用面膜,如炭黑面膜,可以有效清洁毛孔,减少黑头和痘痘,从而改善黑色痘坑的外观。\n4. 如果黑色痘坑比较明显,可以考虑采取医学美容手段,如激光治疗、微针、微晶瓷等,以>达到更好的效果。总之,如果您想改善黑色痘坑的外观,需要耐心和坚持。同时,也要注意避免刺激皮肤,保持皮肤清洁和滋润。"}
{"text": "今天才开始涨乳,孩子已经快三个月了,吸奶器也吸不出来,孩子也吸不出来,该怎么办 1. 经常喂奶将孩子放在乳房上经常喂奶,这有助于减轻乳腺堵塞和疼痛。\n2. 使用热敷使用热毛巾或热水袋在
乳房上进行热敷,这有助于促进血液循环,减轻乳腺堵塞。\n3. 使用吸奶器尝试使用吸奶器进行吸奶,可以轻松地将过多的乳汁吸出来,减轻乳腺堵塞和疼痛。\n4. 按摩乳房使用按摩的方式,轻轻地按摩乳房,>促进乳汁的流动和排出,缓解乳腺堵塞和疼痛。如果以上方法都没有效果,建议咨询医生或产科护士的建议。他们会根据你的具体情况给出更具体的建议和治疗方案。"}
{"text": "西安哪个治淋巴瘤医院最好,能治好淋巴瘤吗,西安哪个治淋巴瘤医院最好,能治好淋巴瘤吗? 淋巴瘤能治好吗与病人的身体机能密切相关。身体机能好,免疫力强,才能抵抗癌肿的发展,耐受各种药
物治疗。因此,提高免疫机能,增强对肿瘤的抵抗力对恶性淋巴瘤患者极为重要。生物细胞治疗可以有效控制癌细胞转移扩散,能够增强机体免疫功能以达到抑制癌细胞生长,同时又不产生副作用,在治癌抗癌的>同时,增强机体免疫力,以达到完全战胜癌症的目的。"}
{"text": "炎性改变的临床表现有些什么? 低弱回声"}
{"text": "您好,我有一个问题想请教,我和我的爱人有三次胎停育,之前在河南省人民医院生殖中心做了很多检查,最后做了免疫治疗,治疗后的阴性会转阳性吗?现在备孕已经大于半年了,还是没有怀孕,请>
问需要怎么处理? 您好,根据您提供的情况,我能理解您的忧虑和焦虑。对于复发性流产的治疗,免疫治疗是其中一种选择,但是其适用范围很有限,因为组织相容性抗原(HLA)抗体与复发性流产之间的关系并>
不是十分明确。免疫治疗可能在一定程度上减轻因HLA抗体导致的反复失败,但是其疗效存在争议,而且很多医生也不推荐这种治疗方式。在您的情况下,如果免疫治疗已经进行过,并且出现了阴性,那么通常情况
下不会再次转为阳性。但是建议需要进行进一步的检查来确定是否需要继续进行免疫治疗。此外,您与您的爱人备孕超过半年了,建议考虑进行一些额外的检查,例如女方可以进行内分泌检查、排卵监测及输卵管>
造影等来确定是否存在相应的问题;男方则可以进行精液检查。若以上检查均没有发现明显的异常,则可以考虑寻求辅助生殖技术,例如人工授精或者体外受精等。总之,建议您先要对您的身体进行全面的检查,>
这样才能明确原因并选择最合适的治疗方式。希望我的回答对您有所帮助。"}{"text": "癫痫可以治疗痊愈吗 ,我们家的小孩由于以前比较顽皮,在一次和别人玩耍时不小心把头给打破了,后来这个是痊愈了,但是这次创伤给他带来了癫痫病的后遗症,前一段时间还发作了,很吓人,幸亏
我们及时发现,然后送医院了,但是我还怕它会再发作。 你好,癫痫病史可以通过治疗完全痊愈的,癫痫病的治疗周期比较长,而且痊愈的时间是和病人的病情严重程度有关系,需要及时关注病情,积极配合医生的治疗。,癫痫病患者在及时治疗之外,患者在生活中还需要注意要保持良好的心情,好的心情对疾病的恢复很有帮助,希望上述的答案可以帮助到你,谢谢!"} {"text": "阐明个人品牌建立的必要性和方法,并给出至少3个行之有效的建议。 个人品牌建立的必要性: \n1. 在职场竞争中脱颖而出:个人品牌可以帮助我们在职场竞争中脱颖而出,让我们更快地被他人认可>
和记住。\n2. 在社交媒体上建立专业形象:在社交媒体上建立一个专业、令人信任的形象,能够增加我们的个人可信度,并且对于求职等方面也有好处。\n3. 凸显自身价值:通过建立个人品牌,我们可以更好地>展示自身价值、经验和技能,从而获得更多的职位机会和收入。\n个人品牌建立的方法:\n1. 创建一个独特的品牌名:选择一个个人品牌名,最好能够简洁易记。\n2. 建立强大的线上和线下存在感:在社交媒体>上和现实生活中展示自己的品牌。比如定期发布有价值的文章、参加有关自己所在领域的活动等等。\n3. 打造个人形象:包括个人形象设计、网站设计、简历设计等等,要保持一致性和专业性。\n行之有效的建议
:\n1. 定期更新自己的社交媒体,分享有关自己所在领域的见解、经验和思考。\n2. 参加行业活动、交流会议,拓展自己的人际关系。\n3. 虚心学习、接收反馈,不断提升自身能力和品牌价值。"}
{"text": "双贝特的副作用(不良反应) 偶有恶心、腹胀、腹泻、嗜睡、无力、脱发、白细胞减少、皮疹、瘙痒、肌强直、肌痉挛、肌酸磷酸激酶及谷草转氨酶升高等。"}
{"text": "刘小平(武汉理工大学化学工程学院教师)。刘小平,武汉理工大学化学工程学院教师。刘小平基本资料 出生年月 1958年2月 学 位 医学学士 刘小平教育经历 1982年1月,毕业于湖北中医学院中药专
业 刘小平工作简历 1982.1-2002.4,湖北中医学院附属医院 2002.5 —— ,武汉理工大学化学工程学院制药工程系 刘小平研究领域 药物新剂型与新技术研究;中药活性成分与制剂研究;药物分析与药品质量标准 百度百科内容由网友共同编辑,如您发现自己的词条内容不准确或不完善,欢迎使用本人词条编辑服务(免费)参与修正。立即前往"}
markli404 commented 7 months ago

请问一下这个lr scheduler和batchsize是怎么设置的

listwebit commented 6 months ago

上面写的有,deepspeed --hostfile=./hostfile --master_port=9901 src/train_bash.py --deepspeed ./ds_config.json --stage pt --do_train --model_name_or_path ../Yi-34B --dataset input_test --finetuning_type full --lora_target q_proj,v_proj --output_dir Yi-34B_output --overwrite_cache --per_device_train_batch_size 4 --gradient_accumulation_steps 4 --lr_scheduler_type cosine --logging_steps 5 --save_steps 300 --learning_rate 5e-5 --num_train_epochs 1.0 --plot_loss --bf16

listwebit commented 6 months ago

上面写的有,deepspeed --hostfile=./hostfile --master_port=9901 src/train_bash.py --deepspeed ./ds_config.json --stage pt --do_train --model_name_or_path ../Yi-34B --dataset input_test --finetuning_type full --lora_target q_proj,v_proj --output_dir Yi-34B_output --overwrite_cache --per_device_train_batch_size 4 --gradient_accumulation_steps 4 --lr_scheduler_type cosine --logging_steps 5 --save_steps 300 --learning_rate 5e-5 --num_train_epochs 1.0 --plot_loss --bf16

Yimi81 commented 6 months ago

你增量预训练的数据集医疗数据和通用数据的配比是怎么样的,而且你评测的两个数据集都是考试相关的,两个是同质的。你能不能多测几个不同测试集。最关键的是,在你实际应用场景效果是否有提升,你有测试过吗?

nuoma commented 6 months ago

你好,你的问题已经在微信群里回答啦。 增量预训练是一件挺难的事情。预训练阶段的语料质量我们已经做了极其严格的把控,你的医疗语料很有可能在通用语料中出现过。增量训练的一些考量我们也在Yi-9B的公众号文章,以及TechReport(https://arxiv.org/abs/2403.04652)中提到过。祝好运