search

facebookresearch / audiocraft

Audiocraft is a library for audio processing and generation with deep learning. It features the state-of-the-art EnCodec audio compressor / tokenizer, along with MusicGen, a simple and controllable music generation LM with textual and melodic conditioning.
MIT License
20.71k stars 2.11k forks source link

CE/PPL values of the default MusicGen small/medium/large models #318

Open Saltb0xApps opened 11 months ago

Saltb0xApps commented 11 months ago

Hi! I've been retraining MusicGen with a custom dataset of 200,000 audio files (15,000 hours of music) and I wanted to know whats the CE/PPL values of the model trained by the facebook research team.

Here are my logs -

Train Summary | Epoch 1 | lr=1.95E-01 | grad_norm=3.172E+00 | grad_scale=65536.000 | ce=6.892 | ppl=1305.620 | duration=4041.047
Valid Summary | Epoch 1 | ce=6.651 | ppl=776.985 | duration=234.961
Train Summary | Epoch 2 | lr=5.86E-01 | grad_norm=1.809E+00 | grad_scale=112790.688 | ce=5.772 | ppl=334.071 | duration=4035.403
Valid Summary | Epoch 2 | ce=6.046 | ppl=426.098 | duration=235.961
Train Summary | Epoch 3 | lr=9.39E-01 | grad_norm=1.151E+00 | grad_scale=188934.875 | ce=4.777 | ppl=124.313 | duration=4042.014
Valid Summary | Epoch 3 | ce=4.992 | ppl=148.382 | duration=237.171
Train Summary | Epoch 4 | lr=1.00E+00 | grad_norm=7.085E-01 | grad_scale=304576.781 | ce=4.067 | ppl=59.522 | duration=4061.017
Valid Summary | Epoch 4 | ce=4.149 | ppl=63.918 | duration=237.357
Train Summary | Epoch 5 | lr=1.00E+00 | grad_norm=5.722E-01 | grad_scale=524288.000 | ce=3.673 | ppl=39.970 | duration=4074.356
Valid Summary | Epoch 5 | ce=3.660 | ppl=39.211 | duration=237.640
Train Summary | Epoch 6 | lr=1.00E+00 | grad_norm=5.055E-01 | grad_scale=986855.625 | ce=3.441 | ppl=31.519 | duration=4091.612
Valid Summary | Epoch 6 | ce=3.403 | ppl=30.300 | duration=238.214
Train Summary | Epoch 7 | lr=1.00E+00 | grad_norm=NAN | grad_scale=1083461.500 | ce=3.314 | ppl=27.733 | duration=4086.873
Valid Summary | Epoch 7 | ce=3.263 | ppl=26.396 | duration=237.573
Evaluate Summary | Epoch 7 | duration=0.002
Train Summary | Epoch 8 | lr=1.00E+00 | grad_norm=INF | grad_scale=670873.938 | ce=3.236 | ppl=25.644 | duration=4094.796
Valid Summary | Epoch 8 | ce=3.160 | ppl=23.822 | duration=241.191
Train Summary | Epoch 9 | lr=1.00E+00 | grad_norm=4.164E-01 | grad_scale=755404.062 | ce=3.177 | ppl=24.181 | duration=4070.976
Valid Summary | Epoch 9 | ce=3.105 | ppl=22.483 | duration=237.005
Train Summary | Epoch 10 | lr=1.00E+00 | grad_norm=4.020E-01 | grad_scale=1217636.250 | ce=3.130 | ppl=23.074 | duration=4057.217
Valid Summary | Epoch 10 | ce=3.052 | ppl=21.378 | duration=236.477
Generate Summary | Epoch 10 | rtf=0.970 | duration=98.467
Train Summary | Epoch 11 | lr=1.00E+00 | grad_norm=NAN | grad_scale=1449087.750 | ce=3.097 | ppl=22.332 | duration=4065.977
Valid Summary | Epoch 11 | ce=3.025 | ppl=20.781 | duration=237.109
Train Summary | Epoch 12 | lr=1.00E+00 | grad_norm=3.826E-01 | grad_scale=524288.000 | ce=3.065 | ppl=21.612 | duration=4052.587
Valid Summary | Epoch 12 | ce=2.994 | ppl=20.141 | duration=235.397
Train Summary | Epoch 13 | lr=9.99E-01 | grad_norm=3.742E-01 | grad_scale=998595.875 | ce=3.042 | ppl=21.117 | duration=4055.510
Valid Summary | Epoch 13 | ce=2.972 | ppl=19.725 | duration=236.145
Train Summary | Epoch 14 | lr=9.99E-01 | grad_norm=INF | grad_scale=626931.688 | ce=3.017 | ppl=20.608 | duration=4055.974
Valid Summary | Epoch 14 | ce=2.958 | ppl=19.413 | duration=237.876
Evaluate Summary | Epoch 14 | duration=0.002
Train Summary | Epoch 15 | lr=9.99E-01 | grad_norm=3.592E-01 | grad_scale=277406.375 | ce=3.001 | ppl=20.276 | duration=4039.137
Valid Summary | Epoch 15 | ce=2.942 | ppl=19.136 | duration=235.714
Train Summary | Epoch 16 | lr=9.99E-01 | grad_norm=3.557E-01 | grad_scale=524288.000 | ce=2.986 | ppl=19.962 | duration=4062.439
Valid Summary | Epoch 16 | ce=2.918 | ppl=18.674 | duration=238.201
Train Summary | Epoch 17 | lr=9.99E-01 | grad_norm=3.499E-01 | grad_scale=932514.812 | ce=2.971 | ppl=19.676 | duration=4049.428
Valid Summary | Epoch 17 | ce=2.910 | ppl=18.543 | duration=238.071
Train Summary | Epoch 18 | lr=9.99E-01 | grad_norm=NAN | grad_scale=1220990.625 | ce=2.957 | ppl=19.393 | duration=4054.944
Valid Summary | Epoch 18 | ce=2.894 | ppl=18.209 | duration=238.235
Train Summary | Epoch 19 | lr=9.98E-01 | grad_norm=3.430E-01 | grad_scale=1106271.125 | ce=2.946 | ppl=19.182 | duration=4056.341
Valid Summary | Epoch 19 | ce=2.875 | ppl=17.898 | duration=233.678
Train Summary | Epoch 20 | lr=9.98E-01 | grad_norm=NAN | grad_scale=1826118.875 | ce=2.933 | ppl=18.946 | duration=4053.734
Valid Summary | Epoch 20 | ce=2.878 | ppl=17.954 | duration=238.512
Generate Summary | Epoch 20 | rtf=0.967 | duration=97.612
Train Summary | Epoch 21 | lr=9.98E-01 | grad_norm=INF | grad_scale=1029120.625 | ce=2.925 | ppl=18.787 | duration=4057.326
Valid Summary | Epoch 21 | ce=2.870 | ppl=17.772 | duration=237.778
Evaluate Summary | Epoch 21 | duration=0.002
Train Summary | Epoch 22 | lr=9.98E-01 | grad_norm=INF | grad_scale=436906.656 | ce=2.915 | ppl=18.601 | duration=4051.297
Valid Summary | Epoch 22 | ce=2.837 | ppl=17.236 | duration=238.153
Train Summary | Epoch 23 | lr=9.98E-01 | grad_norm=3.351E-01 | grad_scale=276232.344 | ce=2.910 | ppl=18.508 | duration=4065.989
Valid Summary | Epoch 23 | ce=2.850 | ppl=17.448 | duration=237.814
Train Summary | Epoch 24 | lr=9.97E-01 | grad_norm=3.341E-01 | grad_scale=524288.000 | ce=2.899 | ppl=18.306 | duration=4061.380
Valid Summary | Epoch 24 | ce=2.818 | ppl=16.886 | duration=238.297
Train Summary | Epoch 25 | lr=9.97E-01 | grad_norm=3.338E-01 | grad_scale=930166.750 | ce=2.890 | ppl=18.146 | duration=4064.348
Valid Summary | Epoch 25 | ce=2.836 | ppl=17.224 | duration=237.584
Train Summary | Epoch 26 | lr=9.97E-01 | grad_norm=INF | grad_scale=1213611.000 | ce=2.885 | ppl=18.056 | duration=4055.300
Valid Summary | Epoch 26 | ce=2.818 | ppl=16.917 | duration=236.579
Train Summary | Epoch 27 | lr=9.97E-01 | grad_norm=3.324E-01 | grad_scale=1108954.625 | ce=2.880 | ppl=17.964 | duration=4055.740
Valid Summary | Epoch 27 | ce=2.811 | ppl=16.784 | duration=234.780
Train Summary | Epoch 28 | lr=9.96E-01 | grad_norm=INF | grad_scale=1057968.250 | ce=2.874 | ppl=17.854 | duration=4049.423
Valid Summary | Epoch 28 | ce=2.797 | ppl=16.551 | duration=235.946
Evaluate Summary | Epoch 28 | duration=0.003
Train Summary | Epoch 29 | lr=9.96E-01 | grad_norm=INF | grad_scale=588356.438 | ce=2.866 | ppl=17.717 | duration=4052.589
Valid Summary | Epoch 29 | ce=2.799 | ppl=16.577 | duration=236.729
Train Summary | Epoch 30 | lr=9.96E-01 | grad_norm=INF | grad_scale=680937.062 | ce=2.860 | ppl=17.596 | duration=4053.852
Valid Summary | Epoch 30 | ce=2.807 | ppl=16.725 | duration=239.560
Generate Summary | Epoch 30 | rtf=0.966 | duration=98.490
Train Summary | Epoch 31 | lr=9.95E-01 | grad_norm=3.350E-01 | grad_scale=262144.000 | ce=2.855 | ppl=17.519 | duration=4055.800
Valid Summary | Epoch 31 | ce=2.797 | ppl=16.533 | duration=237.001
Train Summary | Epoch 32 | lr=9.95E-01 | grad_norm=3.351E-01 | grad_scale=501646.031 | ce=2.853 | ppl=17.485 | duration=4058.609
Valid Summary | Epoch 32 | ce=2.777 | ppl=16.245 | duration=237.534
Train Summary | Epoch 33 | lr=9.95E-01 | grad_norm=INF | grad_scale=610495.312 | ce=2.847 | ppl=17.380 | duration=4069.220
Valid Summary | Epoch 33 | ce=2.778 | ppl=16.216 | duration=238.855
Train Summary | Epoch 34 | lr=9.94E-01 | grad_norm=INF | grad_scale=613514.250 | ce=2.843 | ppl=17.303 | duration=4066.966
Valid Summary | Epoch 34 | ce=2.772 | ppl=16.147 | duration=238.563
Train Summary | Epoch 35 | lr=9.94E-01 | grad_norm=3.370E-01 | grad_scale=524288.000 | ce=2.838 | ppl=17.217 | duration=4058.645
Valid Summary | Epoch 35 | ce=2.768 | ppl=16.079 | duration=239.987
Evaluate Summary | Epoch 35 | duration=0.003
Train Summary | Epoch 36 | lr=9.93E-01 | grad_norm=3.385E-01 | grad_scale=912388.562 | ce=2.836 | ppl=17.193 | duration=4059.485
Valid Summary | Epoch 36 | ce=2.766 | ppl=16.029 | duration=236.384
Train Summary | Epoch 37 | lr=9.93E-01 | grad_norm=INF | grad_scale=1074069.250 | ce=2.832 | ppl=17.110 | duration=4066.007
Valid Summary | Epoch 37 | ce=2.774 | ppl=16.147 | duration=236.824
Train Summary | Epoch 38 | lr=9.93E-01 | grad_norm=INF | grad_scale=1076081.875 | ce=2.830 | ppl=17.088 | duration=4074.932
Valid Summary | Epoch 38 | ce=2.761 | ppl=15.961 | duration=238.125
Train Summary | Epoch 39 | lr=9.92E-01 | grad_norm=INF | grad_scale=723537.562 | ce=2.825 | ppl=16.997 | duration=4073.309
Valid Summary | Epoch 39 | ce=2.764 | ppl=16.006 | duration=238.470
Train Summary | Epoch 40 | lr=9.92E-01 | grad_norm=3.433E-01 | grad_scale=702740.500 | ce=2.822 | ppl=16.953 | duration=4073.740
Valid Summary | Epoch 40 | ce=2.768 | ppl=16.067 | duration=237.359
Generate Summary | Epoch 40 | rtf=0.993 | duration=99.507
Train Summary | Epoch 41 | lr=9.91E-01 | grad_norm=NAN | grad_scale=821149.750 | ce=2.819 | ppl=16.900 | duration=4055.152
Valid Summary | Epoch 41 | ce=2.761 | ppl=15.970 | duration=239.916
Train Summary | Epoch 42 | lr=9.91E-01 | grad_norm=3.458E-01 | grad_scale=262144.000 | ce=2.816 | ppl=16.848 | duration=4066.595
Valid Summary | Epoch 42 | ce=2.737 | ppl=15.582 | duration=237.338
Evaluate Summary | Epoch 42 | duration=0.002
Train Summary | Epoch 43 | lr=9.90E-01 | grad_norm=3.487E-01 | grad_scale=479842.594 | ce=2.809 | ppl=16.730 | duration=4072.973
Valid Summary | Epoch 43 | ce=2.757 | ppl=15.915 | duration=239.928

I'm training a 420.37M param model which is focused on hip-hop music only and the output seems to be good so far, but it still has some ways to be comparable to the default model.

Mainly looking for PPL & CE values of the default small/medium/large model training.

ajayarora1235 commented 11 months ago

i have a feeling your learning rates might be too high? thought it would be 1e-4 or so.

RevolGMPHL commented 9 months ago

what is your training setting? it seems different with my training log. My ppl is around 30