search

amazon-science / transformer-gan

Apache License 2.0
50 stars 11 forks source link

Quantitative metrics in the paper #1

Open wzk1015 opened 2 years ago

wzk1015 commented 2 years ago

Hi, I try to reproduce your experiment result of Training Set in Table 1. I use PCU, ISR, PRS from muspy and APS, PR, TUP, IOI from mgeval. I run these metrics on the whole Maestro-V1 dataset (2369 songs), but four of them are somewhat different from the paper, so I wonder where is wrong of my implementation.

Metric Source Name Result in paper Result I reproduced
PCU muspy muspy.n_pitch_classes_used 7.81 11.98
PRS muspy muspy.polyphony_rate (threshold=3) 0.40 0.16
PR mgeval pitch_range 67.98 73.60
TUP mgeval total_used_pitch 65.76 72.27

I wonder if I am using the correct methods in mgeval and muspy? Or did you preprocess the data in some way?

Also, are you planning to release the code for evaluation in Table 1?

Thanks for your help!

li-car-fei commented 9 months ago

Hi, I try to reproduce your experiment result of Training Set in Table 1. I use PCU, ISR, PRS from muspy and APS, PR, TUP, IOI from mgeval. I run these metrics on the whole Maestro-V1 dataset (2369 songs), but four of them are somewhat different from the paper, so I wonder where is wrong of my implementation.

Metric Source Name Result in paper Result I reproduced PCU muspy muspy.n_pitch_classes_used 7.81 11.98 PRS muspy muspy.polyphony_rate (threshold=3) 0.40 0.16 PR mgeval pitch_range 67.98 73.60 TUP mgeval total_used_pitch 65.76 72.27 I wonder if I am using the correct methods in mgeval and muspy? Or did you preprocess the data in some way?

Also, are you planning to release the code for evaluation in Table 1?

Thanks for your help!

Please, have you solved this problem?

wzk1015 commented 9 months ago

Hi, I try to reproduce your experiment result of Training Set in Table 1. I use PCU, ISR, PRS from muspy and APS, PR, TUP, IOI from mgeval. I run these metrics on the whole Maestro-V1 dataset (2369 songs), but four of them are somewhat different from the paper, so I wonder where is wrong of my implementation. Metric Source Name Result in paper Result I reproduced PCU muspy muspy.n_pitch_classes_used 7.81 11.98 PRS muspy muspy.polyphony_rate (threshold=3) 0.40 0.16 PR mgeval pitch_range 67.98 73.60 TUP mgeval total_used_pitch 65.76 72.27 I wonder if I am using the correct methods in mgeval and muspy? Or did you preprocess the data in some way? Also, are you planning to release the code for evaluation in Table 1? Thanks for your help!

Please, have you solved this problem?

Actually no. Stop working on this project after that:)

li-car-fei commented 9 months ago

Hi, I try to reproduce your experiment result of Training Set in Table 1. I use PCU, ISR, PRS from muspy and APS, PR, TUP, IOI from mgeval. I run these metrics on the whole Maestro-V1 dataset (2369 songs), but four of them are somewhat different from the paper, so I wonder where is wrong of my implementation. Metric Source Name Result in paper Result I reproduced PCU muspy muspy.n_pitch_classes_used 7.81 11.98 PRS muspy muspy.polyphony_rate (threshold=3) 0.40 0.16 PR mgeval pitch_range 67.98 73.60 TUP mgeval total_used_pitch 65.76 72.27 I wonder if I am using the correct methods in mgeval and muspy? Or did you preprocess the data in some way? Also, are you planning to release the code for evaluation in Table 1? Thanks for your help!

Please, have you solved this problem?

Actually no. Stop working on this project after that:)

I tested with muspy library on other datasets. muspy library metrics calculations should be accurate.