Evaluation Metrics

[hedonistrh]

One-line summary Create a summary for evaluation metrics.

Reason We need to understand our networks can or can not capture-imitate the style of Turkish Music.

Design We already discussed some metrics at our first meeting. (I will add them as a comment in this issue)

• We will start with existing works to understand what they use to evaluate their generation. But, they are related with mostly western musics. So that, we also need some specific metrics for Turkish Music. (Some specific discriminator for makam, usul etc.)
• We will check open source libraries like mir-eval etc.
• The Folk RNN paper and @sertansenturk's master thesis can be helpful for us.

Acceptance Criteria

• Each of them should related with symbolic representation. (one-hot, piano-roll etc.)
• Each of metrics should be explained in a paragraph.
• If there is open source implementation of it, please indicate that.

Outcome We will have a list of evaluation metrics and their summaries.

[hedonistrh]

Some candidates for evaluation metrics:

• Note distribution
• A song’s length
• Long term structure (for instance AABA)
• A note-to-note transition
• Perplexity/Cross Entropy
• Makam Classification
• Usul Classification
  • User studies
• Note Distribution of the first section, second section etc.

[sertansenturk]

How many points is this ticket?

[hedonistrh]

Sorry for my mistake. :) I would like to assign 5 point, if it is fine for you.

[hedonistrh]

This paper can be helpful. I will investigate it.

• Qualified note rate(QN) computes the ratio of the number of the qualified notes (notes no shorter than three time steps, i.e., a 32th note) to the total number of notes. Low QN implies overly-fragmented music. Source

I will add some info about following:

• Note Prediction(Log-likelihood and accuracy)
• Distribution of Pitches
• Distribution of Number of Tokens
• Each section will end with a resolution
• Ending transcription on tonic
• Transition matrix of pitch and duration
• Check metric structure conserved or not
• Cross Entropy
• Mutual Information with Time

[sertansenturk]

@hedonistrh is there any work left to do from your side, or are you simply waiting for the reviews. If it's the latter, could you move the ticket to "Under review"?

[hedonistrh]

I was planning to add more metrics. But, I think those metrics will be enough at that point. So that, I move the ticket to Under Review :) @sertansenturk

[hedonistrh]

Now, we have our evaluation metrics. We will decide which ones are focused on at the next meeting.