Script to Aggregate & Clean Dataset

[Ljferrer]

Rules:

• Assert detected language == English
• Skip songs with lines > 80 chars

Copy fields:

• [x] artist: " "
• [x] title: " "
• [x] year: "YYYY-MM-DD"
• [x] image: "url"
• [x] raw_lyrics: "..."

Clean lyrics fields:

• [x] sections: [ {type: [Header], lines: [ " ", ... ] } ]
  • Remove ad-libs (wut wut)

[Ljferrer]

Examples of dirty data

Over 80 chars:

• rawLyrics/lyrics_nickiminaj_goodluck.json (long) Full lyrics to Nicki Minaj's track "Good Luck" will be posted upon release. These lyrics are made up from a leaked snippet
• rawLyrics/lyrics_kanyewest_lookatyourphoneastoolmay212018.json (long) Look at your phone as tool not an obligation. Would you walk around with a hammer in your pocket? You would pick up a hammer when you needed it you would never be addicted or obligated to it. Use your phone like a hammer only pick it up when you need it.
• rawLyrics/lyrics_lecrae_elshaddai.json (long) Here we are the last song and uh, I pray that after the music stops that the focus will be on you Lord people will see you for who you are and worhsip you with their lives, Cam share your heart with us
• rawLyrics/lyrics_kendricklamar_d.json (long) "Nah, what you get outta, outta not writing your stuff down on paper, you get the more-you know the...expression, the feeling yourself. You know what you want to say; instead of reading a paper that sounds read, it sounds narrated. Unless you wanna do it narrated...

Ambiguous Headers:

• rawLyrics/lyrics_50cent_jerk.json

  
  Repeat 1 (2x)

Loving me like me Especially when wife be acting funny Cause it don't need for no money And it can be so lovely Take your hand and aim to please Don't drip a drip on your Dungarees Take a towel and cover your jeans Move it back and forth till you see babies

Repeat 1 (2x)

I'm a jerk, if your a jerk How do it work? Show me how you move it back and forth now baby For the guys and the girls Move it all around now work, oh yeah

[Ljferrer]

Preliminary Formatter committed in 0bf669710c33cb1ea3426e4efbcd28e651b45110

• [ ] Need to check with @thammegowda to verify that formatting noise is within tolerances.

[thammegowda]

@Ljferrer what do you mean by formatting noise?

[Ljferrer]

[My Considerations] 1) Sometimes verses get clipped because of a \n\n in the middle (denoting a pause) (I'm using double newline as a section separator) 2) Sometimes verse headers do not have [brackets] 3) Sometimes a chorus repeat is denoted with a number, [HOOK 2x], or (Chorus), etc. 4) I can’t always extract just the lyrics 5) Somehow, a fair bit of skits/interludes/non-poetry got downloaded (Maybe 2-5% of the files have something like this)

[In Conclusion] I’m just keeping the stuff that was formatted as expected.

[On Second Thought] PS. Re: 4) Maybe I could use spaCy language detection to see if each sentence is above an ‘en’ score threshold —> consider that natural language

PPS. Re: 5) Maybe it doesn’t matter

Sent from my iPhone

On Jun 4, 2019, at 6:23 PM, Thamme Gowda notifications@github.com wrote:

@Ljferrer what do you mean by formatting noise?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Ljferrer]

Tried using spaCy to to detect a language threshold for each line. It does not work well with the slang. It can detect the language given the whole document, however.

[Ljferrer]

Closed by b0e27a0f7d0770ec25e9019c476bdc970c60606d