first exploration - aaron

[aaronsadholz]

Just took my first look at the data. Was able to get the feather files working from Eduardo's instructions, and made a couple of simple plots. One shows each artist's location on the world map and the other plots which artists have the most songs in the dataset for a specified year.

I'll move onto looking at the bag of words next.

[edublancas]

Thanks for the update!

[aaronsadholz]

Just took a first look at doing some topic modeling on the data. Results are in my exploration folder.

I used 2 different data representations:

• Term Frequency (tf) - the word count data as we recieved it
• Term Frequency Inverse Document Frequency (tfidf) - transforms the tf data to reduce weight of words which appear in every song, and increase weight of words which appear in less songs.

I used 2 different topic modeling techniques from sklearn:

• Non-Negative Matrix Factorization
• Latent Dirichlet Allocation

Results at this point don't seem to clearly identify different topics, but I see a lot of room for improvement:

• These models require specifying the number of topics to identify, I've only tried with 10 so far. I think exploring different numbers of topics may be helpful.
• The topics tend to group words by language. By removing other languages, we can hopefully create more contextual topics.
• There are plenty of other hyper parameters which I didn't explore

[edublancas]

Thanks for the update! Maybe the word embeddings representation can help here, will update you when I get that to work

[aaronsadholz]

Here's an update on the topic modeling, this summary is also included in my exploration notebook.

After exploring a few basic models, here are my initial takeaways from the model I believe performed the best:

• lda model
• term frequency as the word representation
• 25 topic counts.

There is still a lot of room for improvement, but I think there is quite a bit we could do with these topics as a supplemental part of this project. Each word is assigned a weight to each topic, so we can provide a score for each song (therefore also each artist, location, etc..) for how prevalent each topic is. Since this is an unsupervised algorithm, we have to assign the topics to the clusters ourselves. Below are the top 10 weighted words in each topic, with the topic name I selected (I'm only selecting topics which appeared clear, there are others in the notebook).

Would love to hear any feedback.

Clear Topics:

• Love: 'love', 'heart', 'alway', 'hold', 'feel', 'onli', 'kiss', 'true', 'pleas', 'mine'
• Pain/Loss/Fear: 'blind', 'death', 'lost', 'fear', 'control', 'learn', 'build', 'scream', 'becom', 'bird'
• Religion: 'world', 'god', 'heaven', 'war', 'angel', 'live', 'earth', 'king', 'fight', 'born'
• Death: 'die', 'burn', 'fire', 'ah', 'dead', 'hell', 'flame', 'citi', 'kill', 'devil'
• Dancing: 'wanna', 'danc', 'caus', 'ride', 'bad', 'beat', 'readi', 'shake', 'move', 'gonna'

Clear Groupings that Don't Indicate a Topic:

• "sing/songy words: 'na', 'da', 'di', 'che', 'la', 'il', 'se', 'mi', 'eu', 'ma'
• Sleep/Night: 'night', 'light', 'dream', 'sky', 'star', 'fli', 'wind', 'dark', 'alon', 'sleep'
• Dutch?: 'ya', 'de', 'ik', 'en', 'van', 'je', 'bien', 'pa', 'dat', 'het'
• Hip/hop genre:'fuck', 'nigga', 'shit', 'yo', 'em', 'ya', 'bitch', 'ai', 'yall', 'ass'
• German:'ich', 'und', 'die', 'du', 'der', 'nicht', 'das', 'ist', 'es', 'ein'
• Spanish: 'de', 'la', 'el', 'en', 'te', 'mi', 'tu', 'se', 'es', 'yo'

Topics I'd say are "a bit of a stretch":

• motivation/inpiration: 'time', 'feel', 'life', 'ca', 'live', 'tri', 'mind', 'believ', 'day', 'chang'

[edublancas]

Those are amazing results! I can't wait to see what we get when I finished the word embeddings part. What do you think @jose-alvarado-guzman @valmikkpatel ?