Data collection

[omersideman]

• Getting a list of viral songs is possible, but where are we going to get songs that havent gone viral, for comparison?
• Where should we get data from?
• Spotify "top tiktok songs" playlists are easiest, but inaccurate and usually contain up to 250 songs.
• Alternatively we can scrape from online charts such as this one , which has seemingly endless songs, sorted by popularity, presumably directly from tiktok. I guess we can use songs at the bottom of the lists as "nonviral" songs? Also most of the songs have spotify links attatched, so we can use only those.
• Todo: figure out scraping - maybe use this or just a chrome extension like link grabber (not compatible with endless scroll so manually)

[omersideman]

update - scraped 6000 songs (30 pages) from chartex, which include numbrer of tiktoks made using the song, the number of likes, spotify links and more. The least popular songs (bottom of the list) have around 150,000 t iktoks, but we can scrape more if necessary. For reference - how i scraped pages:

1. Open the page in Google Chrome
2. Then go to console ; right click and enable LogXMLHttpRequests.
3. Now reload the page and scroll slowly. When new products are populated, you will see different URLs
4. Left click on that URL and it will be highlighted in the Network tab of the Chrome dev tools. From there you can see the json content in the "response" tab and copy it over.