Main idea of the project

[ian-flores]

Infrastructure:

• Neo4j DB
• Computing backend for the data acquisition script
• Computing backend to serve REST API
• Storage backend to store the new artists info
• Metadata store (cache for artists)

Data acquisition backend:

• Get artist albums from Spotify
• Look at all of the tracks
• Get featuring with other artists (edges in graph)
• Find those artists albums and keep going
  • Make sure not to visit artists if the cache hasn't expired
    • Cache should last at least 1 month
• Store the information in a Neo4j DB

Web backend:

• REST API
  • Serving back data from the graph database
  • Accepting new artists into the database to scrape them

Web frontend:

• Serving web visualizations
  • By genre
  • By artists
  • Of collaborations
  • By decade
• Accepting new artists to store in the DB

[ian-flores]

Data acquisition backend to-do:

Go code

• [ ] Check if artist latest scan is older than 1 month on metadata repository
• [x] Get artist's albums from Spotify
• [x] Get all of the tracks for an album
• [x] Get all of the collaborators in each track
• [x] Build map of collaborators
• [ ] Check if collaborator exists in Graph DB
• [ ] Create collaborator if it doesn't exist
  • Add artist name, followers, genres
• [ ] Check if edge exists (check relationship (ie edge) with attribute of song ID matching)
  • Create edge if it doesn't exist, with song ID as an attribute of the edge
• [ ] If artist doesn't exist on metadata repo
  • [ ] Add artist to the metadata repository with the last scanned date
• [ ] If artist exists update the last scanned date
• [ ] Add new collaborators to the metadata repository with OG scan date set to null and last scan date set to null

Metadata repo

• [ ] Deploy a Google Cloud Firestore instance using pulumi

Neo4j

• [ ] Deploy a Neo4j aurora instance