Closed DeveloperHobbit closed 4 years ago
DeveloperHobbit
commented
4 years ago 18/03/2020 As this was the first piece of development first had to create a view and controller for the base page of our project. The method which will get the users music taste will be inside this controller
DeveloperHobbit
commented
4 years ago 18/03/2020 - As authentication system not in place yet, will use dummy values when developing the playlist generation elements
DeveloperHobbit
commented
4 years ago 18/03/2020 - For the following development we followed the tutorial found at the following link: https://medium.com/@raquel.sae.randall/setting-up-a-spotify-api-in-rails-8d60732fe93.
Firstly we needed to go to the Spotify for developers dashboard where we created an application record. Any application can request dara from Spotify Web API however if your application seeks access to a users personal data it must be registered.
Added the three following gems:
gem 'rack-cors' - Anytime you want to bring resources into your web app that come from different origins, you'd implement Cross-Origin Resource Sharing, or CORS for short. CORS has standardized the way we retrieve cross-origin resources. How? …
"...it uses additional HTTP headers to tell a browser to let a web application running at one origin (domain) have permission to access selected resources from a server at a different origin."
What this means is that CORS is going ahead and telling your browser which resources your app is allowed to access when it's trying to access a resource that is not at its original domain (aka cross-origin).
gem 'active_mode_serializers' - Serialization is the process of converting an object into a stream of bytes to store the object or transmit it to memory, a database, or a file. Its main purpose is to save the state of an object in order to be able to recreate it when needed. The reverse process is called deserialization. Serializers are what convert the objects. This gem aims to do this process for different data structures.
gem 'rspotify' - A ruby wrapper for the Spotify web API including all the needed built in Spotify methods. This is the API which we will use to interact with the Spotify API during our development as it is a more user-friendly method of developing with the Spotify API.
These gems are needed to connect to the spotify API from a ruby on rails application.
Added the default config for cors following the tutorial - so that data objects we will be using can be correctly serialized/unserialized so that they can be processed
Created the Spotify setup config file which will hold our authentication. In this file added the line "RSpotify::authenticate(
This is known as app authorization when Spotify authorizes your app to access the Spotify platform. The
at00976
commented
4 years ago 20/03/2020 - * Created the model for a track named 'Track' with the attributes name artist image preview
Finished the tutorial on https://medium.com/@raquel.sae.randall/setting-up-a-spotify-api-in-rails-8d60732fe93 but came across and error with the routes. The specific error was "uninitialized constant Tracks Did you mean? Rack". This error was fixed by removing an unnecessary namespace in the routes.rb file.
After fixing the above error another error was thrown up which was "unknown attribute 'artists' for Track." This was fixed by changing artists to artist.
at00976
commented
4 years ago 26/03/2020 - While researching the RSpotify Gem Annabel found the Github for the development of the gem which contains useful for information about the functions that this gem adds. https://github.com/guilhermesad/rspotify
Changed the random method so that it only displays info of songs that have a valence score between 0.6 and 0.7 to test the retrieval of the valence scores.
Added an av_val method that works out the average valence score for the playlist by finding the mean. The valence score for each song was retrieved using audio_features.valence, audio_features is an inbuilt class in RSpotify with the attributes being the information that can be retrieved by the Spotify API.
Came up with the idea to use multiple of the users playlists by adding all the playlists to an array. The algorithm will then pick songs from this array to create the new playlist.
at00976
commented
4 years ago 27-28/03/2020 - When trying to continue development on 27/03/2020 the command 'rails s' stopped working with the error 'exception with response: 400 bad request'. On the 28/03/2020 Annabel found out that the code causing the error was the RSpotify::authenticate line in spotify_setup.rb. Removing this line allows the 'rails s' command to work again however the playlist generation is dependent on this line meaning that it cannot be left out of the code.
Authenticate is a class method for RSpotify that is needed to access restricted data. It needs two parameters, a client key and a secret key, these are provided by Spotify when the developers register their app. Spotify has 3 ways of authenticating an app as explained on their web page here https://developer.spotify.com/documentation/general/guides/authorization-guide/ at the moment to test the playlist generation we are using the Client Credentials Flow but when the full authentication system is added our app will be using an authentication system that is more similar to the Authorisation Code Flow as this flow has the user grant our webpage access to their data.
at00976
commented
4 years ago 01/04/2020 - The problem with the authentication was solved by refreshing the secret key parameter in the RSpotify::authenticate line.
Added a new method called "new_playlist" this method take the value calculated in av_val and displays the songs that have a valence score that is at least 0.05 higher then the average valence but not more than 0.15 higher. These are dummy values that represent a 5-15% increase in valence value from the average. Once the mood system and weather system has been implemented these values can be changed into parameters that represent how much of an increase we want based on the users mood and the weather.
The increasing complexity of the algorithm led to a "429 too many requests" error that is most likely caused by the algorithm requesting too much information from the Spotify API in too short a time frame. This error can sometimes be fixed by restarting the server but a full fix needs to be developed before deployment of the application.
02/04/2020 - To fix the "too many requests" issue the algorithm was optimised to remove unnecessary calls to the Spotify API so that it didn't hit the request limit. One of the key changes was in the loop that checked if the song was in the valence range or not, instead of calling audio_features.valence twice in the if statement it was changed to set a variable called 'val' at the start of the loop so that audio_features.valence was only called once per loop instead of twice.
DeveloperHobbit
commented
4 years ago 02/04/2020 Considered the idea of using multiple playlists to calculate the average valence score rather than just using a single playlist like we had to work currently. Said this would be an extended feature which we could add once we have the all functionality working for a single playlist
DeveloperHobbit
commented
4 years ago /02/04/2020 Me and Annabel discussed how at the moment our program generates subsets of songs from the playlist which is used to calculate the average valence value meaning no new songs are generated. When researching saw there is a method from rspotify from the built in recommendations class called generate that is used to generate new songs. We will use this to generate new songs based of the average valence value
at00976
commented
4 years ago 02/04/2020 - To develop the app further I attempted to integrate the recommendation function that the RSpotify API includes so that the generated playlist will contain a mix of songs that were already in the user's playlist and some that weren't. When trying to get the function to work it had the error "URI::InvalidURIError". This error was caused by using the Playlist.find function when the Recommendations.generate function only needed the playlist key, changing this allowed me to generate songs based on an existing playlist.
By using seed_genres instead of seed_tracks I could get the recommendation function to display data however a lot of the information displayed was unnecessary and trying to use ".tracks" as suggested by the documentation for the RSpotify gem led to a "NoMethodError".
While looking into the documentation for the recommendations method I found that you could set the function to recommend songs based on a target valence as well as decide how long to make the playlist meaning that once I've figured out how to use the function it can be used to greatly improve how we generate playlists.
at00976
commented
4 years ago Note: the development discussed in this comment is counted as a part of Sprint 3
04/04/2020 - After trying out the 'Recommendations' method and learning how it worked we developed the recommend method that takes a playlist and recommends 20 songs based on a selection of songs in the playlist. We started by creating a loop that took the playlist and created an array of Spotify IDs for the songs in the playlist, the Spotify ID is an unique key used to identify songs. This array of IDs is needed to seed the 'Recommendations' method.
Originally we were going to use the whole playlist as a seed but we found out the the seed tracks array could only contain a maximum of 5 songs, after discussing the best way to pick 5 songs from the playlist to use in the seed we decided the best method would be to pick a random block of 5 songs from the array and use those songs as the seed. This means when the recommend method is run it will recommend songs based on a different seed even if the same playlist is used.
A small problem that needs to be fixed in the future is the amount of information that is displayed by the 'Recommendations' method. At the moment is displays all the information available when in reality we only need the track information, the problem with this is that although the official documentation for RSpotify says to use '.tracks' to get just the track information this does not work so we will need to find a new way to get the information to display in the way we want it to.
stringlytyped
commented
4 years ago I've merged in the authentication code from the master branch and reviewed the changes made by @at00976 and @DeveloperHobbit. Some notes:
@DeveloperHobbit: You seem to misunderstand the circumstances in which CORS would be needed. If you wanted to use JavaScript running on a webpage at example.com to access information provided by a web server running at google.com, Google would need to use CORS to explicitly indicate that cross-origin requests from example.com are to be allowed. Otherwise, the user's web browser would block the communication to enforce the Same-Origin Policy. Since we aren't performing any cross-origin communication, we do not need to use CORS in any fashion.
We probably do not need to be able to serialize the Tracks model into JSON. However, I will leave the code for that in for now in case it does come in useful.
Storing the client_secret in plain-text in a config file that is committed to source control is bad practice because of the security implications. You should rotate the client_secret through the Spotify Developer Dashboard to prevent misuse of your Spotify account.
Integrating the playlist generation with the authentication system is still ongoing.
stringlytyped
commented
4 years ago The recommendations code was originally defined in the TracksController but this is not the best place for it. This code should be in a model because models are responsible for handling data and business logic.
As such, I have moved the recommend method to the User model (and renamed it recommended_tracks). I have also rewritten it to recommend tracks based on the user's past listening history. To do this, I have implemented another method called random_top_tracks. This method retrieves 5 random songs from the user's top 50 tracks to be used as seeds for the recommendations API.
Similarly, I have moved the method for calculating the average valence of a playlist to the Playlist model. The code to populate the playlist with songs is still outstanding.
The user's OAuth tokens are now stored in a credentials_json field in the User record and are accessible as a hash through the credentials method. To use these credentials to authorize a call to the Spotify API, you can now make use of the new to_rspotify_user method.
Finally, the relationships between the various models has be re-thought and the migrations rewritten to be cleaner.
stringlytyped
commented
4 years ago The code to populate playlists has now been added. The User, Playlist, and Track models had to be extended to support this functionality. Refer to code comments for method documentation.
Initially, the playlists were not being generated correctly and the average valence was not near the target. To debug this, a number of information methods (such as valence_distribution) were added to the Playlist model. These methods may be useful for unit tests so they have been left in the code base.
Additionally, a simple playlist view has been added to show the playlist generated for the currently logged in user.
18/03/2020 - When trying to run rails server got an error saying "Your yarn packages are out of date" which is a package manager for java script. Had to research how to update it to be able to run the rails server.