crews-be is an Express application that finds Spotify album credits in Discogs. It exposes one endpoint that, given the album id, requests the album data from Spotify API and immediately after searches for it in Discogs. As it finds credits for it, it updates the state (in app.locals) with such information. It's the backend that supports the crews app also available in my Github profile.
Application keys are needed for both Spotify and Discogs.
Application is deployed at Heroku
The app accepts the following environment configuration variables:
| Name | Description | Required | Default |
|---|---|---|---|
| consumerKey | Discogs consumerKey value, given when an app is created in the Discogs developers site |
:white_check_mark: | |
| consumerSecret | Discogs consumerSecret value, given when an app is created in the Discogs developers site |
:white_check_mark: | |
| throttleTime | Amount of milliseconds that the general queue of Discogs takes between operations. This is related to Discogs API rate limits | 1100 (this is also the recommended value) | |
| PAUSE_NEEDED_AFTER_429 | Amount of milliseconds that the queue of operations is paused after getting 429 from Discogs. That means that no Discogs requests are performed during that time | 30000 | |
| clientId | Given by Spotify when creating a new application | :white_check_mark: | |
| clientSecret | Given by Spotify when creating a new application | :white_check_mark: | |
| clientId | Given by Spotify when creating a new application | :white_check_mark: | |
| PORT | Port where Express listens to requests | 3001 | |
| CORS_ALLOW_ORIGIN | Value of the Access-Control-Allow-Origin header in the response |
* |
npm install.env file.npm start.localhost:<PORT>/data/album/:spotifyAlbumIdYou'll notice the app responds very quickly to the client with an empty bestMatch with 0 progress. That just means the search started. You can keep requesting the album and check how the progress goes, but the console will also inform you about it. Ideally, a web client should poll the endpoint to find out about new data and stop when progress is 100. Since the Discogs API is the main limitant here and it's restricted to one request per second, it's not recommended to request the album more than once per second, since there won't be anything new before that.
Once the search finishes the data remains in the state for as long as the app is in memory, so subsequent requests for the album should show all the data found about the album.
The Discogs API exposes a search endpoint. The details of the results then have to be fetched one by one using the release endpoint.
Discogs API requests are limited. Some albums can throw tens and even hundreds of results. This can make some searches very long to complete. It is possible though, and considerably likely, that the first results are the most relevant. With this in mind the app exhibits the following behaviors:
429 Too Many Requests has still been observed rarely (known issue) so it's handled.progress value. This does not reflect found data, just how many operations have been performed to complete the search. The total amount of operations is given by the sum of the times the Discogs search endpoint has to be called and the amount of releases that need to be individually requested.bestMatch object that contains all the found data for the album. Subsequent requests to the album endpoint will have it updated or not as the progress increases.Clients are suppossed to poll the album endpoint until progress reaches 100. All this also allows to start many searches at the same time and use the Discogs API resources efficiently for all the clients.
The app outputs info about the searches both to console and to disk (/log folder). In the console, a tag with the name of the album and artist being searched lets you know what every message is about. On disk, the logs folder contains a file for every album. The name of such file is the Spotify album ID. Logging is done using winston
Test are contained in *.spec.js files along the module they're testing. Frameworks used are mocha, sinon and Node's assert.