DJDoubleD / QobuzDownloaderX-MOD

Downloads streams directly from Qobuz. Experimental refactoring of QobuzDownloaderX by AiiR
GNU General Public License v3.0
483 stars 16 forks source link

[Feature Request] Creating list of releases on label's page #106

Open viviancels opened 1 day ago

viviancels commented 1 day ago

Hello. Recently, the Qobuz introduced limitation on the number of releases in the list on the label page =1000. Example: https://www.qobuz.com/se-en/label/the-dance-division/download-streaming-albums/287307 If you copy-past the link and download it using your Swedish account, you will receive 1406 releases For some reason, some releases did not download automatically when downloading the label, I downloaded manualy, also inserting links with "se-en" and using Swedish account : https://www.qobuz.com/se-en/album/gonna-be-alright-gregoir-cruz/hlt9f1gly5l5a https://www.qobuz.com/se-en/album/scooby-doo-pa-pa-don-lore-v/duiggmbjp9nrb https://www.qobuz.com/se-en/album/memories-dj-jump/a28ys78et29ec https://www.qobuz.com/se-en/album/secrets-kery-fay/hdpyntt15gp5a

Another example 1000 releases limitation: https://www.qobuz.com/se-en/label/armada-music/download-streaming-albums/248166 In search shows 5416 releases on Label, in fact, there were much fewer releases in the summer

I always used to create a list of releases manually using regular expressions in source code page: .*?.*?href="(.*?)" .*?tooltip">(.*?)(.*?).*?|.* $3 $2 https://www.qobuz.com$1\n But now, due to the limitation of the number of releases on label's page, this makes no sense. However, QobuzDownloaderX-MOD somehow learns about the rest of the releases that are not visible on the label's page (even if it misses some releases).

Describe the solution you'd like

Creating a list (album artist - album title (full date) link) would allow you to find out which releases are available

DJDoubleD commented 21 hours ago

There is a difference between the main qobuz.com website, which is basically a storefront and the Qobuz streaming platforms (web app and mobile apps). I don't know what the main site uses to fetch the max 1000 albums from a label (I'm guessing they are using the catalog/search endpoint as it seems to be limited to 1000 results as well).

In any case, QobuzDownloaderX-MOD uses the same API as the web player's label page, which is still limited to 10000 results (see #101). I can only assume the results from that API are accurate and have no way of automatically comparing them to what the main site shows.

Implementing a list feature in the application would be theoretically possible, but there is no way that winforms will be able to display thousands of results in a performant way (at least that I know of). Rendering performance is the reason why I limit the search results to 15 items.

In any case, as there is only 1 API endpoint that's used to retrieve the albums from a label, it is fairly simple to emulate what the application does to retrieve the albums for download. As an example, I've attached a small python script you can use to generate a json file with all (up to 10000) albums from a label. (had to zip it because github doesn't allow .py files, apparently) The output from this list is what QobuzDownloaderX-MOD will use when you download a label.

Maybe this will help you in finding the discrepancies and get the albums that are on the main site, but not in the downloaded items.

viviancels commented 18 hours ago

@DJDoubleD thank you. The script requests app_id, authorization token and label id

DJDoubleD commented 16 hours ago

@DJDoubleD thank you. The script requests app_id, authorization token and label id

@viviancels Yes, I made those input variables deliberate as I try not to publish the app_id on GitHub. You can get the app_id using this method (I just tried it and still works). Normally it doesn't change, so you can hard code it in the script yourself if you want.

The authorization token is just the token for the account you use, see my wiki for instructions if you need to get it from your personal account. The label id is in the label url that you use to download the label. So in your example https://www.qobuz.com/se-en/label/the-dance-division/download-streaming-albums/287307, the label id is 287307.

viviancels commented 9 hours ago

@DJDoubleD thank you. It works.

1) For those who use regular expressions to clear code and create a spreadsheet for excel, I will leave the code that I used to create a list of label releases: find: .*?media_count.*?name": "(.*?)",.*?title": "(.*?)",.*?url": "(.*?)",.*?release_date_download": "(.*?)",.*? replace: $1\t$2\t$4\thttps://www.qobuz.com/$3\n . matches newline (single line) Example: https://regex101.com/r/KFqh6J/1

For data release I choosed release_date_download but there are also release_date_original and release_date_stream, I don't know what the difference is

2)

      "label": {
        "name": "The Dance Division",
        "id": 287307,
        "albums_count": 1537,
        "supplier_id": 17,
        "slug": "the-dance-division"

It is unclear why parameter albums_count is variable: 1568, 1537, 1557, 1406. Number 1406 for the latest release iat the bottom of the list and the actual number of label releases, which is obtained if the se-en link and the Sweden account. Although, as I wrote above, some releases that are not on the list are also available for Sweden, this is still a mystery. 1568 is the number that the Qobuz site shows if you insert the label name into the search bar, apparently this is the total number of releases available in all countries. I used 25 tokens and corresponding country codes in label's link. the result was 1422 releases. In some countries, downloads were interrupted by an error on the few same releases, something related to the api. Like this: [ERROR] Communication problem with Qobuz API. Details saved to error log [ERROR] Label Download Task ERROR. Details saved to error log.

Fortunately, there was no mistake with Sweden and a couple of other countries.

3) I saw the json code and noticed that albums (releases) have two identifiers:

  1. "url": "/se-en/album/-/m31zcug4bfd3b",
  2. "id": "m31zcug4bfd3b", I believe that QobuzDownloaderX-MOD uses the second identifier and adds the beginning of the address to it https://www.qobuz.com/fr-fr/album/. Perhaps it would be more accurate to use the first identifier (/se-en/album/-/m31zcug4bfd3b), due to the fact that when a release has territorial restrictions, the repository page cannot be opened if its address (link) contains the code of another country (se-en). The first release identifier contains the country code specified in the label reference. I am writing this here so as not to create a new topic.