Some songs are not downloaded

[cdytoby]

BeatSyncConsole 0.9.0
Playlist username: cdytoby

By checking this list: https://bsaber.com/songs/new/page/8/?bookmarked_by=cdytoby
I know I have 154 songs (20 per page and 14 in the last page), but the console only read 151 items.

I can't find all the songs are missing, however, I can confirm this song is missing:
https://bsaber.com/songs/1e1/

Here is my BeatSync config:

{
  "RegenerateConfig": false,
  "DownloadTimeout": 30,
  "MaxConcurrentDownloads": 2,
  "RecentPlaylistDays": 7,
  "AllBeatSyncSongsPlaylist": false,
  "BeastSaber": {
    "Enabled": true,
    "Username": "cdytoby",
    "MaxConcurrentPageChecks": 10,
    "Bookmarks": {
      "Enabled": true,
      "MaxSongs": 0,
      "CreatePlaylist": true,
      "PlaylistStyle": "Append"
    },
    "Follows": {
      "Enabled": false,
      "MaxSongs": 20,
      "CreatePlaylist": true,
      "PlaylistStyle": "Append"
    },
    "CuratorRecommended": {
      "Enabled": false,
      "MaxSongs": 20,
      "CreatePlaylist": true,
      "PlaylistStyle": "Append"
    }
  },
  "BeatSaver": {
    "Enabled": false,
    "FavoriteMappers": {
      "Enabled": true,
      "MaxSongs": 20,
      "CreatePlaylist": true,
      "PlaylistStyle": "Append",
      "SeparateMapperPlaylists": false
    },
    "Hot": {
      "Enabled": false,
      "MaxSongs": 20,
      "CreatePlaylist": true,
      "PlaylistStyle": "Append"
    },
    "Downloads": {
      "Enabled": false,
      "MaxSongs": 20,
      "CreatePlaylist": true,
      "PlaylistStyle": "Append"
    },
    "Latest": {
      "Enabled": false,
      "MaxSongs": 20,
      "CreatePlaylist": true,
      "PlaylistStyle": "Append"
    }
  },
  "ScoreSaber": {
    "Enabled": false,
    "TopRanked": {
      "Enabled": true,
      "MaxSongs": 20,
      "CreatePlaylist": true,
      "PlaylistStyle": "Replace",
      "IncludeUnstarred": true,
      "MaxStars": 0.0,
      "MinStars": 0.0
    },
    "LatestRanked": {
      "Enabled": true,
      "MaxSongs": 20,
      "CreatePlaylist": true,
      "PlaylistStyle": "Replace",
      "IncludeUnstarred": true,
      "MaxStars": 0.0,
      "MinStars": 0.0
    },
    "Trending": {
      "Enabled": false,
      "MaxSongs": 20,
      "CreatePlaylist": true,
      "PlaylistStyle": "Append",
      "RankedOnly": false,
      "IncludeUnstarred": true,
      "MaxStars": 0.0,
      "MinStars": 0.0
    },
    "TopPlayed": {
      "Enabled": false,
      "MaxSongs": 20,
      "CreatePlaylist": true,
      "PlaylistStyle": "Append",
      "RankedOnly": false,
      "IncludeUnstarred": true,
      "MaxStars": 0.0,
      "MinStars": 0.0
    }
  },
  "LastRun": "2020-09-06T12:31:33.0698541+02:00"
}

[zxsdklfhskyrvk]

I just used this to download from beastsaber as well, and noticed a similar gap in song count from the site (484), and what your code grabs (474). This gap appears to be very close to the page count.

cdytoby has just over 154 songs, and reported 3 missing (154 / 50 = 3.xxx) I have 484 songs and am missing 10. (484 / 50 = 9.xxxx)

I have to wonder if there is a bug in the API where it is only providing 49 songs per page, or if you are miscounting them by 1. My best guess is that you may have misaligned to an array size. It seems like an "off by 1" bug, which implies an array/loop miscount to me. I'd suggest looking at that aspect of the code.

Having an option for better debug logging, would probably quickly identify the cause. (debug doesn't shows more details regarding the beastsaber page scrapes)

The other possibility is that there are somehow duplicates in beastsaber, which doesn't seem likely, but could happen.

[Zingabopp]

@cdytoby This may actually be a bug in BeastSaber's API. The song you listed isn't in the any of the pages in the API requests:

[zxsdklfhskyrvk]

Any chance you can increase the debug logging for that SongFeedReader? (if it dumps out each song/url it sees) That would help. I think I found another missing song. https://bsaber.com/songs/7027/

[zxsdklfhskyrvk]

I did a simple API check. The API seems to keep the results in order with the web view, no matter what (normally anyway). I could be wrong here.

If you compare my users profiles web view ( https://bsaber.com/songs/new/page/3/?bookmarked_by=qucpjeqphxf0chmt9bhc) from the api (as tested below), there is a difference in total results of 484 (web) vs 474 (API).

If you start looking at the specific results, you will uncover an issue with this song (https://bsaber.com/songs/7027/). This song doesn't appear in the API results. This is just an example that I can show. This issue may be totally random - no idea.

If you change the api result count from 50 to 20, my missing song shows up. It is a API bug. Some bad math with the result size. It isn't as simple as use 20 instead of 50, as I think "20" may have dropped a different song.

From the API, using "count by 50", it shorts me 10 songs compared to the web view. This "loss" varies based on the count value.

0 on this "count by 50" page should be song 7027. (which implies overall song #51)

https://bsaber.com/wp-json/bsaber-api/songs/?bookmarked_by=qucpjeqphxf0chmt9bhc&page=2&count=50

It appears as #8 on this "count by 20" page (which implies overall song #49) https://bsaber.com/wp-json/bsaber-api/songs/?bookmarked_by=qucpjeqphxf0chmt9bhc&page=3&count=20

[cdytoby]

I want to inform you that I'm changing my playlist right now because I found some new songs. So now the list won't stay this way. For now the bug is not yet solved.

BTW I can't find the place to report beastsaber api issue except Discord channel. I'm very inactive there so I'll leave it to you. Thanks in advance.

[Zingabopp]

There's a BeastSaber Discord server, the bug has already been reported. Hopefully it gets fixed soon.

[zxsdklfhskyrvk]

I heard from the bsaber team, and this appears to be fixed now. @cdytoby You should test and verify, but it fixed my issue with the beatsyncconsole app.

[cdytoby]

The number matches the count in BeastSaber website, so I believe this is solved.

Thanks. I'll close the Ticket.