andruzzzhka / BeatSaberScrappedData

ScoreSaber/BeatSaver scraped data
37 stars 1 forks source link

Missing some maps #11

Closed ibillingsley closed 4 months ago

ibillingsley commented 9 months ago

I noticed a few maps I have are missing from combinedScrappedData.json (and presumably the Protobuf data). These are not new, outdated, or deleted.

https://beatsaver.com/maps/2bad2 https://beatsaver.com/maps/27bfd https://beatsaver.com/maps/2ef1a https://beatsaver.com/maps/2c348 https://beatsaver.com/maps/27cee https://beatsaver.com/maps/27ca5 https://beatsaver.com/maps/32c07 https://beatsaver.com/maps/31fb3 https://beatsaver.com/maps/27ca4 https://beatsaver.com/maps/2c401 https://beatsaver.com/maps/2bba3 https://beatsaver.com/maps/32b94 https://beatsaver.com/maps/2c8a2 https://beatsaver.com/maps/3026f https://beatsaver.com/maps/2c4cb

Not sure if there's anything special about these particular maps, or even if this data is intended to be 100% complete. Maybe they were unpublished at one point and then republished?

kinsi55 commented 4 months ago

Theres some long running issue in the scraper where some maps are just missing in the result (Inconsistently, they come and go) - I have yet to find the cause for this

kinsi55 commented 4 months ago

I have given it another look just now and I believe to have found the issue - Or at the very least significantly improved upon it. This will be reflected in the dataset at some point soon as I'm working on an updated format for Songdetails / Proto to include some new stuff

kinsi55 commented 4 months ago

The fix is now applied, lmk if you should still happen to find missing maps

ibillingsley commented 4 months ago

Awesome, I can confirm all 4000+ maps in my library are now present.
(I checked by sorting BetterSongList by BeatSaver Date. Missing, outdated, or deleted maps end up at the bottom)