search

Zingabopp / SyncSaberService

Automatically downloads Beat Saber maps
MIT License
42 stars 4 forks source link

SyncSaberConsole crashes while downloading thousands of pages #27

Closed omgitsraven closed 4 years ago

omgitsraven commented 4 years ago

When I launch SyncSaberConsole, it does this:

[SyncSaberLib @ 18:53 - Info] SyncSaberService 0.2.2
[SyncSaberLib @ 18:53 - Info] Using Beat Saber directory: C:\Program Files (x86)\Steam\steamapps\common\Beat Saber
[SyncSaberLib @ 18:53 - Info] Scrapes loaded, 0 BeatSaverSongs and 0 ScoreSaber difficulties loaded
[SyncSaberLib @ 18:53 - Info] Playlist SyncSaber Playlist found in old playlist format.

[SyncSaberLib @ 18:53 - Info] Running the latest version of SyncSaberService.

[SyncSaberLib @ 18:53 - Info] Found 22 songs cached by SongCore.
[SyncSaberLib @ 18:53 - Info] Playlist BeatSaver Newest found in old playlist format.
[SyncSaberLib @ 18:53 - Info] Playlist BeastSaber Curator Recommended found in old playlist format.
[SyncSaberLib @ 18:53 - Info] Playlist BeastSaber Followings found in old playlist format.
[SyncSaberLib @ 18:53 - Info] Playlist BeastSaber Bookmarks found in old playlist format.
[SyncSaberLib @ 18:53 - Info] Playlist ScoreSaber Top Ranked found in old playlist format.
[SyncSaberLib @ 18:53 - Info] Playlist ScoreSaber Trending found in old playlist format.
[SyncSaberLib @ 18:53 - Info] Playlist ScoreSaber Top Played found in old playlist format.
[SyncSaberLib @ 18:53 - Info] Playlist ScoreSaber Latest Ranked found in old playlist format.
[SyncSaberLib @ 18:53 - Info] Scraping new songs. Last Beat Saver scrape was at 0001-01-01 12:00:00 AM.
[SyncSaberLib @ 18:53 - Info] 25117 songs available on 2511 pages
[SyncSaberLib @ 18:53 - Info] On page 0 / 2511
[SyncSaberLib @ 18:53 - Info] On page 10 / 2511
[SyncSaberLib @ 18:53 - Info] On page 20 / 2511
[SyncSaberLib @ 18:54 - Info] On page 30 / 2511
[SyncSaberLib @ 18:54 - Info] On page 40 / 2511
[SyncSaberLib @ 18:54 - Info] On page 50 / 2511
[SyncSaberLib @ 18:54 - Info] On page 60 / 2511
[SyncSaberLib @ 18:54 - Info] On page 70 / 2511
[SyncSaberLib @ 18:54 - Info] On page 80 / 2511
[Program.cs_Main(299) @ 18:54 - Exception] Uncaught exception in Main() - SyncSaberLib.Web.HttpGetException-Exception getting page (https://beatsaver.com/api/maps/latest/89): Too Many Requests
   at SyncSaberLib.Web.WebUtils.GetPageText(String url, Boolean exceptionOnHttpNotSuccessful)
   at SyncSaberLib.Web.BeatSaverReader.ScrapeBeatSaver(Int32 requestDelay, Boolean onlyGetNew, Int32 maxPages)
   at SyncSaberLib.SyncSaber.ScrapeNewSongs()
   at SyncSaberConsole.Program.Main(String[] args) in C:\Users\Jared\Source\Repos\Zingabopp\SyncSaberService\SyncSaberConsole\Program.cs:line 141
Press any key to continue...

Why is it downloading 2511 pages of content? All I want is my BeastSaber bookmarks; my SyncSaberService.ini page looks like this:

[SyncSaberService]
BeatSaberPath = C:\Program Files (x86)\Steam\steamapps\common\Beat Saber
BeastSaberUsername = ravenworks
SyncCuratorRecommendedFeed = False
SyncBookmarksFeed = True
SyncFollowingsFeed = False
SyncTopPPFeed = False
SyncFavoriteMappersFeed = False
MaxCuratorRecommendedPages = 1
MaxBookmarksPages = 50
MaxFollowingsPages = 1
MaxScoreSaberSongs = 1
MaxBeatSaverPages = 1
DeleteOldVersions = True
DownloadTimeout = 5
MaxConcurrentDownloads = 1
MaxConcurrentPageChecks = 1
LoggingLevel = Info

Is there anything that I need to change in my .ini file to keep SyncSaber from crashing while attempting to scrape pages I don't care about anyway?

Zingabopp commented 4 years ago

The way SyncSaberService works is it scrapes all the data from BeatSaver and ScoreSaber and then uses that to match up with BeastSaber songs and whatnot. The download should include pre-scraped data so it would stop when it gets the new songs it doesn't have. BeatSyncConsole (SyncSaberService's replacement) won't be doing that when I finally finish it.

If you go to the downloaded zip and put the .json files from the ScrapedData folder into your SyncSaberService\ScrapedData, hopefully that should sort of fix it. Apparently, I forgot to handle Beat Saver's rate limits X_x

omgitsraven commented 4 years ago

Ahh, okay—I was trying to figure out how to get it to re-download songs that I'd deleted, so I deleted all the files that looked like it was storing history in to try and start fresh; I should've tried redownloading the whole install, sorry about that.

Zingabopp commented 4 years ago

Oh, if you want to redownload songs, delete SyncSaberHistory.json in the root folder iirc

omgitsraven commented 4 years ago

Will keep it in mind, thanks :)