search

ShokoAnime / ShokoServer

Repository for Shoko Server.
https://shokoanime.com/
MIT License
412 stars 74 forks source link

Regular HTTP Bans #1182

Open Rand-Random opened 1 month ago

Rand-Random commented 1 month ago

Basically running into a ban almost immediatly after a ban is lifted, and this over and over again. I have no clue what shoko is actually currently doing at all, it seems trying to fetch rather random series.

Shoko seems to queue more series to update, than it is able to fetch, already tried to cleared the queue only for a new file of the current season to again trigger this update cycle.

Current queue is:

Get AniDB Anime Data
  Anime: Ranma 1/2: Chuugoku Nekonron Daikessen! Okite Yaburi no Gekitou Hen!!
  Anime: Ranma 1/2: Kessen Tougenkyou! Hanayome o Torimodose!!
  Anime: Ranma 1/2: Akumu! Shunmin Kou
  Anime: Ranma 1/2: Nettou Uta Gassen
  Anime: Inuyasha
  Anime: Urusei Yatsura
  Anime: Party kara Tsuihou Sareta Sono Chiyushi, Jitsu wa Saikyou ni Tsuki
  Anime: Hoshi Furu Oukoku no Nina
  Anime: Seirei Gensouki 2
  Anime: Houkago Shounen Hanako-kun (2024)
  Anime: Natsume Yuujinchou Shichi
  Anime: Kamierabi (2024)
  Anime: Futoku no Guild
  Anime: Shin Tennis no Ouji-sama: U-17 World Cup Semifinal
  Anime: The Peak of True Martial Arts 3rd Season
  Anime: Chi. Chikyuu no Undou ni Tsuite
  Anime: Arifureta Shokugyou de Sekai Saikyou Season 3
  Anime: Kidou Senshi Gundam: Fukushuu no Requiem
  Anime: Chi. Chikyuu no Undou ni Tsuite
  Anime: Ooi! Tonbo (2024)
  Anime: Touhai: Ura Rate Mahjong Touhai Roku
  Anime: Kimi wa Meido-sama.
  Anime: Punirunes: Puni 2
  Anime: Youkai Gakkou no Sensei Hajimemashita!
  Anime: Trillion Game
  Anime: Ranma 1/2 (2024)
  Anime: Shounen ga Otona ni Natta Natsu
  Anime: Shinkansen Henkei Robo Shinkalion: Change the World
  Anime: Dungeon ni Deai o Motomeru no wa Machigatte Iru Darouka: Familia Myth V
  Anime: Kinoko Inu
  Anime: NegaPosi Angler
  Anime: Hitoribocchi no Isekai Kouryaku
  Anime: Kekkon Surutte, Hontou Desuka
  Anime: Ao no Hako
  Anime: Dan Da Dan
  Anime: Tono to Inu
  Anime: Mecha-ude (2024)
  Anime: Sayounara Ryuusei, Konnichiwa Jinsei

No idea why it tries to update eg. "Inuyasha" or "Shin Tennis no Ouji-sama: U-17 World Cup Semifinal." Didn't recently add anything to Inuyasha, why is it fetching data? And the series "Shin Tennis no Ouji-sama: U-17 World Cup Semifinal" never heard about it and I don't have it in my library, so why?

As I said, I already once cleared the queue, but it again filled up, so I currently don't know what to do?

20:17 (8:17 pm) ban was lifted, 20:21 (8:21 pm) again banned. Full log: log.txt

[2024-10-10 20:17:30:512] Info|Shoko.Server.Providers.AniDB.HTTP.AniDBHttpConnectionHandler > AniDB HTTP ban (12h) is over

[had to move the log into a attached file because it exceeded the limit of 65536 characters an github post can have)

[2024-10-10 20:21:49:286] Info|GetAniDBAnimeJob_AnimeID:405_ForceRefresh:true_DownloadRelations:true_RelDepth:3 > Processing GetAniDBAnimeJob: 405
[2024-10-10 20:21:51:203] Warn|Shoko.Server.Providers.AniDB.HTTP.AniDBHttpConnectionHandler > AniDB HTTP Banned!
[2024-10-10 20:21:51:203] Error|Quartz.Core.JobRunShell > Job AniDB.GetAniDBAnimeJob_AnimeID:405_ForceRefresh:true_DownloadRelations:true_RelDepth:3 threw a JobExecutionException: : Parameters: refire = False, unscheduleFiringT
rigger = False, unscheduleAllTriggers = False 
 Quartz.JobExecutionException: One or more errors occurred. (Exception of type 'Shoko.Server.Providers.AniDB.AniDBBannedException' was thrown.)
 ---> System.AggregateException: One or more errors occurred. (Exception of type 'Shoko.Server.Providers.AniDB.AniDBBannedException' was thrown.)
 ---> Shoko.Server.Providers.AniDB.AniDBBannedException: Exception of type 'Shoko.Server.Providers.AniDB.AniDBBannedException' was thrown.
   at Shoko.Server.Providers.AniDB.HTTP.AniDBHttpConnectionHandler.<>c__DisplayClass11_0.<<GetHttpDirectly>b__0>d.MoveNext() in D:\a\ShokoServer\ShokoServer\Shoko.Server\Providers\AniDB\HTTP\AniDBHttpConnectionHandler.cs:line 64
--- End of stack trace from previous location ---
   at Shoko.Server.Providers.AniDB.HTTP.AniDBHttpConnectionHandler.GetHttpDirectly(String url) in D:\a\ShokoServer\ShokoServer\Shoko.Server\Providers\AniDB\HTTP\AniDBHttpConnectionHandler.cs:line 55
   at Shoko.Server.Providers.AniDB.HTTP.AniDBHttpConnectionHandler.GetHttp(String url) in D:\a\ShokoServer\ShokoServer\Shoko.Server\Providers\AniDB\HTTP\AniDBHttpConnectionHandler.cs:line 40
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Shoko.Server.Scheduling.Jobs.AniDB.GetAniDBAnimeJob.GetResponse(SVR_AniDB_Anime anime)
   at Shoko.Server.Scheduling.Jobs.AniDB.GetAniDBAnimeJob.Process() in D:\a\ShokoServer\ShokoServer\Shoko.Server\Scheduling\Jobs\AniDB\GetAniDBAnimeJob.cs:line 102
   at Shoko.Server.Scheduling.Jobs.BaseJob.Execute(IJobExecutionContext context) in D:\a\ShokoServer\ShokoServer\Shoko.Server\Scheduling\Jobs\BaseJob.cs:line 33
   --- End of inner exception stack trace ---
   at Shoko.Server.Scheduling.Jobs.BaseJob.Execute(IJobExecutionContext context) in D:\a\ShokoServer\ShokoServer\Shoko.Server\Scheduling\Jobs\BaseJob.cs:line 50
   at Quartz.Core.JobRunShell.Run(CancellationToken cancellationToken) [See nested exception: System.AggregateException: One or more errors occurred. (Exception of type 'Shoko.Server.Providers.AniDB.AniDBBannedException' was thrown.)
 ---> Shoko.Server.Providers.AniDB.AniDBBannedException: Exception of type 'Shoko.Server.Providers.AniDB.AniDBBannedException' was thrown.
   at Shoko.Server.Providers.AniDB.HTTP.AniDBHttpConnectionHandler.<>c__DisplayClass11_0.<<GetHttpDirectly>b__0>d.MoveNext() in D:\a\ShokoServer\ShokoServer\Shoko.Server\Providers\AniDB\HTTP\AniDBHttpConnectionHandler.cs:line 64
--- End of stack trace from previous location ---
   at Shoko.Server.Providers.AniDB.HTTP.AniDBHttpConnectionHandler.GetHttpDirectly(String url) in D:\a\ShokoServer\ShokoServer\Shoko.Server\Providers\AniDB\HTTP\AniDBHttpConnectionHandler.cs:line 55
   at Shoko.Server.Providers.AniDB.HTTP.AniDBHttpConnectionHandler.GetHttp(String url) in D:\a\ShokoServer\ShokoServer\Shoko.Server\Providers\AniDB\HTTP\AniDBHttpConnectionHandler.cs:line 40
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Shoko.Server.Scheduling.Jobs.AniDB.GetAniDBAnimeJob.GetResponse(SVR_AniDB_Anime anime)
   at Shoko.Server.Scheduling.Jobs.AniDB.GetAniDBAnimeJob.Process() in D:\a\ShokoServer\ShokoServer\Shoko.Server\Scheduling\Jobs\AniDB\GetAniDBAnimeJob.cs:line 102
   at Shoko.Server.Scheduling.Jobs.BaseJob.Execute(IJobExecutionContext context) in D:\a\ShokoServer\ShokoServer\Shoko.Server\Scheduling\Jobs\BaseJob.cs:line 33]
Rand-Random commented 1 month ago

As I said

And the series "Shin Tennis no Ouji-sama: U-17 World Cup Semifinal" never heard about it and I don't have it in my library, so why (does it try to update the series)?

So I went and looked into my logs to find the series "Shin Tennis no..." and found the following log. GetAniDBCalendarJob seems to add it to my queue, if I am reading the logs correctly? why?

[2024-10-10 06:44:45:591] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Checking LAN Connectivity…
[2024-10-10 06:44:45:622] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Found a local gateway to use
[2024-10-10 06:44:45:622] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Checking WAN Connectivity…
[2024-10-10 06:44:45:622] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Successfully connected to 3/4 internet service endpoints
[2024-10-10 07:09:48:833] Info|GetAniDBCalendarJob > Processing GetAniDBCalendarJob
[2024-10-10 07:14:44:181] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Checking LAN Connectivity…
[2024-10-10 07:14:44:212] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Found a local gateway to use
[2024-10-10 07:14:44:212] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Checking WAN Connectivity…
[2024-10-10 07:14:44:212] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Successfully connected to 3/4 internet service endpoints
[2024-10-10 07:14:48:831] Info|GetUpdatedAniDBAnimeJob_ForceRefresh:true > Processing GetUpdatedAniDBAnimeJob
[2024-10-10 07:14:48:831] Info|GetUpdatedAniDBAnimeJob_ForceRefresh:true > 12:08:34.8323973 since last UPDATED command
[2024-10-10 07:14:48:917] Info|GetUpdatedAniDBAnimeJob_ForceRefresh:true > Scheduling update for anime: Kamierabi (2024) (18334)
[2024-10-10 07:14:48:953] Info|GetUpdatedAniDBAnimeJob_ForceRefresh:true > Scheduling update for anime: Futoku no Guild (17244)
[2024-10-10 07:14:48:987] Info|GetUpdatedAniDBAnimeJob_ForceRefresh:true > Scheduling update for anime: Shin Tennis no Ouji-sama: U-17 World Cup Semifinal (18054)
[2024-10-10 07:14:49:209] Info|GetUpdatedAniDBAnimeJob_ForceRefresh:true > Scheduling update for anime: Zhen Wu Dianfeng 3rd Season (18342)
[2024-10-10 07:14:49:225] Info|GetUpdatedAniDBAnimeJob_ForceRefresh:true > Scheduling update for anime: Chi. Chikyuu no Undou ni Tsuite (17480)
[2024-10-10 07:14:49:258] Info|GetUpdatedAniDBAnimeJob_ForceRefresh:true > Updating 5 anime records, and 3 group status records
[2024-10-10 07:14:49:303] Info|GetAniDBReleaseGroupStatusJob_AnimeID:18334_ForceRefresh:true > Processing GetAniDBReleaseGroupJob: 18334
[2024-10-10 07:14:50:990] Info|RefreshAnimeStatsJob_AnimeID:18334 > Processing RefreshAnimeStatsJob for Kamierabi (2024)
[2024-10-10 07:14:50:998] Info|Shoko.Server.Services.AnimeSeriesService > Starting Updating STATS for SERIES Kamierabi (2024) - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:51:006] Info|Shoko.Server.Services.AnimeSeriesService > Finished updating stats for SERIES Kamierabi (2024) in 8.3715ms
[2024-10-10 07:14:51:006] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Kamierabi from Top Level (recursively) - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:51:006] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Kamierabi - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:51:085] Info|RefreshAnimeStatsJob_AnimeID:18334 > Processing RefreshAnimeStatsJob for Kamierabi (2024)
[2024-10-10 07:14:51:100] Info|Shoko.Server.Services.AnimeSeriesService > Starting Updating STATS for SERIES Kamierabi (2024) - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:51:100] Info|Shoko.Server.Services.AnimeSeriesService > Finished updating stats for SERIES Kamierabi (2024) in 7.3267ms
[2024-10-10 07:14:51:100] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Kamierabi from Top Level (recursively) - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:51:100] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Kamierabi - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:51:239] Info|GetAniDBReleaseGroupStatusJob_AnimeID:17244_ForceRefresh:true > Processing GetAniDBReleaseGroupJob: 17244
[2024-10-10 07:14:53:023] Info|RefreshAnimeStatsJob_AnimeID:17244 > Processing RefreshAnimeStatsJob for Futoku no Guild
[2024-10-10 07:14:53:023] Info|Shoko.Server.Services.AnimeSeriesService > Starting Updating STATS for SERIES Futoku no Guild - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:53:040] Info|Shoko.Server.Services.AnimeSeriesService > Finished updating stats for SERIES Futoku no Guild in 7.8986ms
[2024-10-10 07:14:53:040] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Futoku no Guild from Top Level (recursively) - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:53:040] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Futoku no Guild - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:53:101] Info|RefreshAnimeStatsJob_AnimeID:17244 > Processing RefreshAnimeStatsJob for Futoku no Guild
[2024-10-10 07:14:53:116] Info|Shoko.Server.Services.AnimeSeriesService > Starting Updating STATS for SERIES Futoku no Guild - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:53:116] Info|Shoko.Server.Services.AnimeSeriesService > Finished updating stats for SERIES Futoku no Guild in 7.413ms
[2024-10-10 07:14:53:116] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Futoku no Guild from Top Level (recursively) - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:53:116] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Futoku no Guild - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:53:572] Info|GetAniDBReleaseGroupStatusJob_AnimeID:17480_ForceRefresh:true > Processing GetAniDBReleaseGroupJob: 17480
[2024-10-10 07:14:55:101] Info|RefreshAnimeStatsJob_AnimeID:17480 > Processing RefreshAnimeStatsJob for Chi. Chikyuu no Undou ni Tsuite
[2024-10-10 07:14:55:101] Info|Shoko.Server.Services.AnimeSeriesService > Starting Updating STATS for SERIES Chi. Chikyuu no Undou ni Tsuite - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:55:119] Info|Shoko.Server.Services.AnimeSeriesService > Finished updating stats for SERIES Chi. Chikyuu no Undou ni Tsuite in 7.4781ms
[2024-10-10 07:14:55:119] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Chi. Chikyuu no Undou ni Tsuite from Top Level (recursively) - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:55:119] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Chi. Chikyuu no Undou ni Tsuite - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:55:178] Info|RefreshAnimeStatsJob_AnimeID:17480 > Processing RefreshAnimeStatsJob for Chi. Chikyuu no Undou ni Tsuite
[2024-10-10 07:14:55:195] Info|Shoko.Server.Services.AnimeSeriesService > Starting Updating STATS for SERIES Chi. Chikyuu no Undou ni Tsuite - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:55:195] Info|Shoko.Server.Services.AnimeSeriesService > Finished updating stats for SERIES Chi. Chikyuu no Undou ni Tsuite in 8.6017ms
[2024-10-10 07:14:55:195] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Chi. Chikyuu no Undou ni Tsuite from Top Level (recursively) - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:55:195] Info|Shoko.Server.Services.AnimeGroupService > Starting Updating STATS for GROUP Chi. Chikyuu no Undou ni Tsuite - Watched Stats: True, Missing Episodes: True
[2024-10-10 07:14:55:270] Info|GetAniDBReleaseGroupJob_GroupID:14642 > Processing GetAniDBReleaseGroupJob: 14642
[2024-10-10 07:14:55:316] Info|GetAniDBReleaseGroupJob_GroupID:16312 > Processing GetAniDBReleaseGroupJob: 16312
[2024-10-10 07:14:55:362] Info|GetAniDBReleaseGroupJob_GroupID:16325 > Processing GetAniDBReleaseGroupJob: 16325
[2024-10-10 07:14:55:408] Info|GetAniDBReleaseGroupJob_GroupID:17144 > Processing GetAniDBReleaseGroupJob: 17144
[2024-10-10 07:14:55:455] Info|GetAniDBReleaseGroupJob_GroupID:17713 > Processing GetAniDBReleaseGroupJob: 17713
[2024-10-10 07:14:55:501] Info|GetAniDBReleaseGroupJob_GroupID:4960 > Processing GetAniDBReleaseGroupJob: 4960
[2024-10-10 07:14:55:546] Info|GetAniDBReleaseGroupJob_GroupID:7587 > Processing GetAniDBReleaseGroupJob: 7587
[2024-10-10 07:14:55:591] Info|GetAniDBReleaseGroupJob_GroupID:9909 > Processing GetAniDBReleaseGroupJob: 9909
[2024-10-10 07:14:55:636] Info|GetAniDBReleaseGroupJob_GroupID:11111 > Processing GetAniDBReleaseGroupJob: 11111
[2024-10-10 07:14:55:683] Info|GetAniDBReleaseGroupJob_GroupID:11276 > Processing GetAniDBReleaseGroupJob: 11276
[2024-10-10 07:14:55:729] Info|GetAniDBReleaseGroupJob_GroupID:12574 > Processing GetAniDBReleaseGroupJob: 12574
[2024-10-10 07:14:55:775] Info|GetAniDBReleaseGroupJob_GroupID:12675 > Processing GetAniDBReleaseGroupJob: 12675
[2024-10-10 07:14:55:820] Info|GetAniDBReleaseGroupJob_GroupID:13075 > Processing GetAniDBReleaseGroupJob: 13075
[2024-10-10 07:14:55:865] Info|GetAniDBReleaseGroupJob_GroupID:13816 > Processing GetAniDBReleaseGroupJob: 13816
[2024-10-10 07:14:55:910] Info|GetAniDBReleaseGroupJob_GroupID:14969 > Processing GetAniDBReleaseGroupJob: 14969
[2024-10-10 07:14:55:955] Info|GetAniDBReleaseGroupJob_GroupID:15975 > Processing GetAniDBReleaseGroupJob: 15975
[2024-10-10 07:14:55:999] Info|GetAniDBReleaseGroupJob_GroupID:16021 > Processing GetAniDBReleaseGroupJob: 16021
[2024-10-10 07:14:56:045] Info|GetAniDBReleaseGroupJob_GroupID:16037 > Processing GetAniDBReleaseGroupJob: 16037
[2024-10-10 07:14:56:089] Info|GetAniDBReleaseGroupJob_GroupID:16172 > Processing GetAniDBReleaseGroupJob: 16172
[2024-10-10 07:14:56:134] Info|GetAniDBReleaseGroupJob_GroupID:16193 > Processing GetAniDBReleaseGroupJob: 16193
[2024-10-10 07:14:56:179] Info|GetAniDBReleaseGroupJob_GroupID:16209 > Processing GetAniDBReleaseGroupJob: 16209
[2024-10-10 07:14:56:223] Info|GetAniDBReleaseGroupJob_GroupID:16619 > Processing GetAniDBReleaseGroupJob: 16619
[2024-10-10 07:14:56:267] Info|GetAniDBReleaseGroupJob_GroupID:17042 > Processing GetAniDBReleaseGroupJob: 17042
[2024-10-10 07:14:56:310] Info|GetAniDBReleaseGroupJob_GroupID:17361 > Processing GetAniDBReleaseGroupJob: 17361
[2024-10-10 07:14:56:354] Info|GetAniDBReleaseGroupJob_GroupID:17504 > Processing GetAniDBReleaseGroupJob: 17504
[2024-10-10 07:14:56:399] Info|GetAniDBReleaseGroupJob_GroupID:17907 > Processing GetAniDBReleaseGroupJob: 17907
[2024-10-10 07:14:56:442] Info|GetAniDBReleaseGroupJob_GroupID:18285 > Processing GetAniDBReleaseGroupJob: 18285
[2024-10-10 07:14:57:177] Info|GetAniDBReleaseGroupJob_GroupID:16740 > Processing GetAniDBReleaseGroupJob: 16740
[2024-10-10 07:44:34:838] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Checking LAN Connectivity…
[2024-10-10 07:44:34:869] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Found a local gateway to use
[2024-10-10 07:44:34:869] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Checking WAN Connectivity…
[2024-10-10 07:44:34:869] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Successfully connected to 3/4 internet service endpoints
revam commented 1 month ago

The point of using the calendar is to add/update the seasonal anime entries. Also it seems like you have enabled fetching related anime entries with a depth above 0 (where 0 means disabled), so it will fetch all the entries currently in the calendar in addition to any related anime entries.

This seems to me like a misconfiguration on your part if you don't want it to update seasonal data, but do correct me if I'm assuming wrong here.

Rand-Random commented 1 month ago

Didn't touch those settings in the last couple of years, so I couldn't even say, if I did those settings. "By Design" seems odd, cause it could be interpreted as its by design to get banned.

Even if, shouldn't there be a sanity check for updating "related" items eg. don't update items that are completed series. eg. Inuyasha.

Inuyasha is long over and the chances it has new data is slim if not 0, and how would it be related to the current season? Is it maybe jumping from relation to relation and it basically ends up fetching all of anidb?

What I did now,

1) I set Related Depth to 0, though isn't it better to uncheck "Related Anime"? {78E11949-C54A-40DF-85B2-F2D90A9E0A6D}

2) cleared the queue (which again increased instead of decreased)

3) added new files of the current season and let shoko scan the files (udp isn't banned so it can add those files)

4) current queue is 

Get AniDB Anime Data
  Anime: Ao no Hako
  Anime: Delico's Nursery
  Anime: Dan Da Dan
  Anime: Hitoribocchi no Isekai Kouryaku
  Anime: Dungeon ni Deai o Motomeru no wa Machigatte Iru Darouka: Familia Myth V
  Anime: Kekkon Surutte, Hontou Desuka
  Anime: Mecha-ude (2024)
  Anime: NegaPosi Angler
  Anime: Sayounara Ryuusei, Konnichiwa Jinsei

5) will report back when the current ban is lifed

harshithmohan commented 1 month ago

I'll just put a screenshot here because I don't want to copy-paste 😔 image

revam commented 1 month ago

@Rand-Random you can also disable the related anime checkbox if you want. It will disable the depth setting when you do.

bigretromike commented 1 month ago

this is same issue as the existence of JMM/Shoko. Until there is system defaults/built-in system to mitigate this issue new people with bigger collection will auto-ban them-self, over and over again. Sadly that is not 100% fault of shoko, as anidb wont release information about leeching-ratio until they ban to mitigate people from "cloning" their database even when there are multiply copies already in wild.

The "best strategy" was/will be adding single small series to import folder, and process it manually. or in batch-of-10.

tamodolo commented 1 month ago

I'll share my log.

server restarted at line 9560 to force anidb check and started to update only to get banned some minutes later. shoko.log

I think something may have changed on anidb side. Or shoko is updating faster than before.

bigretromike commented 1 month ago

From the logs its seams that you leeched off anidb with requests and 3 minute later got ban. As it always been (more or less). That is sadly the "first" experience with bigger collections.

Also it looks like HttpRateLimiter did fire just before ban, but then luck weren't on your side.

But for real, @ElementalCrisis from this log it looks like he been asking anidb for 2 series per 1second. (00:31:40, 00:31:38, 00:31:34), 3series per 1 second (00:31:36, 00:31:28), 4 series per 1 second (00:31:20) - I wont 100% put my life for that log as its verbose if not clutter, but there is something there that can help with ban issue.

I'm maybe old and rusty, but I don't recall being able to leech that fast from anidb in the best years of my life ;-)

ElementalCrisis commented 1 month ago

Hmm. @da3dsoul or @revam can you look into this? If its an actual issue we've somehow created with the new queue, we'll want to address it sooner rather than later.

harshithmohan commented 1 month ago

But for real, @ElementalCrisis from this log it looks like he been asking anidb for 2 series per 1second. (00:31:40, 00:31:38, 00:31:34), 3series per 1 second (00:31:36, 00:31:28), 4 series per 1 second (00:31:20) - I wont 100% put my life for that log as its verbose if not clutter, but there is something there that can help with ban issue.

Where? 🤔 I see one series per 2 seconds

bigretromike commented 1 month ago

But for real, @ElementalCrisis from this log it looks like he been asking anidb for 2 series per 1second. (00:31:40, 00:31:38, 00:31:34), 3series per 1 second (00:31:36, 00:31:28), 4 series per 1 second (00:31:20) - I wont 100% put my life for that log as its verbose if not clutter, but there is something there that can help with ban issue.

Where? 🤔 I see one series per 2 seconds

Sorry, I probably took Getting AniDB Anime Data for job and ignored Job Started. You are right 1 per 2 sec (more or less).

00:31:45 and 00:31:46 are 1 each second. 00:31:50 and 00:31:51 are 1 each second.

Maybe that trigger something ?

Rand-Random commented 1 month ago

Update on my side:

After disabling the "related anime" feature as @revam suggested, the ban didn't occur.

I got curious and wanted to re-enable the feature and see if the ban again happens or not. I re-enabled the feature and manually triggered the action "Update AniDB Calendar" but nothing happened. Nothing as in no ban, and no series got added to the queue.

6 days later with the setting enabled, a ban again occurred.

Currently there are 100 items in the queue to be updated

Kidou Senshi Gundam: Thunderbolt - December Sky
Kidou Senshi Gundam II: Ai Senshi Hen
Kidou Senshi Gundam (1981)
Kidou Senshi Gundam Unicorn
Kidou Senshi Gundam ZZ
Kidou Senshi Gundam: Gyakushuu no Char
Dr.
Kidou Senshi Gundam AGE: Memory of Eden
Kidou Senshi Gundam SEED Destiny
Kidou Senshi Gundam SEED MSV ASTRAY
Kidou Senshi Gundam SEED: Meidou no Sora
Kidou Senshi Gundam SEED: Harukanaru Akatsuki
Kidou Senshi Gundam SEED: Kokuu no Senjou
Gundam Evolve
Gundam Build Fighters: GM no Gyakushuu
Gundam Build Fighters: Battlogue
Mokei Senshi Gunpla Builders Beginning G
Kidou Senshi Gundam: Tekketsu no Orphans (2016)
Kidou Senshi Gundam: Tekketsu no Orphans - Urdr Hunt
Kidou Senshi Gundam: Tekketsu no Orphans Tokubetsu Hen
Kidou Senshi Gundam: Suisei no Majo (2023)
Gundam Build Divers Re:Rise
SD Gundam World Heroes
Kidou Senshi SD Gundam
Kidou Butouden G Gundam
Daisuki! Hello Kitty
Kidou Senshi SD Gundam Mk-II
Kidou Senshi SD Gundam Gaiden
Kidou Senshi SD Gundam no Gyakushuu: SD Sengokuden - Abaowakuujou no Shou
SD Gundam Force
Kidou Senshi SD Gundam Matsuri
Musha Knight Commando: SD Gundam Scramble
Kidou Senshi SD Gundam no Gyakushuu: Arashi o Yobu Gakuensai
Chou Den'ei Ban SD Gundam Sangokuden: Brave Battle Warriors
Shin Kidou Senki Gundam Wing
Turn A Gundam II: Gekkou Chou
Turn A Gundam I: Chikyuukou
Shin Kidou Senki Gundam Wing: Endless Waltz Tokubetsu Hen
Gundam-W: Endless Waltz
Shin Kidou Senki Gundam Wing: Operation Meteor
Gundam Build Metaverse
Kidou Senshi Gundam Double O (2008)
Kidou Senshi Gundam Double O Special Edition I: Celestial Being
Kidou Senshi Gundam ZZ
Gundam Neo Experience: 0087 - Green Divers
Kidou Senshi Z Gundam: A New Translation - Hoshi no Kodou wa Ai
Kidou Senshi Z Gundam: A New Translation - Hoshi o Tsugu Mono
Kidou Senshi Z Gundam: A New Translation - Koibito-tachi
Turn A Gundam II: Gekkou Chou
Turn A Gundam I: Chikyuukou
Kidou Senshi Gundam MS Igloo 2: Juuryoku Sensen
Iribitari Gal ni Manko Tsukawasete Morau Hanashi
Shinkansen Henkei Robo Shinkalion: Change the World
Dungeon ni Deai o Motomeru no wa Machigatte Iru Darouka: Familia Myth V
Kinoko Inu
NegaPosi Angler
Hitoribocchi no Isekai Kouryaku
Ao no Hako
Dan Da Dan
Kinsou no Vermeil: Gakeppuchi Majutsushi wa Saikyou no Yakusai to Mahou Sekai o Tsukisusumu
Mecha-ude (2024)
Sayounara Ryuusei, Konnichiwa Jinsei
Trillion Game
Rurouni Kenshin: Meiji Kenkaku Romantan - Kyoto Douran
Tono to Inu
Dr. Stone: Stone Wars
Kidou Senshi Gundam: Fukushuu no Requiem
Log Horizon
Log Horizon (2014)
Black Clover (2017)
Mairimashita! Iruma-kun
Mairimashita! Iruma-kun (2021)
Rick and Morty: The Anime
Mahou Tsukai no Yome
Ranma 1/2 (2024)
Bang Dream! Ave Mujica
Gekijouban Boku to Roboco
Ubel Blatt
Nanatsu no Taizai: Mokushiroku no Yonkishi (2024)
Punirunes: Puni 2
Neko ni Tensei Shita Ojisan
Hoshi Furu Oukoku no Nina
Kamonohashi Ron no Kindan Suiri 2nd Season
MF Ghost (2024)
Haigakura
Raise wa Tanin ga Ii
Natsume Yuujinchou Shichi
Seirei Gensouki 2
Pochars
Yarinaoshi Reijou wa Ryuutei Heika o Kouryakuchuu
Tono to Inu
Dragon Ball Daima
Maou 2099
Bananya: Around the World
Kimitachi wa Dou Ikiru ka
Blue Lock vs. U-20 Japan
Meitantei Conan
Thunderbolt Fantasy: Touri-ken Yuuki 4
Gekijouban Blue Lock: Episode Nagi
Mahou Tsukai ni Narenakatta Onnanoko no Hanashi.

Some "relations" are rather odd, eg. this anime Kidou Senshi Gundam: Fukushuu no Requiem | https://anidb.net/anime/18096

is "related" (depth 3) to this anime: Dr. | https://anidb.net/anime/8424

anyone has any idea why? and if I have some settings wrong, that those two are considered related.

I have 2 files attached. log of ban.txt is the current ban, starting with the job "GetAniDBCalendarJob". log of Dr. relation.txt is the log how the series "Dr." got added to the queue.

Also rather strange IMHO is that "Dr." was added with the timestamp 16th of October

[2024-10-16 04:26:43:575] Info|RefreshAnimeStatsJob_AnimeID:8424 > Processing RefreshAnimeStatsJob for Dr.
[2024-10-16 04:26:43:575] Info|GetAniDBImagesJob_AnimeID:8424_OnlyPosters:true > Processing GetAniDBImagesJob for Dr.
[2024-10-16 04:26:43:632] Info|GetAniDBAnimeJob_AnimeID:9719_ForceRefresh:true_DownloadRelations:true_RelDepth:3 > Processing GetAniDBAnimeJob: 9719

why would it sill be in my current queue?

image

2 days later the ban happend.

[2024-10-18 04:42:04:234] Info|GetAniDBImagesJob_AnimeID:6784 > Processing GetAniDBImagesJob for Fate/Stay Night: Unlimited Blade Works
[2024-10-18 04:42:04:285] Info|RenameMoveFileJob_VideoLocalID:4643 > Processing RenameMoveFileJob: Fate⁄Stay Night։ Unlimited Blade Works\[UTW] Fate⁄Stay Night։ Unlimited Blade Works (Blu-ray) [F094815B].mkv
[2024-10-18 04:42:04:359] Info|GetAniDBAnimeJob_AnimeID:7206_ForceRefresh:true_DownloadRelations:true_RelDepth:3 > Processing GetAniDBAnimeJob: 7206
[2024-10-18 04:42:06:111] Warn|Shoko.Server.Providers.AniDB.HTTP.AniDBHttpConnectionHandler > AniDB HTTP Banned!
[2024-10-18 04:42:06:111] Error|Quartz.Core.JobRunShell > Job AniDB.GetAniDBAnimeJob_AnimeID:7206_ForceRefresh:true_DownloadRelations:true_RelDepth:3 threw a JobExecutionException: : Parameters: refire = False, unscheduleFiringTrigger = False, unscheduleAllTriggers = False 
 Quartz.JobExecutionException: One or more errors occurred. (Exception of type 'Shoko.Server.Providers.AniDB.AniDBBannedException' was thrown.)
revam commented 1 month ago

implementation detail of the calendar update job is that it won't add the entry to the queue if it has been updated within 2 days. and related anime won't be updated if they have been recently updated within 24h.

revam commented 3 weeks ago

@Rand-Random tell me if the bans became less frequent or not after updating to 5.0.0.30 please.

Rand-Random commented 3 weeks ago

@revam thanks for your continuous hard work, though it will take some time till I manage to look at it

I am currently occupied with https://github.com/ShokoAnime/ShokoServer/issues/1190

currently switching to shoko relay, there is also 1 issue you may be the person to talk with

Hope its okay, to re-open the issue, why does Shoko match the following:

[2024-10-29 17:05:10:057] Info|SearchTmdbJob_AnimeID:18756 > Linking anime OVA Incha Couple ga You Gal-tachi to Sex Training Suru Hanashi (18756) to show 無責任艦長タイラー OVA (128948)
[2024-10-29 17:05:10:057] Info|Shoko.Server.Providers.TMDB.TmdbLinkingService > Adding TMDB show link: AniDB (AnimeID=18756) → TMDB Show (ID=128948)

I don't see any resemblance.

Rand-Random commented 2 weeks ago

@revam

downloaded the version 5.0.0.43 (d66b282) yesterday and today I am banned

after downloading and "installing" the new version, I re-enabled the options: Always Download Related Anime = true Related Depth = 3

The log is this:

log.txt