search

ShokoAnime / ShokoServer

Repository for Shoko Server.
https://shokoanime.com/
MIT License
407 stars 74 forks source link

AniDB Ban prevention system - need little tweak #517

Closed bigretromike closed 7 years ago

bigretromike commented 7 years ago

While importing collection to fresh db in current master after some time I got hit by BAN. Its not instant ban as the current system does decent job in preventing it . Maybe its because of few hours of asking anidb or one more second per request... dunno . Just wanted to inform that current system is a god send compare to pre 3.7 ones but still some tweaks are needed.

[2016-11-18 15:34:45:676] Info|AniDBHelper.logoutTimer_Elapsed => AniDBCommand_Ping.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_COMMAND: PING
[2016-11-18 15:34:45:863] Info|AniDBHelper.logoutTimer_Elapsed => AniDBCommand_Ping.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE in 187.2003 ms - 300 PONG

[2016-11-18 15:35:35:752] Info|AniDBHelper.logoutTimer_Elapsed => AniDBCommand_Ping.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_COMMAND: PING
[2016-11-18 15:35:35:955] Info|AniDBHelper.logoutTimer_Elapsed => AniDBCommand_Ping.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE in 202.8003 ms - 300 PONG

[2016-11-18 15:35:54:955] Info|BackgroundWorker.OnDoWork => CommandProcessorGeneral.workerCommands_DoWork => CommandRequest_ProcessFile.ProcessCommand Processing File: 7506
[2016-11-18 15:35:54:971] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_COMMAND: FILE size=236093750&ed2k=DB15775293CAE6DFD84032F4FC3D686B&fmask=7FF8FFF9FE&amask=0000FCC0&s=LhRs4
[2016-11-18 15:35:55:111] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE_TRUNC in 140.4003ms - 220 FILE
175484|649|7055|203|208222602||0|65|236093750|db15775293cae6dfd84032f4fc3d686b|fccd91671fb0d682e318a1f6034a3a12|edfb152af811d1fe2aa27e40570c5150e0cde465|0cf62303|high|unknown|Vorbis (Ogg Vorbis)'Vorbis (Ogg Vorbis)|101'103|XviD|954|640x480|mkv|japanese'english|english'english'english|1624||1019088000|Kamyla - 1 - Volume 1 - [HL](0cf62303).mkv|2|0|0|0||||1|Volume 1|||111|1|Hentai Legacy|HL

[2016-11-18 15:35:56:562] Info|BackgroundWorker.OnDoWork => CommandProcessorGeneral.workerCommands_DoWork => CommandRequest_ProcessFile.ProcessCommand Processing File: 7507
[2016-11-18 15:36:01:071] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_COMMAND: FILE size=246821023&ed2k=F00356AB231D7D9106323429F9669CBD&fmask=7FF8FFF9FE&amask=0000FCC0&s=LhRs4
[2016-11-18 15:36:01:258] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE_TRUNC in 187.2004ms - 220 FILE
175485|649|7056|203|208222604||0|65|246821023|f00356ab231d7d9106323429f9669cbd|02b185e1a6ab0d770d991fdd0db45796|bf791f1b4307758f3225240cd1fc23f87f11f60e|e4562d6b|high|unknown|Vorbis (Ogg Vorbis)'Vorbis (Ogg Vorbis)|78'80|XviD|1057|640x480|mkv|japanese'english|english|1618||1029542400|Kamyla - 2 - Volume 2 - [HL](e4562d6b).mkv|2|0|0|0||||2|Volume 2|||411|1|Hentai Legacy|HL

[2016-11-18 15:36:02:178] Info|BackgroundWorker.OnDoWork => CommandProcessorGeneral.workerCommands_DoWork => CommandRequest_ProcessFile.ProcessCommand Processing File: 7508
[2016-11-18 15:36:02:178] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_COMMAND: FILE size=568757911&ed2k=FB3095CA3FFDCFD2B29C8534CBDD243D&fmask=7FF8FFF9FE&amask=0000FCC0&s=LhRs4
[2016-11-18 15:36:02:397] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE_TRUNC in 218.4004ms - 220 FILE
1089875|8678|133288|11119|208218633||0|129|568757911|fb3095ca3ffdcfd2b29c8534cbdd243d|1ea45f5f3d9477fd0e9249c91db79a4e|017d018cceaf09fc31808a8a5cfff74daf52689b|946beb96|very high|DVD|AC3|192|H264/AVC|2248|1024x576|mkv|japanese|english|1822||1321574400|Aku no Onna Kanbu - 1 - Volume 1 - [Anim8S](946beb96).mkv|2|0|0|0||||1|Volume 1|Shouaku no Chikara: Kakusei|???? ??|356|3|Anim8S|Anim8S

[2016-11-18 15:36:02:490] Info|AniDBHelper.GetAnimeInfoHTTP => AniDBHTTPCommand_GetFullAnime.Process => AniDBHTTPHelper.GetAnimeXMLFromAPI AniDB HTTP - Getting anime XML data from AniDB: 8678
[2016-11-18 15:36:07:482] Info|AniDBHelper.GetAnimeInfoHTTP => AniDBHTTPCommand_GetFullAnime.Process => AniDBHTTPHelper.GetAnimeXMLFromAPI AniDB HTTP - Got anime XML Data From AniDB: 8678 - 4992.0088 - <?xml version="1.0" encoding="UTF-8"?><anime id="8678" restricted="true">
<type>OVA</type>
<episodec
[2016-11-18 15:36:08:044] Info|BackgroundWorker.OnDoWork => CommandProcessorImages.workerCommands_DoWork => CommandRequest_DownloadImage.ProcessCommand Processing CommandRequest_DownloadImage: 972
[2016-11-18 15:36:08:824] Info|BackgroundWorker.OnDoWork => CommandProcessorImages.workerCommands_DoWork => CommandRequest_DownloadImage.ProcessCommand Image downloaded: C:\ProgramData\ShokoServer\images\AniDB\86\60826.jpg
[2016-11-18 15:36:08:886] Info|BackgroundWorker.OnDoWork => CommandProcessorImages.workerCommands_DoWork => CommandRequest_DownloadImage.ProcessCommand Processing CommandRequest_DownloadImage: 3123
[2016-11-18 15:36:09:027] Info|BackgroundWorker.OnDoWork => CommandProcessorImages.workerCommands_DoWork => CommandRequest_DownloadImage.ProcessCommand Image downloaded: C:\ProgramData\ShokoServer\images\AniDB_Creator\29\39498.jpg
[2016-11-18 15:36:09:089] Info|BackgroundWorker.OnDoWork => CommandProcessorImages.workerCommands_DoWork => CommandRequest_DownloadImage.ProcessCommand Processing CommandRequest_DownloadImage: 15259
[2016-11-18 15:36:09:510] Info|BackgroundWorker.OnDoWork => CommandProcessorImages.workerCommands_DoWork => CommandRequest_DownloadImage.ProcessCommand Image downloaded: C:\ProgramData\ShokoServer\images\AniDB_Char\38\77326.jpg
[2016-11-18 15:36:09:604] Info|BackgroundWorker.OnDoWork => CommandProcessorImages.workerCommands_DoWork => CommandRequest_DownloadImage.ProcessCommand Processing CommandRequest_DownloadImage: 15260
[2016-11-18 15:36:09:869] Info|BackgroundWorker.OnDoWork => CommandProcessorImages.workerCommands_DoWork => CommandRequest_DownloadImage.ProcessCommand Image downloaded: C:\ProgramData\ShokoServer\images\AniDB_Char\42\84741.jpg
[2016-11-18 15:36:50:866] Info|AniDBHelper.logoutTimer_Elapsed => AniDBCommand_Ping.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_COMMAND: PING
[2016-11-18 15:36:51:022] Info|AniDBHelper.logoutTimer_Elapsed => AniDBCommand_Ping.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE in 156.0003 ms - 300 PONG

[2016-11-18 15:37:40:942] Info|AniDBHelper.logoutTimer_Elapsed => AniDBCommand_Ping.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_COMMAND: PING
[2016-11-18 15:37:41:098] Info|AniDBHelper.logoutTimer_Elapsed => AniDBCommand_Ping.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE in 156.0003 ms - 300 PONG

[2016-11-18 15:38:07:041] Info|BackgroundWorker.OnDoWork => CommandProcessorGeneral.workerCommands_DoWork => CommandRequest_ProcessFile.ProcessCommand Processing File: 7509
[2016-11-18 15:38:07:041] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_COMMAND: FILE size=517678076&ed2k=CDBD01F25793AFF803CBFF6C36065EF1&fmask=7FF8FFF9FE&amask=0000FCC0&s=LhRs4
[2016-11-18 15:38:07:244] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE_TRUNC in 202.8003ms - 220 FILE
1089953|8678|135408|11119|208378538||0|133|517678076|cdbd01f25793aff803cbff6c36065ef1|972d476304822397122a5b5e01a2e77a|21d2535f89100b56904dab8fc618a8e44c9757aa|d2776b89|very high|DVD|AC3|192|H264/AVC|2077|1024x576|mkv|japanese|english|1780||1332460800|Aku no Onna Kanbu - 2 - Licentious Beast`s Discipline - [Anim8S](d2776b89).mkv|2|0|0|0||||2|Licentious Beast`s Discipline|Injuu no Shitsuke|??????|356|3|Anim8S|Anim8S

[2016-11-18 15:38:08:601] Info|BackgroundWorker.OnDoWork => CommandProcessorGeneral.workerCommands_DoWork => CommandRequest_ProcessFile.ProcessCommand Processing File: 7510
[2016-11-18 15:38:13:109] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_COMMAND: FILE size=288587345&ed2k=1E65C52B2775A61E1BFC8CBEFE7391D8&fmask=7FF8FFF9FE&amask=0000FCC0&s=LhRs4
[2016-11-18 15:38:13:328] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE_TRUNC in 218.4004ms - 220 FILE
290610|2024|20061|3085|208268667||0|65|288587345|1e65c52b2775a61e1bfc8cbefe7391d8|b3941821ecb0856ab7304493e505ce7c|04e5bb5ce77c8c943cc7df22413510ded714a371|d220fa4a|high|DVD|Vorbis (Ogg Vorbis)'Vorbis (Ogg Vorbis)|104'100|H264/AVC|1097|640x480|mkv|japanese'english|english|1768||1085702400|Mashou no Kao - 1 - Enslaved - [KH](d220fa4a).mkv|2|0|0|0||||1|Enslaved|Toriko|?(???)|140|1|KHMER-HENTAI|KH

[2016-11-18 15:38:13:437] Info|AniDBHelper.GetAnimeInfoHTTP => AniDBHTTPCommand_GetFullAnime.Process => AniDBHTTPHelper.GetAnimeXMLFromAPI AniDB HTTP - Getting anime XML data from AniDB: 2024
[2016-11-18 15:38:14:045] Info|AniDBHelper.GetAnimeInfoHTTP => AniDBHTTPCommand_GetFullAnime.Process => AniDBHTTPHelper.GetAnimeXMLFromAPI AniDB HTTP - Got anime XML Data From AniDB: 2024 - 608.4011 - <error>Banned</error>
[2016-11-18 15:43:16:452] Info|AniDBHelper.ForceLogout => AniDBCommand_Logout.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_COMMAND: LOGOUT s=LhRs4
[2016-11-18 15:43:16:592] Info|AniDBHelper.ForceLogout => AniDBCommand_Logout.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE in 140.4003 ms - 203 LOGGED OUT

[2016-11-18 16:11:32:211] Info|BackgroundWorker.OnDoWork => CommandProcessorGeneral.workerCommands_DoWork => CommandRequest_ProcessFile.ProcessCommand Processing File: 7511
[2016-11-18 16:11:32:460] Info|AniDBHelper.Login => AniDBCommand_Login.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE in 249.6004 ms - 200 kys8W LOGIN ACCEPTED

[2016-11-18 16:11:39:184] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_COMMAND: FILE size=629250132&ed2k=356D745CF0E9AA6252BF50FBD2332B02&fmask=7FF8FFF9FE&amask=0000FCC0&s=kys8W
[2016-11-18 16:11:39:434] Info|AniDBHelper.GetFileInfo => AniDBCommand_GetFileInfo.Process => AniDBUDPCommand.ProcessCommand AniDB UDP - UDP_RESPONSE_TRUNC in 249.6005ms - 220 FILE
1112649|8914|136748|7684|212081795||0|0|629250132|356d745cf0e9aa6252bf50fbd2332b02|66087264f7549c972d929144ab91d237|c2cc50eff2554cc9f6a3d79a04bfdd56e93d2289|a4726398|high|Blu-ray|AAC|172|H264/AVC|2817|1920x1080|mkv|japanese|english|1681||1343347200|Code Geass: Hangyaku no Lelouch - Nunnally in Wonderland - 1 - OVA - [OZC](a4726398).mkv|2|0|0|0||||1|OVA|||380|8|OZC Anime|OZC

[2016-11-18 16:11:39:543] Info|AniDBHelper.GetAnimeInfoHTTP => AniDBHTTPCommand_GetFullAnime.Process => AniDBHTTPHelper.GetAnimeXMLFromAPI AniDB HTTP - Getting anime XML data from AniDB: 8914
[2016-11-18 16:11:40:151] Info|AniDBHelper.GetAnimeInfoHTTP => AniDBHTTPCommand_GetFullAnime.Process => AniDBHTTPHelper.GetAnimeXMLFromAPI AniDB HTTP - Got anime XML Data From AniDB: 8914 - 608.401 - <error>Banned</error>
bigretromike commented 7 years ago

Oh, also the BAN notification vanish after some time - which looks like the ban is no more

bigretromike commented 7 years ago

@hidden4003 you did great job with current ban prevention system, could you make it perfect ?:-)

hidden4003 commented 7 years ago

HTTP ban is unavoidable I'm afraid as there is no authentication like with UDP so essentially they only are able to control by IP.

bigretromike commented 7 years ago

Oh so http and UDP are different. Maybe slowing down http traffic would help ?

hidden4003 commented 7 years ago

Read this comment: https://github.com/japanesemediamanager/ShokoServer/issues/379#issuecomment-256856048 I don't think there is a solution unless anidb allows to login for http api as well.

bigretromike commented 7 years ago

I don't see anything that login would resolve other that they use flood defense and lowering number of packets would 'maybe' not trigger it. But you are the expert of that matter her so I leave it close as is. thanks for info

CuddleBear92 commented 7 years ago

what i personally dont get on this is that there never was an http ban til these newest updates. i could run the queue for hours on hours before getting an udp ban at times. getting http banned within the half hour almost all the time tho.

so are we getting more info from anidb now because we are also using http? cause it wasnt like this before at all.

the server is practically useless for me in so many ways because of these bans.

ElementalCrisis commented 7 years ago

I think your misunderstanding the issue, something changed on AniDB's side which they have not been able to pinpoint thats causing the issue.

bigretromike commented 7 years ago

for me its not "something" but more of flood-prevention on firewall - also if thats the case the easiest way would be "tell use the limit anidb" and we tweak our script so it wont flood that much or just randomly put values and testing it against flood. so more flood for them as more people test the script :-)

Also there been so much time that there could be a change made to our system that made it work faster/stronger, and in long run (few hours) that 1-2 query per second or minut could lead to flood detection.