Local Cache

[Tyderion]

Cache responses to FILE command locally to reduce load on anidb server

[runecalico]

A local cache would be quite valuable for me, as I frequently have to restart processing a folder due to .. reasons and as the number of files can easily reach 20k+, a restart from scratch will mean a non-trivial amount of time, connections and risk of failure (again).

If possible I would allow for some configuration on "how long" to keep responses in cache, probably don't need any finer granularity beyond "days", as allowing for several days would easily assist with situations where bans occur (as you need to wait 24 hrs between bans)

[Tyderion]

In the vNext branch there currently is a very simplistic implementation of a local cache (with sqlite), but it has no support for an expiration date/duration yet. I can implement an expiration date soon and you can try that branch

[Tyderion]

@runecalico I just merged an update into the vNext branch which will cache data of the FILE command for 30 days (by default). You can configure the time (in days) in your configuration file (see #26)

[runecalico]

@Tyderion I'll try that branch ..

[runecalico]

I got sidetracked, and stopped testing for a bit.

So far the local cache seems to work, I am however uncertain if I'm hitting a bug or just not understanding something in the application ..

Let me explain ..

I am expecting, that as soon as a file is hashed there should be udp api communication requesting info from AniDB, the first such request would trigger authentication and then it should start scheduling commands as more files are hashed ..

Except .. sometimes that doesn't seem to happen .. it's also not something that always occurs so it's hard for me to determine if it's just me not waiting long enough (since I'm testing interactively, I am kinda impatient ) ..

So I spent some time to deploy and configure a new Windows Server 2022 VM for testing, and write scripts that I can kick off that will record logs and am trying (hard) to run them and .. wait a day and see what happens..

For example, here is a run that is still ongoing .. a "small" number of files 1965

Within 2 minutes of starting the application I see the UDP auth log the authentication

2024-10-06 06:21:58:431 -0500 [pool-2-thread-1] TRACE udpapi.UdpApi - 1 commands in queue. Scheduled next command
2024-10-06 06:21:58:431 -0500 [pool-2-thread-1] TRACE udpapi.UdpApi - Command needs login, not logged in, queueing login and command
2024-10-06 06:21:58:446 -0500 [pool-2-thread-1] INFO udpapi.UdpApi - Scheduling command Command(action=AUTH, identifier=auth, tag=null, needsLogin=false, parameters={nat=1, comp=1, protover=3, pass=***, client=aniaddcli, clientver=5, enc=UTF8, user=***}, queryId=1) in 0ms at 2024-10-06T06:21:58.442-05:00[America/Chicago]
2024-10-06 06:21:58:448 -0500 [pool-1-thread-2] DEBUG udpapi.Send - Send command auth
....
2024-10-06 06:21:58:667 -0500 [pool-1-thread-1] INFO udpapi.UdpApi - Received message: auth-1 200 nuU9v xxx.xxx.xxx.xxx:58183 LOGIN ACCEPTED
, cancelling check for lost command.
...
2024-10-06 06:21:58:670 -0500 [pool-1-thread-2] INFO udpapi.ParseReply - Reply: auth-1 200 nuU9v xxx.xxx.xxx.xxx:58183 LOGIN ACCEPTED\n
...
2024-10-06 06:21:58:675 -0500 [pool-1-thread-2] INFO udpapi.UdpApi - Successfully logged in
2024-10-06 06:21:58:676 -0500 [pool-1-thread-2] TRACE udpapi.UdpApi - Send scheduled, not scheduling next command

Now 2hrs later it is still hashing files .. but no udpapi logs since the that initial auth ..

Is this expected?

I'm running this test using Ubuntu 22.04 LTS running under Windows Services Linux (WSL) v2, OpenJDK 21 Headless JRE via a bash script on Windows Server 2022

[Tyderion]

no worries, i also didn't have much time lately (and it doesn't look like i'll have any time until end of year).

Is this expected? No, i'd say something is not working as it should in that run. If I find time i will check it, but i can't promise that i'll have time before december/january.