Anthropic and LlamaCPP not caching

[daniel-luper]

Hi! Love this project btw. I'm wanting to contribute, and while exploring the add-on I found this bug (maybe feature 👀?? I guess some people might want a unified cache across all models but it's not what I would expect).

Description

• Expected behavior: when a DescriptionService X processes an image that hasn't been cached by X, then the image/description pair gets added to X's cache. When the image has been cached by X (not Y, otherwise you wouldn't have a way of generating a different description using a different model), then the cached description gets returned.
• Actual behavior: the OpenAI models write to cache as expected but the Anthropic and LlamaCPP description services never write to cache. Additionally, there's a single cache rather than a cache per model, so a cached GPT4 description can't get replaced with a GPT4O description.

Root Cause

• Not writing to cache: these lines are missing from the Anthropic and LlamaCPP description services:

  if cache_descriptions:
  cache.read_cache()
  cache.cache[base64_image] = content
  cache.write_cache()

• Loading another model's cache: the cache's key is just a base64 encoding of the image. The key should also include the model used to generate the description.

Possible Solution

To improve readability and cohesiveness while eliminating duplicate code, move the caching code out of the description services and into a new class (e.g. CachedDescriptionService). The new class is a DescriptionService that can wrap any other DescriptionService Y, and adds caching functionality to Y (see Decorator pattern).

Then, use separate files for seperate models instead of one unified cache across all models. This would have the added benefit of helping prevent the memory error in #26.

---

I can implement this and make a PR. What do you think? Do you have any questions?

[cartertemm]

Hey @daniel-luper,

Great catch, and your solution seems well thought out. I especially like the idea of a caching decorator for the sake of cleanliness. Ideally, the current cache would be preserved as a fallback so that users don't have to delete it upon upgrading to the new version. That said, I don't think it's mandatory based on the number of people using this feature.

Feel free to start working on this. If you'd like help along the way, don't hesitate to reach out!

[daniel-luper]

Thanks! I'll go ahead and get started on it.

[daniel-luper]

@cartertemm how would you recommend reading and writing logs? I tried using the logger from import logging and checked the %TMP%/nvda.log file, but my logs didn't get written. I already made sure to set the NVDA logging level from the General settings to "debug".

[cartertemm]

You can always use NVDA's logHandler, as such:

import logHandler

nvda_log = logHandler.log
nvda_log.info("Test msg")

HTH