[Bug] HybridCache not subscriptable

[hudson-ai]

Build currently fails due to gemma2's usage of a HybridCache which doesn't support tuple slicing like the friendlier DynamicCache.

"Fixing" this issue (just throwing the cache away...) immediately uncovered another one -- the HybridCache has a maximum size. If we don't set this manually, it is set to the sequence length of the first token sequence the model is called with. Trying to do another forward pass with more tokens leads to exceptions deep down inside of gemma's implementation. Current "fix" is to again... throw the cache away.

Hoping for something more elegant. But I don't think this is too insane for now.

Note: now taking advantage of Cache.crop for cache implementations that support it. This should prevent conversion back and forth from the "legacy" cache format that we previously assumed. (Should fix #986).

[codecov-commenter]

:warning: Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 31.25000% with 22 lines in your changes missing coverage. Please review.

Project coverage is 63.53%. Comparing base (917fe35) to head (8d17370).

Files with missing lines	Patch %	Lines
guidance/models/transformers/_transformers.py	31.25%	22 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1047 +/- ## ========================================== - Coverage 71.96% 63.53% -8.43% ========================================== Files 63 63 Lines 4769 4797 +28 ========================================== - Hits 3432 3048 -384 - Misses 1337 1749 +412 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[hudson-ai]

Passing all the model specific tests in the CI Tests workflow. General tests failing due to azurecloud auth issues -- not sure if I am able to rerun with the right perms. But I believe all is fine and dandy with the PR.

[hudson-ai]

Changes since submitting PR:

• When we overflow the size of the cache, reallocate a cache with double the size rather than deleting the old cache, which would cause the next forward pass to only create a cache that is "just big enough".
  • To be seen whether we need to be a bit more conservative here to avoid OOM issues when the sequence length is long.
  • We only do this for Static and Hybrid caches. Other cache types will be deleted until we write implementations for doubling their sizes. We give a warning in this case.
• When we need to reset the cache due to backtracking, we now call Cache.reset if possible in order to avoid reallocating the cache. This also prevents the cache-doubling from resetting.

[hudson-ai]

@paulbkoch any feedback?