dottxt-ai / outlines

Structured Text Generation
https://dottxt-ai.github.io/outlines/
Apache License 2.0
8.97k stars 451 forks source link

Make `cache` support methods #1145

Open brandonwillard opened 1 month ago

brandonwillard commented 1 month ago

Currently, the cache function doesn't truly support methods (see #1129). Let's consider

brandonwillard commented 1 month ago

I noticed from the helpful test case in #1129—and the relevant vllm code—that the cache decorator is being directly applied to the function object, so we may not even be able to tell when a function is a method in order to give a warning. We might be able to handle the case of a method descriptor object, but that wouldn't have helped in the examples above.

In other words, there's probably not a general way to warn people that they could be doing something undesirable for caching. Because of that, we might want to make the caching utilities private and leave it up to the user to determine how caching should work, since all we can really guarantee is that underlying objects are serializable in a way that can work for caching.