search

knuddelsgmbh / jtokkit

JTokkit is a Java tokenizer library designed for use with OpenAI models.
https://jtokkit.knuddels.de/
MIT License
518 stars 38 forks source link

Add LazyEncodingRegistry #24

Closed blackdiz closed 1 year ago

blackdiz commented 1 year ago

A lazy initialization implementation of EncodingRegistry. It does not register any encoding until either the getEncoding(EncodingType) or getEncoding(String) method is called. When one of these methods is called, the requested EncodingType is registered.

blackdiz commented 1 year ago

Hi, @tox-p, in addition to the suggestions you provided, I have also created a new BaseEncodingRegistryTest to reuse tests for both LazyEncodingRegistryTest and DefaultEncodingRegistryTest.