Closed mattdangerw closed 5 months ago
Exposed base classes. Sets us on a path for better documentation, a more "introspectable" library, and allow sub-classing.
Sure, that's definitely a great addition.
Enable from_preset() on base classes for any subclass. This gives us similar functionality to "auto classes" in huggingface, without the extra overhead of needing a new symbol.
Makes perfect sense as well.
An ability to register new tasks/backbones/tokenizers from out of tree code with keras_nlp.utils.register_preset_class().
Why does this need to be special cased? Wouldn't register_keras_serializable
just work?
ValueError: Preset has type
MistralTokenizer
which is not a a subclass or equal to calling classBertTokenizer
. Callfrom_preset
directly onMistralTokenizer
instead.
To help with debugging, provide all and any bits of relevant information, e.g. Preset 'kaggle://keras/mistral/keras/mistral_7b_en' has type MistralTokenizer ...
-- helps parsing error messages. Otherwise it takes a few seconds of pondering to figure out what "preset" refers to.
Likewise "Call MistralTokenizer.from_preset('kaggle://keras/mistral/keras/mistral_7b_en' )
directly instead" -- now you just need to copy/paste that code.
Why does this need to be special cased? Wouldn't register_keras_serializable just work?
Good question. In short I think it could with keras.saving.get_custom_objects()
. But that would mean occasionally needing to walk the full list of custom objects to discover what is registered. E.g. if we are looking for a classifier that can load a FooBarBackbone
, we would need to walk all custom objects until we find a classifier subclass where classifier.backbone_cls == FooBarBackbone
.
But in practice that's probably negligible compared to the real slowdowns here (allocating memory for a llm, downloading weights, reading weights for a file, etc). I will play around with this.
Reworked out subclassing experience to be as smooth as possible.
keras.saving.register_keras_serializable()
to mark new backbones, etc. Good suggestion.classproperty
annotations, as that is not standard python. (This makes the diff huge, sorry!)bert_base_en
we should reserve only for KerasNLP proper. User models should always show who is publishing with "kaggle://username/model/keras/variant"
.Ok this turned into quite the project, but think this is ready for review.
Most of the diff is from trying to clean up our subclass experience by removing cruft of the subclasses themselves.
This PR grew as I was writing it, and now tries out a number of new things:
from_preset()
on base classes for any subclass. This gives us similar functionality to "auto classes" in huggingface, without the extra overhead of needing a new symbol.keras.saving.register_keras_serializable()
.Colab playground: https://colab.research.google.com/gist/mattdangerw/da885f050fa8baef9b4f9a4ec68d6567/kerasnlp-base-classes.ipynb