Open seanshpark opened 6 months ago
We may need to support external data file on circle spec (like onnx spec) to save large weight.
We may need to support external data file
circle schema from 0.7 has this in changes from tflite
(for table Operator)
// When an op is using custom_options in a model that is larger than 2GB, then
// we instead use the following attributes to find the buffer location which
// is stored outside of flatbuffers, the offset is calculated relative to the
// beginning of the file and is only valid if > 1
large_custom_options_offset: ulong;
large_custom_options_size: ulong;
(for table Buffer)
// In a model that is larger than 2GB, then buffers instead uses the following
// attributes to find stored data, which is outside of flatbuffers
// the offset is calculated relative to the beginning of the file and is only
// valid if > 1.
offset: ulong;
size: ulong;
I need to look inside tflite implementation but, it seems large data are added at the end of the file, flat buffer managed part, as single file.
Now it's time to support this.
modules that may need revision; progress with draft changes
modules using luci/import, luci/export
how to test this?
issues
How tflite does
tensorflow/lite/core/model_builder.cc
// Only run validator on models less than 2GB
if (allocation->bytes() < flatbuffer_size_max) {
flatbuffers::Verifier base_verifier(
reinterpret_cast<const uint8_t*>(allocation->base()),
allocation->bytes());
if (!VerifyModelBuffer(base_verifier)) {
TF_LITE_REPORT_ERROR(error_reporter,
"The model is not a valid Flatbuffer buffer");
return nullptr;
}
}
tensorflow/compiler/mlir/lite/flatbuffer_export.cc
// check if Flatbuffer builder can no longer hold the given amount of the data
inline bool IsModelBiggerThan2GB(const uint64_t data_size) {
return data_size > flatbuffer_size_max - builder_.GetSize();
}
How to export ?
require_use_buffer_offset
flag and exit with errorrequire_use_buffer_offset
flag set, run second phase with use_buffer_offset
set to store all Buffers into outside of FB areaAs of testing, let's use new Buffer.offset
if size > 2MB
Or as an option, we can add a flag in tflite recipe to use Buffer.offset
New fields to support
table Operator {
...
large_custom_options_offset: ulong;
large_custom_options_size: ulong;
...
}
table Buffer {
...
offset: ulong;
size: ulong;
}
large_custom_options_offset: ulong; large_custom_options_size: ulong;
we need this option from the start ? IMHO, this is a kind of custom something.
this is a kind of custom something.
Yes, for CircleCustom.
. We don't need to implement this at this moment :)
Name for this feature... (I thought I've left a mention about this but now I can't find it...)
Extended Buffer
Issue to gather issues, task for large models, such as
things to do with our LLM model