[compiler] Support large models

[seanshpark]

Issue to gather issues, task for large models, such as

• flatbuffer size > 2G
• quantize < 8 bit
• ...

---

• support 4bit weights: #12717
• quantize with fake-quantize: #12738
• end-to-end validation for int4: #12797
• onnx with INT16: #12816

---

things to do with our LLM model

• convert model to circle -> done with internal onnx2circle
• optimize circle model -> conversion seems ok
• ...

[hseok-oh]

We may need to support external data file on circle spec (like onnx spec) to save large weight.

[seanshpark]

We may need to support external data file

circle schema from 0.7 has this in changes from tflite

(for table Operator)
  // When an op is using custom_options in a model that is larger than 2GB, then
  // we instead use the following attributes to find the buffer location which
  // is stored outside of flatbuffers, the offset is calculated relative to the
  // beginning of the file and is only valid if > 1
  large_custom_options_offset: ulong;
  large_custom_options_size: ulong;

(for table Buffer)
  // In a model that is larger than 2GB, then buffers instead uses the following
  // attributes to find stored data, which is outside of flatbuffers
  // the offset is calculated relative to the beginning of the file and is only
  // valid if > 1.
  offset: ulong;
  size: ulong;

I need to look inside tflite implementation but, it seems large data are added at the end of the file, flat buffer managed part, as single file.

[seanshpark]

Now it's time to support this.

modules that may need revision; progress with draft changes

• [x] res of tfl schema 2.16.1
• [x] mio-tflite2121
• [x] mio-circle08
• [x] tflchef
• [x] tflite2circle
• [x] luci/import
• [x] luci/export
• [x] circledump
• [x] tfldump
• [ ] tfl-inspect
• [x] circle-inspect
• [ ] ...

modules using luci/import, luci/export

• [x] circle-eval-diff
• [x] circle-execution-plan
• [x] circle-interpreter
• [x] circle-mqpsolver
• [x] circle-opselector
• [ ] circle-partitioner
• [ ] circle-quantizer
• [ ] circle2circle
• [ ] common-artifacts
• [x] dalgona
• [ ] embedded-import-value-test
• [x] fme-apply
• [ ] fme-detect
• [ ] luci/tester
• [ ] luci-eval-driver
• [ ] minmax-embedder
• [ ] q-implant
• [x] record-minmax

how to test this?

• [x] tflchef to produce ~huge model~ model with new offset/size
  • [x] use TF2.16.1 schema in res
  • [x] check huge tflite is produced
  • [x] check tfldump
  • [x] temporary tflite2circle to produce ~huge circle~ new offset/size circle
  • [x] check circledump

issues

• TF 2.12.1 doesn't support size > 2G --> we need to upgrade TF some day

[seanshpark]

How tflite does

tensorflow/lite/core/model_builder.cc

  // Only run validator on models less than 2GB
  if (allocation->bytes() < flatbuffer_size_max) {
    flatbuffers::Verifier base_verifier(
        reinterpret_cast<const uint8_t*>(allocation->base()),
        allocation->bytes());
    if (!VerifyModelBuffer(base_verifier)) {
      TF_LITE_REPORT_ERROR(error_reporter,
                           "The model is not a valid Flatbuffer buffer");
      return nullptr;
    }
  }

tensorflow/compiler/mlir/lite/flatbuffer_export.cc

  // check if Flatbuffer builder can no longer hold the given amount of the data
  inline bool IsModelBiggerThan2GB(const uint64_t data_size) {
    return data_size > flatbuffer_size_max - builder_.GetSize();
  }

[seanshpark]

How to export ?

• go with two phase
• first phase is to save normal model, and check if size > 2GB, set require_use_buffer_offset flag and exit with error
• if the first phase exits with error and require_use_buffer_offset flag set, run second phase with use_buffer_offset set to store all Buffers into outside of FB area

[seanshpark]

As of testing, let's use new Buffer.offset if size > 2MB Or as an option, we can add a flag in tflite recipe to use Buffer.offset

[seanshpark]

New fields to support

table Operator {
...
  large_custom_options_offset: ulong;
  large_custom_options_size: ulong;
...
}

table Buffer {
...
  offset: ulong;
  size: ulong;
}

[chunseoklee]

large_custom_options_offset: ulong; large_custom_options_size: ulong;

we need this option from the start ? IMHO, this is a kind of custom something.

[seanshpark]

this is a kind of custom something.

Yes, for CircleCustom.. We don't need to implement this at this moment :)

[seanshpark]

Name for this feature... (I thought I've left a mention about this but now I can't find it...)

Extended Buffer