search

iree-org / iree

A retargetable MLIR-based machine learning compiler and runtime toolkit.
http://iree.dev/
Apache License 2.0
2.84k stars 611 forks source link

Memory use for MobileNetV2 UINT8 2-3x larger than TFlite #10293

Open mariecwhite opened 2 years ago

mariecwhite commented 2 years ago

For MobileNetV2 UINT8, peak resident set size on IREE is 2-3x larger on both mobile and server CPU compared with TFLite. For all other models under test, IREE memory use is usually 1-2x smaller than TFLite.

hanhanW commented 2 years ago

What compilation flags do you use?

ScottTodd commented 2 years ago

In addition, please fill in the "version information" field when filing issues like this. Code size, performance, memory use, and program compatibility can all change between commits/versions, and issues are much more difficult to triage and resolve without knowing what version of the software was used when the issue was noticed. If we're missing an easy way to get version information with whatever tools you are using, we should also be fixing that.

mariecwhite commented 2 years ago

Thanks Scott. Git commit hash is 223d02f16.

Compiler flags for mobile:

"${IREE_COMPILE_PATH}" \
    --iree-input-type=tosa \
    --iree-hal-target-backends=llvm-cpu \
    --iree-llvm-target-triple=aarch64-none-linux-android29 \
    --iree-llvm-debug-symbols=false \
    --iree-vm-bytecode-module-strip-source-map=true \
    --iree-vm-emit-polyglot-zip=false \
    "${IREE_MODEL_DIR}/${MODEL_NAME}.mlir" \
    --o "${IREE_MODEL_DIR}/llvm-cpu/${MODEL_NAME}.vmfb" || true

  echo -e "\tCompiling ${MODEL_NAME}_mmt4d.vmfb for aarch64..."
  "${IREE_COMPILE_PATH}" \
    --iree-input-type=tosa \
    --iree-hal-target-backends=llvm-cpu \
    --iree-llvm-target-triple=aarch64-none-linux-android29 \
    "--iree-flow-mmt4d-target-options=arch=aarch64 features=+dotprod" \
    --iree-llvm-target-cpu-features=+dotprod \
    --iree-llvm-debug-symbols=false \
    --iree-vm-bytecode-module-strip-source-map=true \
    --iree-vm-emit-polyglot-zip=false \
    "${IREE_MODEL_DIR}/${MODEL_NAME}.mlir" \
    --o "${IREE_MODEL_DIR}/llvm-cpu/${MODEL_NAME}_mmt4d.vmfb" || true

  echo -e "\tCompiling ${MODEL_NAME}_im2col_mmt4d.vmfb for aarch64..."
  "${IREE_COMPILE_PATH}" \
    --iree-input-type=tosa \
    --iree-hal-target-backends=llvm-cpu \
    --iree-llvm-target-triple=aarch64-none-linux-android29 \
    "--iree-flow-mmt4d-target-options=arch=aarch64 features=+dotprod" \
    --iree-llvm-target-cpu-features=+dotprod \
    --iree-flow-enable-conv-img2col-transform \
    --iree-llvm-debug-symbols=false \
    --iree-vm-bytecode-module-strip-source-map=true \
    --iree-vm-emit-polyglot-zip=false \
    "${IREE_MODEL_DIR}/${MODEL_NAME}.mlir" \
    --o "${IREE_MODEL_DIR}/llvm-cpu/${MODEL_NAME}_im2col_mmt4d.vmfb" || true

Flags for desktop:

echo "Compiling ${MODEL_NAME}.vmfb for llvm-cpu..."
  "${IREE_COMPILE_PATH}" \
    --iree-input-type=tosa \
    --iree-llvm-target-cpu-features=host \
    --iree-hal-target-backends=llvm-cpu \
    --iree-llvm-debug-symbols=false \
    --iree-vm-bytecode-module-strip-source-map=true \
    --iree-vm-emit-polyglot-zip=false \
    "${IREE_MODEL_DIR}/${MODEL_NAME}.mlir" \
    --o "${IREE_MODEL_DIR}/llvm-cpu/${MODEL_NAME}.vmfb"
 done