search

onnx / onnx-mlir

Representation and Reference Lowering of ONNX Models in MLIR Compiler Infrastructure
Apache License 2.0
766 stars 320 forks source link

An inference driver in a local env without onnx-mlir #3005

Open chentong319 opened 5 days ago

chentong319 commented 5 days ago

A model can be compiled directly with the onnx-mlir docker image. A python script using subprocess is provided at onnx-mlir/docker/onnx-mlir.py. I also tried to do the same task with docker package to hide the details of start docker and mount the disk. ( will be checked in).

Now many of us run into the issue: how to use the compiled result, say model.so, to performance inference?

Inside the onnx-mlir container, we use the script PyRuntime.py to create and run the ExecutionSession. This pyruntime.py relies on the built library PyRuntimeC..so, which convert python tensor to c array with pybind11. If I tried to the compiled PyRuntimeC..so, along with PyRuntime.py from docker image into my local env. However, the PyRuntimeC.*.so cannot be imported by PyRuntime.py in the local env because of the mismatch of the python version between the python inside the docker and the one in the local env.

ldd lib/PyRuntimeC.cpython-310-s390x-linux-gnu.so 
    linux-vdso64.so.1 (0x000003ffd1ffe000)
    libpython3.10.so.1.0 => not found
    libstdc++.so.6 => /lib/s390x-linux-gnu/libstdc++.so.6 (0x000003ff9a200000)
    libm.so.6 => /lib/s390x-linux-gnu/libm.so.6 (0x000003ff9a100000)
    libgcc_s.so.1 => /lib/s390x-linux-gnu/libgcc_s.so.1 (0x000003ff9a500000)
    libc.so.6 => /lib/s390x-linux-gnu/libc.so.6 (0x000003ff99f00000)
    /lib/ld64.so.1 (0x000003ff9a880000)

Q1: can we make the two python versions the same? I do not know the answer. Even if it solves the problem, this solution has obvious limitation.

Then I tried to build the PyRuntimeC.*.so locally. But the current code for ExecutionSession is using llvm::sys::DynamicLibrary to handle the model.so. We need to break that dependence to llvm. We will keep the pybind11 locally. Q2. Can we make this approach work?

@gongsu832 suggested to use cython to break all the dependences. Here is an educational link he provided.

tungld commented 4 days ago

Currently PyRuntimeC..so is built during onnx-mlir building. Can we separate them? say, PyRuntimeC.so will be built later when we build the python interface package. By that way, the python package would use the current python verson in the user machine.

AlexandreEichenberger commented 3 days ago

Exactly, as we discussed with Tong, the idea would be to split the Runtime dir into a RuntimeSupport and RuntimeForInference with all the code that is needed for the Python interface to go in the RuntimeForInference and the stuff that is helping our binary run (e.g. compute random, unique maps..., print tensors...) goes into the RuntimeSupport. That later one can use LLVM galore, as it is compiled with onnx-mlir. RuntimeForInference, however, should be ideally gcc/clang agnostic, not dependent on anything LLVM/MLIR so that it can be build "from scratch" in "any" environment without dependences on anything but standard packages. It would still be built with onnx-mlir, but it can also be built without it.

Note the names are my own, and can be changed as you may see fit.