search

intel-analytics / ipex-llm

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, MiniCPM, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, GraphRAG, DeepSpeed, vLLM, FastChat, Axolotl, etc.
Apache License 2.0
6.63k stars 1.26k forks source link

Can Orca TF2Estimator accept tf graph defined model? #3228

Open hkvision opened 3 years ago

hkvision commented 3 years ago

Currently I only see TF2Estimator.from_keras, want to ask is tf graph defined model supported for TF2? Especially we want to run dien: https://github.com/alibaba/ai-matrix/blob/master/macro_benchmark/DIEN_TF2/script/model.py

yangw1234 commented 3 years ago

looks like the model is defined in tf1 api

hkvision commented 3 years ago

Seems yes, but it can run with tf2. Can our TF2Estimator accept this? Since we need the internal tf2 whl for performance benefits and thus can't use tf1...

yangw1234 commented 3 years ago

Seems yes, but it can run with tf2. Can our TF2Estimator accept this? Since we need the internal tf2 whl for performance benefits and thus can't use tf1...

How about 1.15.0up3?

jason-dai commented 3 years ago

Seems yes, but it can run with tf2. Can our TF2Estimator accept this? Since we need the internal tf2 whl for performance benefits and thus can't use tf1...

So it will not be distributed? Or maybe distributed inference only?

hkvision commented 3 years ago

As you suggest, we can either:

LongxingTan commented 3 years ago

Seems yes, but it can run with tf2. Can our TF2Estimator accept this? Since we need the internal tf2 whl for performance benefits and thus can't use tf1...

@hkvision , so it works fine with tf2 saved model? It works the same as tf1 model, or any special tutorial for tf2 available? Thanks

jason-dai commented 3 years ago

Seems yes, but it can run with tf2. Can our TF2Estimator accept this? Since we need the internal tf2 whl for performance benefits and thus can't use tf1...

@hkvision , so it works fine with tf2 saved model? It works the same as tf1 model, or any special tutorial for tf2 available? Thanks

See https://analytics-zoo.readthedocs.io/en/latest/doc/Orca/QuickStart/orca-tf2keras-quickstart.html Currently it supports tf.keras, and loading from checkpoints.

zhanglirong1999 commented 2 years ago

I tried to convert the graph model to orca-supported keras model. But in the end it didn't work out. The reasons are as follows:

  1. I refer to the following link to try to convert the graph(frozen model) to a saved model https://stackoverflow.com/questions/44329185/convert-a-graph-proto-pb-pbtxt-to-a-savedmodel-for-use-in-tensorflow-serving-o/44329200#44329200

But in the end only get a metaGraph. Metagraph is still a graph and its variable folder is empty. And metagraphs could not be further converted to the models we needed, such as keras.

  1. And then I refer to the following link to try to add lambda layer in keras and wrap my model in lambda. https://stackoverflow.com/questions/54122211/tensorflow-graph-to-keras-model

Unfortunately, lambda does not support output as a graph model. So it's hard to use lambda layer to achieve it.

  1. I found an issue in keras: https://github.com/keras-team/keras/issues/5273 Found a comment from one of the builders of keras fchollet, he said:" Fundamentally, you cannot "turn an arbitrary TensorFlow checkpoint into a Keras model". What you can do, however, is build an equivalent Keras model then load into this Keras model the weights contained in a TensorFlow checkpoint that corresponds to the saved model"

Therefore, after trying the above two methods and combining fchollet's comments, it can be seen that it is more difficult to convert the graph to a keras model, and the graph model like Dien does not provide the source code, it is hard to build an equivalent Keras model then load into this Keras model since we don't have source code and just have a graph model.