Optimum uses tensorflow by default when you have both torch and tf

barathbheeman commented 1 year ago

System Info

optimum.version.__version__ : 1.7.1
python version: 3.8
system: docker running nvidia/cuda:11.7.0-devel-ubuntu20.04
torch.__version__: 1.13.1+cu117
tensorflow.__version__: 2.11.0
transformers.__version__: 4.26.1

Who can help?

No response

Information

[x] The official example scripts
[ ] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

Following the hugginface blog tutorial on optimum here: https://huggingface.co/blog/optimum-inference

running pip install optimum[onnxruntime-gpu]. There were multiple connection timeouts, but that maybe a different issue.

and then running the following code snippet from the tutorial

from pathlib import Path
from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForQuestionAnswering

model_id = "deepset/roberta-base-squad2"
onnx_path = Path("onnx")
task = "question-answering"

# load vanilla transformers and convert to onnx
model = ORTModelForQuestionAnswering.from_pretrained(model_id, from_transformers=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# save onnx checkpoint and tokenizer
model.save_pretrained(onnx_path)
tokenizer.save_pretrained(onnx_path)

# test the model with using transformers pipeline, with handle_impossible_answer for squad_v2
optimum_qa = pipeline(task, model=model, tokenizer=tokenizer, handle_impossible_answer=True)
prediction = optimum_qa(question="What's my name?", context="My name is Philipp and I live in Nuremberg.")

print(prediction)
# {'score': 0.9041663408279419, 'start': 11, 'end': 18, 'answer': 'Philipp'}

I see the follwing output in the jupyter notebook:

/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
2023-03-08 07:20:35.187102: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-08 07:20:35.826999: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-03-08 07:20:35.827063: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-03-08 07:20:35.827071: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
The argument `from_transformers` is deprecated, and will be removed in optimum 2.0.  Use `export` instead

Expected behavior

optimum seems to use the tensorflow installed on the system. This may not be an issue per se but I have both TF and torch installed for different use cases, I want optimum to use torch, because that's what I use to train my huggingface models for mainly classification tasks. I can't seem to find a way to force optimum to use torch instead. Any help will be appreciated. Thanks.

fxmarty commented 1 year ago

Thank you for the report @barathbheeman! I believe this is due to https://github.com/huggingface/transformers/blob/b338414e614a30af5f940269484ef15bf716d078/src/transformers/pipelines/__init__.py#L87-L88 in transformers. Tensorflow is actually not used, just imported, which arguably is very bad if it is not useful.

I remember internal discussions about this issue, @Narsil is probably most familiar with.

Narsil commented 1 year ago

Yes, but removing those imports are going to be tedious (I have started the work on some branch, felt like a never-ending fight to move imports around).

barathbheeman commented 1 year ago

Thank you for the report @barathbheeman! I believe this is due to https://github.com/huggingface/transformers/blob/b338414e614a30af5f940269484ef15bf716d078/src/transformers/pipelines/__init__.py#L87-L88 in transformers. Tensorflow is actually not used, just imported, which arguably is very bad if it is not useful.

I remember internal discussions about this issue, @Narsil is probably most familiar with.

Thanks for the quick response. so the import's actually happening in transformers and not optimum. It's nice that I can force it to use torch using the 'framework=pt' argument while creating pipelines. However, it would be nice if I can completely ignore one of the frameworks.

I suppose there's no quick fix available. If you have no more comments or recommendations, feel free to close the close the issue. Thanks!

huggingface / optimum