search

OpenNMT / OpenNMT-tf

Neural machine translation and sequence learning using TensorFlow
https://opennmt.net/
MIT License
1.45k stars 391 forks source link

Export to TensorFlow Lite #490

Closed lockder closed 3 years ago

lockder commented 5 years ago

Hi there I'm trying to export a ner model to tf-lite i found the current export is using the folder assets and will say ValueError: SavedModels with assets/ directory are not supported. I know the v2 branch is not using the assets folder anymore, but this export is for production, so I would like to use the stable version of opennmt-tf

is there any workaround to make it work?

Thanks!

guillaumekln commented 5 years ago

TensorFlow Lite has incomplete support for sequence models. In particular, it does not support vocabulary lookup (used to convert tokens to ids) and control flow ops (used during dynamic decoding in sequence to sequence models).

See the Known Limitations in their guide.

I'm not aware of any workaround.

lockder commented 5 years ago

I knew about sequence to sequence but since its sequence_tagger I tought it was a problem of compatibility. So until they support at least the vocabulary lookup it will not work

lockder commented 5 years ago

thanks you very much!

lockder commented 5 years ago

https://www.tensorflow.org/lite/guide/roadmap

lockder commented 5 years ago

I just wanted to let you know, this year should be posible to do it :) So its great news

guillaumekln commented 5 years ago

Thanks. I will keep an eye on how this evolves. Let's keep this issue open to track future support.

lockder commented 5 years ago

cool thanks you @guillaumekln

huangjian2015 commented 4 years ago

Hi there I'm trying to export a ner model to tf-lite i found the current export is using the folder assets and will say ValueError: SavedModels with assets/ directory are not supported. I know the v2 branch is not using the assets folder anymore, but this export is for production, so I would like to use the stable version of opennmt-tf

is there any workaround to make it work?

Thanks!

Hi there I'm trying to export a ner model to tf-lite i found the current export is using the folder assets and will say ValueError: SavedModels with assets/ directory are not supported. I know the v2 branch is not using the assets folder anymore, but this export is for production, so I would like to use the stable version of opennmt-tf

is there any workaround to make it work?

Thanks!

Hello, now I want to use tflite to quantize the weights trained from opennmt, and I met similar problems, so have you found some solutions? If you could give me some advice, I would appreciate you so much!

lockder commented 4 years ago

we finally decided not to export to tensorflow lite waiting while tensorflow allowed to export everything

El lun., 13 jul. 2020 a las 10:56, huangjian2015 (notifications@github.com) escribió:

Hi there I'm trying to export a ner model to tf-lite i found the current export is using the folder assets and will say ValueError: SavedModels with assets/ directory are not supported. I know the v2 branch is not using the assets folder anymore, but this export is for production, so I would like to use the stable version of opennmt-tf

is there any workaround to make it work?

Thanks!

Hi there I'm trying to export a ner model to tf-lite i found the current export is using the folder assets and will say ValueError: SavedModels with assets/ directory are not supported. I know the v2 branch is not using the assets folder anymore, but this export is for production, so I would like to use the stable version of opennmt-tf

is there any workaround to make it work?

Thanks!

Hello, now I want to use tflite to quantize the weights trained from opennmt, and I met similar problems, so have you found some solutions? If you could give me some advice, I would appreciate you so much!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/OpenNMT/OpenNMT-tf/issues/490#issuecomment-657421175, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEIX7Z5BOYF22GEQ7RSJ53R3LD4TANCNFSM4ITEJAKQ .

guillaumekln commented 3 years ago

TensorFlow Lite has incomplete support for sequence models. In particular, it does not support vocabulary lookup (used to convert tokens to ids) and control flow ops (used during dynamic decoding in sequence to sequence models).

This comment appears to be still valid for TensorFlow 2.4. They are some improvements but conversion still fails on these parts of the model (vocabulary lookup and dynamic decoding).

Right now I think the best path to go mobile is to convert models to CTranslate2 which supports ARM64 compilation.

guillaumekln commented 3 years ago

The master branch now includes an exporter to TensorFlow Lite thanks to @gcervantes8. The new export format are tflite and tflite_float16. Example:

onmt-main [...] export --export_dir ~/my-tflite-model --export_format tflite

The exporter currently supports RNN-based sequence to sequence models. For now, the conversion of Transformer models are failing and other model types are not implemented.

See https://github.com/OpenNMT/OpenNMT-tf/pull/789 for more information about the required configuration and how to run the model on Android.

I'm closing this general issue since the main workflow has been implemented. If you are using a model which currently can't be exported to TensorFlow Lite, please open another issue for this model specifically.