Closed wasertech closed 2 years ago
Good idea, thanks for the PR! We should add some instructions to the error message for this case: https://github.com/coqui-ai/STT/blob/178b4f146fdd260fbfb0cbb109343bbd6acdbc55/training/coqui_stt_training/export.py#L131-L141
May 22 saw changes to protobuff introduced to python wihich means we need to regenerate our protos. Workaround are:
See the error message:
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tflite-venv/lib/python3.7/site-packages/coqui_stt_training/export.py", line 13, in <module>
import tensorflow as tf
File "/tflite-venv/lib/python3.7/site-packages/tensorflow/__init__.py", line 102, in <module>
from tensorflow_core import *
File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/__init__.py", line 28, in <module>
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "<frozen importlib._bootstrap>", line 1019, in _handle_fromlist
File "/tflite-venv/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
module = self._load()
File "/tflite-venv/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
module = _importlib.import_module(self.__name__)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/python/__init__.py", line 52, in <module>
from tensorflow.core.framework.graph_pb2 import *
File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/core/framework/graph_pb2.py", line 16, in <module>
from tensorflow.core.framework import node_def_pb2 as tensorflow_dot_core_dot_framework_dot_node__def__pb2
File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/core/framework/node_def_pb2.py", line 16, in <module>
from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/core/framework/attr_value_pb2.py", line 16, in <module>
from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/core/framework/tensor_pb2.py", line 16, in <module>
from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/core/framework/resource_handle_pb2.py", line 16, in <module>
from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/core/framework/tensor_shape_pb2.py", line 42, in <module>
serialized_options=None, file=DESCRIPTOR),
File "/tflite-venv/lib/python3.7/site-packages/google/protobuf/descriptor.py", line 560, in __new__
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
It's still missing some linting errors, do you not have the pre-commit hook installed? Instructions here: https://github.com/coqui-ai/STT/blob/main/CONTRIBUTING.rst#code-linting
Ah, nevermind, looks like those are from the other PR, which means the answer to my question is yes :P
Ah, nevermind, looks like those are from the other PR, which means the answer to my question is yes :P
I'll make sure to hook it up!
EDIT: I've installed it but I won't retroactively lint the code, is there something I can do?
Hi, I'm stuck in something similar and will appreciate a lot some support.
My model trained well and performed tests but didn't export anything to the export_dir
. The official docker image didn't come with the tflite-venv
as @wasertech suggests to use so I created myself following this script inside the docker:
apt update
apt install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt install python3.7
apt install python3.7-venv
cd /
python3.7 -m venv tflite-venv
/tflite-venv/bin/pip3.7 install coqui_stt_training
/tflite-venv/bin/pip3.7 install protobuf==3.20.1
/tflite-venv/bin/pip3.7 install torch
export PATH=$PATH:/tflite-venv/bin
/tflite-venv/bin/python3.7 -m coqui_stt_training.export --checkpoint_dir /STT/stt-data/hau/checkpoints/hau-from-en-v0.1 --export_dir /STT/stt-data/hau/exported-models/hau-from-en-v0.1
Although, I'm still getting this:
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tflite-venv/lib/python3.7/site-packages/coqui_stt_training/export.py", line 317, in <module>
main()
File "/tflite-venv/lib/python3.7/site-packages/coqui_stt_training/export.py", line 305, in main
export()
File "/tflite-venv/lib/python3.7/site-packages/coqui_stt_training/export.py", line 136, in export
tflite_model = converter.convert()
File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/lite/python/lite.py", line 983, in convert
**converter_kwargs)
File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/lite/python/convert.py", line 449, in toco_convert_impl
enable_mlir_converter=enable_mlir_converter)
File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/lite/python/convert.py", line 200, in toco_convert_protos
raise ConverterError("See console for info.\n%s\n%s\n" % (stdout, stderr))
tensorflow.lite.python.convert.ConverterError: See console for info.
/bin/bash: toco_from_protos: command not found
I checked and toco_from_protos
can be reached from the docker's bash as I added /tflite-venv/bin
to the PATH
.
For your information, this is how I pulled the docker image in the first place:
docker run -it --entrypoint /bin/bash --name stt-train --gpus all --mount type=bind,source="$(pwd)"/stt-data,target=/STT/stt-data ghcr.io/coqui-ai/stt-train
Hi, I'm stuck in something similar and will appreciate a lot some support. My model trained well and performed tests but didn't export anything to the
export_dir
. The official docker image didn't come with thetflite-venv
as @wasertech suggests to use so I created myself following this script inside the docker:apt update apt install software-properties-common add-apt-repository ppa:deadsnakes/ppa apt install python3.7 apt install python3.7-venv cd / python3.7 -m venv tflite-venv /tflite-venv/bin/pip3.7 install coqui_stt_training /tflite-venv/bin/pip3.7 install protobuf==3.20.1 /tflite-venv/bin/pip3.7 install torch export PATH=$PATH:/tflite-venv/bin /tflite-venv/bin/python3.7 -m coqui_stt_training.export --checkpoint_dir /STT/stt-data/hau/checkpoints/hau-from-en-v0.1 --export_dir /STT/stt-data/hau/exported-models/hau-from-en-v0.1
Although, I'm still getting this:
Traceback (most recent call last): File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/tflite-venv/lib/python3.7/site-packages/coqui_stt_training/export.py", line 317, in <module> main() File "/tflite-venv/lib/python3.7/site-packages/coqui_stt_training/export.py", line 305, in main export() File "/tflite-venv/lib/python3.7/site-packages/coqui_stt_training/export.py", line 136, in export tflite_model = converter.convert() File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/lite/python/lite.py", line 983, in convert **converter_kwargs) File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/lite/python/convert.py", line 449, in toco_convert_impl enable_mlir_converter=enable_mlir_converter) File "/tflite-venv/lib/python3.7/site-packages/tensorflow_core/lite/python/convert.py", line 200, in toco_convert_protos raise ConverterError("See console for info.\n%s\n%s\n" % (stdout, stderr)) tensorflow.lite.python.convert.ConverterError: See console for info. /bin/bash: toco_from_protos: command not found
I checked and
toco_from_protos
can be reached from the docker's bash as I added/tflite-venv/bin
to thePATH
.For your information, this is how I pulled the docker image in the first place:
docker run -it --entrypoint /bin/bash --name stt-train --gpus all --mount type=bind,source="$(pwd)"/stt-data,target=/STT/stt-data ghcr.io/coqui-ai/stt-train
resolved for now by adding this line to /tflite-venv/lib/python3.7/site-packages/tensorflow_core/lite/python/convert.py
_toco_from_proto_bin = "/tflite-venv/bin/toco_from_protos"
@alpoktem the virtual environment was officially added to the image with version 1.4.0 so your trick isn’t necessary anymore. Just pull the latest stable image.
Also it’s bad practice to comment about your issues under someone else’s closed PR. Either open a new discussion or a ticket next time. We would have seen your message and would have tell you to pull the image from main to get the venv for tflite export.
Hi, Hola,
Estic de vacances fins al 12 de Gener. Si necessites alguna cosa urgent, si us plau, escriu a @.***
I am taking time off until 12th of January. If you need anything urgent please write to @.***
Estoy de vacaciones hasta el 13 de Enero. Si necesitas algo urgente, por favor, escribe a @.***
--
Alp Öktem Col·lectivaT, SCCL web:https://collectivat.cat/ @.*** @collectivat
This PR makes available a new
venv
usingpython3.7
to export your checkpoints to the TFLite format.Instead of using the image' system python (3.8), use python 3.7 from venv at
/tflite-venv/bin/python
.See my logs here. See #2220 for context.