Error with running external language model (Transformer XL)

[arashdehghani]

Hi every body, I want to use external language model for ASR. according to documentation found in openseq2deq page, it must be launched in 2 steps:

1. run inference to dump logits to pickle.
2. run beam search decoder with specific parameters.

I could successfully dump logits to pickle in step 1 But when i run scripts/decoder.py it gets me this Error: No module named 'ctc_decoders' I search a lot to find this script or module but no result!! any help please. Thank you Arash

[vsl9]

Can you please run ./scripts/install_decoders.sh?

[arashdehghani]

Thank you @vsl9 I'll run it.

[xw1324832579]

@arashdehghani Hello, have you run ./scripts/install_decoders.sh ?And have you fixed the problem above? After I run ./scripts/install_decoders.sh, I still got the same error no module named 'ctc_decoders' as you. If you solved it, please tell the result ,thank you

[xw1324832579]

Can you please run ./scripts/install_decoders.sh? @vsl9 Hi, I have run ./scripts/install_decoders.sh but still get the same error with @arashdehghani File "scripts/decode.py", line 11, in from ctc_decoders import Scorer ImportError: No module named 'ctc_decoders'

Could you please tell the solutions, is there any step I forgot to run , or should i change scripts/decode.py ?

[arashdehghani]

Hi @xw1324832579 this script is expired. @vsl9 can you please update it for other OpenSeq2Seq users?

[blisc]

Can you upload the log from install_decoders.sh? Are you sure it completed successfully?

[arashdehghani]

Hi @blisc this is the log i get from running install_decoders.sh:

`arman@arman-pc:~/Projects/ASR/test_english/OpenSeq2Seq-master$ ./scripts/install_decoders.sh

• git clone https://github.com/PaddlePaddle/DeepSpeech Cloning into 'DeepSpeech'... remote: Enumerating objects: 18, done. remote: Counting objects: 100% (18/18), done. remote: Compressing objects: 100% (14/14), done. remote: Total 2448 (delta 4), reused 9 (delta 1), pack-reused 2430 Receiving objects: 100% (2448/2448), 915.03 KiB | 596.00 KiB/s, done. Resolving deltas: 100% (1572/1572), done.
• cd DeepSpeech
• git checkout a76fc69 Note: checking out 'a76fc69'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:

git checkout -b

HEAD is now at a76fc69 Merge pull request #247 from ericxk/patch-1

• cd ..
• mv DeepSpeech/decoders/swig_wrapper.py DeepSpeech/decoders/swig/ctc_decoders.py
• mv DeepSpeech/decoders/swig ./decoders
• rm -rf DeepSpeech
• cd decoders
• sed -i s/.decode('utf-8')//g ctc_decoders.py
• sed -i s/.decode("utf-8")//g ctc_decoders.py
• sed -i s/name='swig_decoders'/name='ctc_decoders'/g setup.py
• sed -i s/-space_prefixes[i]->approx_ctc/space_prefixes[i]->score/g decoder_utils.cpp
• sed -i s/py_modules=['swig_decoders']/py_modules=['ctc_decoders', 'swig_decoders']/g setup.py
• chmod +x setup.sh
• ./setup.sh Cloning into 'kenlm'... remote: Enumerating objects: 13149, done. remote: Total 13149 (delta 0), reused 0 (delta 0), pack-reused 13149 Receiving objects: 100% (13149/13149), 5.24 MiB | 581.00 KiB/s, done. Resolving deltas: 100% (7533/7533), done.

Download and extract openfst ... --2019-07-01 07:33:09-- http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Resolving www.openfst.org (www.openfst.org)... 206.196.111.47 Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 07:35:21-- (try: 2) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 07:37:33-- (try: 3) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 07:39:47-- (try: 4) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 07:42:01-- (try: 5) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 07:44:17-- (try: 6) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 07:46:34-- (try: 7) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 07:48:50-- (try: 8) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 07:51:08-- (try: 9) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 07:53:26-- (try:10) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 07:55:46-- (try:11) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 07:58:06-- (try:12) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 08:00:31-- (try:13) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 08:03:01-- (try:14) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 08:05:25-- (try:15) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 08:07:49-- (try:16) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 08:10:12-- (try:17) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 08:13:05-- (try:18) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 08:15:29-- (try:19) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Retrying.

--2019-07-01 08:17:49-- (try:20) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz Connecting to www.openfst.org (www.openfst.org)|206.196.111.47|:80... failed: Connection timed out. Giving up.

tar (child): openfst-1.6.3.tar.gz: Cannot open: No such file or directory tar (child): Error is not recoverable: exiting now tar: Child returned status 2 tar: Error is not recoverable: exiting now

Cloning into 'ThreadPool'... remote: Enumerating objects: 82, done. remote: Total 82 (delta 0), reused 0 (delta 0), pack-reused 82 Unpacking objects: 100% (82/82), done.

Install decoders ... sh: 1: swig: not found running install error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the installation directory:

[Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/test-easy-install-23436.write-test'

The installation directory you specified (via --install-dir, --prefix, or the distutils default setting) was:

/usr/local/lib/python3.6/dist-packages/

Perhaps your account does not have write access to this directory? If the installation directory is a system-owned directory, you may need to sign in as the administrator or "root" account. If you do not have administrative access to this machine, you may wish to choose a different installation directory, preferably one that is listed in your PYTHONPATH environment variable.

For information on other options, you may wish to consult the documentation at:

https://setuptools.readthedocs.io/en/latest/easy_install.html

Please make the appropriate changes for your system and try again. `

it seems the download links are out of date. we try to use other existing download links instead but it doesn't work!

[shiv6146]

@arashdehghani I think you need to install swig before running install_decoders.sh

[vsl9]

@arashdehghani, there are two issues:

1. the script can't download openfst-1.6.3.tar.gz; please check your internet connection, firewall settings and make sure that you can open http://www.openfst.org/
2. as @shiv6146 mentioned, please install swig: sudo apt-get install swig or conda install swig

[arashdehghani]

@vsl9 @shiv6146 Thanks a lot for your helps.