Closed funderburkjim closed 6 years ago
Hi there! Building pynini_cdrewrite.cc is one of the longest steps of the process, because it translates to a huge amount of machine code. However, I am able to do it with relatively weak computers (for instance, a $100 ARM laptop with just 4 GB of RAM) when given enough time---it takes maybe an hour. If you can wait a bit longer and it's not running out of RAM, I would just allow that. Since you're compiling in a virtual environment you may want to make sure your VM has enough computing resources for the compilation. Or if that's not feasible, you can transpile---compile on another machine with more juice.
Another possibility---and this is a bit of a hack, but will work for many people, is the following. I'm assuming you're using the most recent version of Pynini, which is 2.0.0:
open src/pynini_cdrewrite.cc with a text editor and comment out (or delete) lines 44 and 45, which read:
REGISTER_FST_OPERATION(PyniniCDRewrite, LogArc, PyniniCDRewriteArgs); REGISTER_FST_OPERATION(PyniniCDRewrite, Log64Arc, PyniniCDRewriteArgs);
What these lines say is that the cdrewrite
function should work for
log-weight arcs. I've actually never had any need to do this---you can just
use tropical arcs and then convert to log arcs pynini.map
, if desired,
and it should make compilation go 3x as fast, at least for this file.
FYI this warning here not an error. It's just a (very minor) bug in how Python is configured on certain platforms; as I understand it it thinks that when compiling C++ code it should use the strict prototype warnings, which doesn't make sense for C++ (I don't even know what a strict prototype is):
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is
valid for C/ObjC but not for C++ [enabled by default]
On Tue, Jul 31, 2018 at 11:51 PM funderburkjim notifications@github.com wrote:
HI. I'm interested in trying pynini, but I can't get it installed.
I'm running ubuntu trusty32 version via a vagrant/virtualbox configuration under windows 10. All the prerequisites, AFAIK, have been installed.
When 'sudo python setup.py install' is run, the compilations start. However, the compilation of cdrewrite.cc never finishes. The last thing I see in terminal is:
i686-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes - fPIC -I/usr/include/python2.7 -c src/pynini_cdrewrite.cc -o build/temp.linux-i686- 2.7/src/pynini_cdrewrite.o -std=c++11 -Wno-unused-function -Wno-unused-local-typedef -funsigned- char cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
I'm not knowledgeable about these kind of complicated installations, so have no idea what might need to be done to finish the installation. What do you suggest?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kylebgorman/Pynini/issues/5, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJuOe1a40thfEB8ixHGDdo9mGTsu8Y8ks5uMSXBgaJpZM4VpzRG .
Thanks for the helpful comments. Based on them, I increased the VM memory to 2GB -- this proved to do the trick; setup.py install worked, and took about 10 minutes.
There was another obstacle that occurred with the setup.py test step.
ImportError: libre2.so.0: cannot open shared object file: No such file or directory
https://stackoverflow.com/questions/8323794/re2-library-loading provided a suggestion;
namely I returned to the re2 folder and issued command sudo ldconfig
.
Magically, now python setup.py test
ran to completion with Ran 158 tests in 0.610s
.
Also I can now import pynini
and begin working with examples in your various tutorial materials on
pynini and openfst.
Incidentally, my motivation is to understand how to apply something from your https://www.safaribooksonline.com/oriole/string-processing-with-pythons-pynini-edit-transducers
tutorial.
Imagine that we have a query string and wish to find the string or strings in a
fixed lexicon it is closest to, in terms of edit distance.
I'd like to be able to apply that in context of a lexicon of about 250k Sanskrit words drawn from the headwords of various Sanskrit dictionaries.
Sounds good, it should be well suited for that. I hope to write more about that kind of edit distance matching in the future ;)
I would guess that you would need at least 2 GB, if not 4 GB, to compile a large project like this.
HI. I'm interested in trying pynini, but I can't get it installed.
I'm running ubuntu trusty32 version via a vagrant/virtualbox configuration under windows 10. All the prerequisites, AFAIK, have been installed.
When 'sudo python setup.py install' is run, the compilations start. However, the compilation of cdrewrite.cc never finishes. The last thing I see in terminal is:
I'm not knowledgeable about these kind of complicated installations, so have no idea what might need to be done to finish the installation. What do you suggest?