Compiling mitlm 0.4 on Ubuntu 9.10

[GoogleCodeExporter]

Just in case anyone will come across the same problem:
The linker was complaining about -lg2c and an undefined reference to
ApplySort(...). Here are few steps leading to successful compilation:

1. install gfortran:
sudo apt-get install gfortran

2. in the Makefile of mitlm, it is needed to set:
LDFLAGS  = -L. -lgfortran -lmitlm 
FC       = gfortran

...the LDFLAGS line is already there, but it is needed to replace -lg2c
with -lgfortran

3. open the file NGramModel.cpp in a text editor and comment out the
definition of ApplySort template method. Then copy it's code and paste it
into NGramModel.h right after the declaration of ApplySort. The code of the
template method simply has to be in the header file.

4. rm src/NgramLM.o

5. make

Thank you Bo-June, for the toolkit.
Best regards,
Michal

Original issue reported on code.google.com by michal.f...@gmail.com on 14 Jan 2010 at 11:29

[GoogleCodeExporter]

Also, when you get errors about missing string functions, add "#include 
<cstring>"
and for missing limits, add "#include <limits>"

Original comment by michal.f...@gmail.com on 18 Jan 2010 at 11:08

[GoogleCodeExporter]

Is it available as a patch file ?

Original comment by jagana...@gmail.com on 18 Jan 2010 at 11:31

[GoogleCodeExporter]

The patch file is attached to this post. Just put it into your mitlm.0.4 
directory
and apply the patch:

patch -p1 -i mitlm.0.4.ubuntu9.10.patch

make clean
make

Original comment by michal.f...@gmail.com on 18 Jan 2010 at 8:57

Attachments:

• mitlm.0.4.ubuntu9.10.patch

[GoogleCodeExporter]

All these issues should be fixed in revision 45.
In revision 45 autotools are used, so you need to install autoconf, automake 
and libtool to build mitlm.
You can build mitlm using the following commands:
./autogen.sh --prefix=$(pwd)/usr
make
make install

On ubuntu you should also be able to create a debian package of mitlm with the 
following command line (you need to run autogen.sh before using it):
dpkg-buildpackage -rfakeroot

Can you please provide feedback on these commands?

Original comment by giuliop...@gmail.com on 17 Nov 2010 at 11:05

[GoogleCodeExporter]

Hi, I downloaded the latest svn revision 45. There is still the problem with 
f77. Compilation stops with this message:

libtool: link: g++ -g -O2 -o .libs/evaluate-ngram src/evaluate-ngram.o  
./.libs/libmitlm.so
./.libs/libmitlm.so: undefined reference to `etime_'
./.libs/libmitlm.so: undefined reference to `s_wsfe'
./.libs/libmitlm.so: undefined reference to `s_cmp'
./.libs/libmitlm.so: undefined reference to `e_wsfe'
./.libs/libmitlm.so: undefined reference to `do_lio'
./.libs/libmitlm.so: undefined reference to `s_wsle'
./.libs/libmitlm.so: undefined reference to `s_copy'
./.libs/libmitlm.so: undefined reference to `do_fio'
./.libs/libmitlm.so: undefined reference to `f_open'
./.libs/libmitlm.so: undefined reference to `e_wsle'
collect2: ld returned 1 exit status
make[1]: *** [evaluate-ngram] Error 1
make[1]: Leaving directory `/home/miso/install/mitlm-svn'
make: *** [all-recursive] Error 1

...which complains about a missing library 'f2c'. When I run this:
g++ -g -O2 -o .libs/evaluate-ngram src/evaluate-ngram.o  ./.libs/libmitlm.so 
-lf2c

another error occurs:
/usr/lib/gcc/i486-linux-gnu/4.4.3/../../../../lib/libf2c.so: undefined 
reference to `MAIN__'
collect2: ld returned 1 exit status

...so I found a workaround here: 
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=337460
...then this command works fine:
g++ -g -O2 -o .libs/evaluate-ngram src/evaluate-ngram.o  ./.libs/libmitlm.so 
-lf2c -u MAIN__

For the autotools, I put these 3 lines into Makefile.am:
evaluate_ngram_LDFLAGS = -lf2c -u MAIN__
estimate_ngram_LDFLAGS = -lf2c -u MAIN__
interpolate_ngram_LDFLAGS = -lf2c -u MAIN__

then ./autogen.sh and make work fine. I am not sure, whether this is the 
correct solution for all systems, but it works on Ubuntu 10.04.

Michal

Original comment by michal.f...@gmail.com on 19 Nov 2010 at 1:26

[GoogleCodeExporter]

Hi Michal,
  I'm trying to replicate the problem on my pc, without success. (I've tried gcc 4.3.5 and 4.4.5, using gfortran)

Obviously the line "libtool: link: g++ -g -O2 -o .libs/evaluate-ngram 
src/evaluate-ngram.o  ./.libs/libmitlm.so" is wrong, because there should be 
some additional token there.

Which fortran compiler are you using? What's the output of the configure script 
in the failing case?

Giulio.

Original comment by giuliop...@gmail.com on 26 Nov 2010 at 1:44

[GoogleCodeExporter]

Hi Giulio,

sorry for the incompleteness of the previous message. The output of 
./autogen.sh is attached as configure.out, then there is config.log with config 
variables and make.out with the full output of make.

I use Ubuntu 10.04 with gcc (Ubuntu 4.4.3-4ubuntu5) 4.4.3

As for the f77 compiler, it points to fort77, which is in version 1.15-8

Version of f2c is 20090411-1 and libf2c2 is 20090411-2

You can see in the config.log, that FLIBS variable is empty. Ideally there 
should be the '-lf2c'. The MAIN__ function can be integrated into the fortran 
code directly.

Michal

Original comment by michal.f...@gmail.com on 26 Nov 2010 at 7:04

Attachments:

• configure.out
• config.log
• make.out

[GoogleCodeExporter]

Thank you for all the details.
I was able to replicate the problem and investigate its causes.
There were two problems:
1) I did not include a test to detect the need of a MAIN__ function.
2) The current autoconf macro to detect F77 compilers is unable to properly 
detect fort77. I already contacted autoconf developers about this issue. I hope 
they will fix it in future autoconf versions.

With the latest subversion revision (47) you should be able to build mitlm 
using the following commands:
1) If f77 is fort77 (the f2c wrapper) and you want to use it
FLIBS="-lf2c -lm" ./autogen.sh --prefix=$(pwd)/usr
make
make install
2) If you want to specify another compiler that is not the default one (e.g., 
on Debian when you have both fort77 and gfortran, and you want to use gfortran) 
you can also use these commands:
F77=gfortran ./autogen.sh --prefix=$(pwd)/usr
make
make install

The issue cannot be solved in mitlm.
It can be fixed by manually patching the configure script before releasing next 
mitlm version.

Original comment by giuliop...@gmail.com on 28 Nov 2010 at 8:22

[GoogleCodeExporter]

You're right, Giulio. Both methods you described, work with the revision 47 on 
my system. Thanks for your work.
Michal

Original comment by michal.f...@gmail.com on 29 Nov 2010 at 11:25

[GoogleCodeExporter]

Thank you for your confirmation.

I think the issue can be closed now, because it cannot be solved in mitlm.
We can add a note about this issue in the README file (suggesting the above 
command lines and pointing to the INSTALL file for further information about 
configure script usage).

Original comment by giuliop...@gmail.com on 30 Nov 2010 at 1:39

[GoogleCodeExporter]

Yes, you can close this issue as SOLVED. That should be enough to put a note 
about this issue in the README file.
Michal

Original comment by michal.f...@gmail.com on 30 Nov 2010 at 9:01

[GoogleCodeExporter]

Hello Michal,

After applying the patch, I still have Fortran link error:

g++ src/estimate-ngram.o -o estimate-ngram -L. -lgfortran -lmitlm  -O3 
-funroll-loops
./libmitlm.a(lbfgsb.o): In function `prn3lb_':
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2788: undefined 
reference to `_gfortran_compare_string'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2791: undefined 
reference to `_gfortran_st_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2791: undefined 
reference to `_gfortran_st_write_done'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2792: undefined 
reference to `_gfortran_st_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2792: undefined 
reference to `_gfortran_st_write_done'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2793: undefined 
reference to `_gfortran_st_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2793: undefined 
reference to `_gfortran_transfer_integer_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2793: undefined 
reference to `_gfortran_transfer_integer_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2793: undefined 
reference to `_gfortran_transfer_integer_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2793: undefined 
reference to `_gfortran_transfer_integer_write'

It looks like the gfortran library, located in

  /usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.so

didn't get linked in. The Makefile was patched successfully:

INC      = -Isrc
CXXFLAGS = -g -Wall -fPIC -fmessage-length=0 $(INC)
#LDFLAGS  = -L. -lg2c -lmitlm
LDFLAGS  = -L. -lgfortran -lmitlm
FFLAGS   = -g -fPIC -fmessage-length=0
FC       = gfortran

Somehow g++ linker wasn't able to located the lib ?

Chan

Original comment by chan...@gmail.com on 9 Mar 2012 at 11:54

[GoogleCodeExporter]

Hi Chan,

I assume you compile it with gcc version 4.6, right? You could try modifying 
LDFLAGS in the Makefile to:

LDFLAGS  = -L. -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -lgfortran -lmitlm

or try directly this:

LDFLAGS  = -L. -l/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.so -lmitlm

if that does not work, this command should tell you whether the correct symbols 
are included in the libgfortran.so:

nm -D /usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.so | grep 
_gfortran_compare_string

...it should give you something like:

000d2b90 T _gfortran_compare_string
000d3320 T _gfortran_compare_string_char4

Let me know if it still doesn't work,
Michal

Original comment by michal.f...@gmail.com on 10 Mar 2012 at 7:47

[GoogleCodeExporter]

Hi Chan,
did you also tried the solutions proposed at Comment 8? In your case the second 
solution should work.

Anyway on my system (Debian wheezy amd64) I need "-lgfortran -lm -lquadmath" to 
link against gfortran.

Bests,
Giulio.

Original comment by giuliop...@gmail.com on 10 Mar 2012 at 1:46

[GoogleCodeExporter]

Hi Michal & Giulio,

I've tried all of your linking options as suggested, but no luck.
It's quite certain that the linker has its access to the lib of
libgfortran.so, and the underscroed symbol ids are matching each
other (one from the linker complaint and the other from the library).
But then the linker still cannot fins the symbols:

chan@ubuntu:~/Speech/MIT_LM/mitlm.0.4$ nm -D 
/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.so | grep _gfortran_compare_string
00000000000e8000 T _gfortran_compare_string
00000000000e87e0 T _gfortran_compare_string_char4

chan@ubuntu:~/Speech/MIT_LM/mitlm.0.4$ g++ src/estimate-ngram.o -o 
estimate-ngram -v -L. -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -lgfortran -lmitlm 
-lquadmath |& more
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.6.1/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.1-9ubu
ntu3' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages
=c,c++,fortran,objc,obj-c++,go --prefix=/usr --program-suffix=-4.6 --enable-shar
ed --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-i
ncluded-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4
.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable
-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-objc-gc --
disable-werror --with-arch-32=i686 --with-tune=generic --enable-checking=release
 --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) 
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.6.1/:/usr/lib/gcc/x86_64-linux-gnu
/4.6.1/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.6.1/:/usr
/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.6.1/:/usr/lib/gcc/x86_64-linux-gnu/
4.6.1/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../..
/lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/..
/lib/:/usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-o' 'estimate-ngram' '-v' '-L.' '-L/usr/lib/gcc/x86_64-linu
x-gnu/4.6' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/4.6.1/collect2 --build-id --no-add-needed --as-ne
eded --eh-frame-hdr -m elf_x86_64 --hash-style=gnu -dynamic-linker /lib64/ld-lin
ux-x86-64.so.2 -z relro -o estimate-ngram /usr/lib/gcc/x86_64-linux-gnu/4.6.1/..
/../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../../x86_
64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.6.1/crtbegin.o -L. -L/usr/li
b/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/gcc/x86_64-linux-gnu/4.6.1 -L/usr/lib/gcc/
x86_64-linux-gnu/4.6.1/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu
/4.6.1/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-l
inux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6.1/../../.. src/es
timate-ngram.o -lgfortran -lmitlm -lquadmath -lstdc++ -lm -lgcc_s -lgcc -lc -lgc
c_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/4.6.1/crtend.o /usr/lib/gcc/x86_64-linux
-gnu/4.6.1/../../../x86_64-linux-gnu/crtn.o
./libmitlm.a(lbfgsb.o): In function `prn3lb_':
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2788: undefined referen
ce to `_gfortran_compare_string'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2791: undefined referen
ce to `_gfortran_st_write'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2791: undefined referen
ce to `_gfortran_st_write_done'
/home/chan/Speech/MIT_LM/mitlm.0.4/src/optimize/lbfgsb.f:2792: undefined referen
ce to `_gfortran_st_write'

These 'write' fortran function calls (from lbfgsb.f) were converted to different
write library functions (depending on the args types) with underscored names,
which are matched each other.

Trying to figure out why the linker cannot resolve these symbols.

Thanks for your help,

Chan

Original comment by chan...@gmail.com on 10 Mar 2012 at 8:51

[GoogleCodeExporter]

This is quite strange. My system is very similar to your (I have gcc-4.6.2, 
compiled with very similar flags, for the same architecture), but I am not able 
to replicate the problem on my system.

Just to be sure:
1) Can you verify that you have also the libgfortran.a file? (find /usr/lib/ 
-name "*gfortran*") On my system it is in the gfortran-4.6 package.
2) Have you tried using the mitlm version on the SVN repository, using the 
autoconf build system?

Original comment by giuliop...@gmail.com on 11 Mar 2012 at 6:07

[GoogleCodeExporter]

Hi Giulio,

Yes, taking MITLM from the svn repository did the trick.
SVN by:

  svn checkout http://mitlm.googlecode.com/svn/trunk/ mitlm

Then I did 'autogen.sh' to auto configure:

  ./autogen.sh

Then a make built all of the LM tools:

  make

Thanks for your help !

Chan

Original comment by chan...@gmail.com on 12 Mar 2012 at 4:03

[GoogleCodeExporter]

Anyway I still do not know why it did not work with the Michal's patch...

Apart from using the SVN source, did you change anything else on your system?

(e.g., what was the answer to my first question at comment 16?)

Original comment by giuliop...@gmail.com on 12 Mar 2012 at 4:09

[GoogleCodeExporter]

Nothing changed other than downloading new source from the svn repository.
All of the gfortran libs were existing:

chan@ubuntu:~/Speech/MIT_LM/mitlm.0.4$ find /usr/lib -name "*gfortran*" -print
/usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0
/usr/lib/x86_64-linux-gnu/libgfortran.so.3
/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.spec
/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.a
/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.so
/usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortranbegin.a

Original comment by chan...@gmail.com on 13 Mar 2012 at 1:38

[GoogleCodeExporter]

Ok, thank you for the feedback. If you will ever know why it failed with the 
simple Makefile method, please let me know.

Bests,
Giulio.

Original comment by giuliop...@gmail.com on 13 Mar 2012 at 9:59

[GoogleCodeExporter]

I followed the steps outlined in comment 17, but  when I run './autogen.sh'
I get the following error

http://pastebin.com/94tds5a2

The last line is the error line:
config.status: error: cannot find input file: `Makefile.in'

I am running ubuntu 11.10

Original comment by mingxiao...@gmail.com on 1 Jun 2012 at 12:32

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

 I had the same error message. Installed libtool and it worked.

Original comment by ramanaga...@gmail.com on 1 Jun 2012 at 3:54

[GoogleCodeExporter]

Yes, libtool automake autoconf are all required to create the files for 
building.
I would also add that autoconf >= 2.69 is also required for proper building 
(fortran detection has been fixed there).

Original comment by giuliop...@gmail.com on 1 Jun 2012 at 10:58

[GoogleCodeExporter]

Original comment by giuliop...@gmail.com on 29 Jan 2013 at 7:13

• Changed state: WontFix