Failure linking 'analyzer' with global variables [macos] [static]

[jgsogo]

Hi! I'm trying to compile the project from sources (release 4.1) in Macos (10.15.3) and I'm getting a link error in the analyzer and the threaded_analyzer (same linking error) when using static linking:

[ 93%] Linking CXX executable ../../../bin/threaded_analyzer
Undefined symbols for architecture x86_64:
  "_g_defines", referenced from:
      _fsm_parse_regex_string in libfoma.a(lex.yy.c.o)
      _yylex in libfoma.a(lex.yy.c.o)
  "_g_defines_f", referenced from:
      _fsm_parse_regex_string in libfoma.a(lex.yy.c.o)
      _yylex in libfoma.a(lex.yy.c.o)
ld: symbol(s) not found for architecture x86_64

Is it intended? Can I compile Freeling statically?

Thanks!

[lluisp]

Static linking is not supported. FreeLing is a large library that uses a lot of other libraries. A static binary would be huge, and I am not sure it makes much sense.

However, if you want to do that, the errors seems to indicate that you are missing some library. You should find out which library provides "_g_defines", get a static version of it, and maybe add it to cmake files.

[jgsogo]

I'll have a look at it. Static linking is preferred by a lot of people in the C++ community, it allows for some optimization (LTO) and I think that another reason is that it avoids DLL-hell and dependency management problems.

Static binary shouldn't be that big, these are the numbers I get from #97 where I've just added the fix for this issue: g_defines is declared in foma library but defined in the main() function of foma.c which have been removed in the copy inside this repository. Static linking detects this issue while dynamic would fail in runtime if used.

I can isolate needed changes, but I would prefer to have a CI working so I can know that my changes are not breaking anything else.

---

Dynamic linking

⇒  ls -la install/bin 
total 4960
drwxr-xr-x  10 jgsogo  staff      320 18 may 22:16 .
drwxr-xr-x   6 jgsogo  staff      192 18 may 21:04 ..
-rwxr-xr-x   1 jgsogo  staff     1028  4 abr 19:49 analyze
-rwxr-xr-x   1 jgsogo  staff   300880 18 may 22:16 analyzer
-rwxr-xr-x   1 jgsogo  staff    49344 18 may 22:16 analyzer_client
-rwxr-xr-x   1 jgsogo  staff    85476 18 may 22:16 convert_model
-rwxr-xr-x   1 jgsogo  staff    37284 18 may 22:16 crfsuite
-rwxr-xr-x   1 jgsogo  staff      999  4 abr 19:49 fl_initialize
-rwxr-xr-x   1 jgsogo  staff   335368 18 may 22:16 threaded_analyzer
-rwxr-xr-x   1 jgsogo  staff  1708792 18 may 22:16 treeler

⇒  ls -la install/lib 
total 26448
drwxr-xr-x  7 jgsogo  staff      224 18 may 22:16 .
drwxr-xr-x  6 jgsogo  staff      192 18 may 21:04 ..
-rwxr-xr-x  1 jgsogo  staff   165972 18 may 22:16 libcrfsuite.dylib
-rwxr-xr-x  1 jgsogo  staff  8853324 18 may 22:16 libdynet.dylib
-rwxr-xr-x  1 jgsogo  staff   304724 18 may 22:16 libfoma.dylib
-rwxr-xr-x  1 jgsogo  staff  3898584 18 may 22:16 libfreeling.dylib
-rwxr-xr-x  1 jgsogo  staff   309740 18 may 22:16 libtreeler.dylib

plus boost, icu, zlib, eigen and bzip2

Static linking

⇒  ls -la ./install/bin 
total 43440
drwxr-xr-x  10 jgsogo  staff       320 18 may 22:20 .
drwxr-xr-x   6 jgsogo  staff       192 18 may 21:36 ..
-rwxr-xr-x   1 jgsogo  staff      1028  4 abr 19:49 analyze
-rwxr-xr-x   1 jgsogo  staff  13898276 18 may 22:20 analyzer
-rwxr-xr-x   1 jgsogo  staff     49344 18 may 22:20 analyzer_client
-rwxr-xr-x   1 jgsogo  staff   1749752 18 may 22:20 convert_model
-rwxr-xr-x   1 jgsogo  staff     37284 18 may 22:20 crfsuite
-rwxr-xr-x   1 jgsogo  staff       999  4 abr 19:49 fl_initialize
-rwxr-xr-x   1 jgsogo  staff   4632912 18 may 22:20 threaded_analyzer
-rwxr-xr-x   1 jgsogo  staff   1844120 18 may 22:20 treeler

[lluisp]

I am really reluctant to make such big changes to the building procedure.

FreeLing is a low-resource project, and we don't have the manpower to change and maintain all this. I don't like either the idea of depending on yet another external platform like travis or the like. Moreover, the update rate is also pretty slow, so continuous integration is not a must.

In any case, I am willing to make the changes needed to allow static linking. It seems that foma.h will define those symbols when compiled with -DMAIN_MODULE, so that may be the easiest way. FreeLing uses limited functions from foma, and that is not causing any problems with dynamic linking, meaning they are not used.

  best

            Lluis

On 18/5/20 22:25, Javier G. Sogo wrote:

I'll have a look at it. Static linking is preferred by a lot of people in the C++ community, it allows for some optimization (LTO) and I think that another reason is that it avoids DLL-hell and dependency management problems.

Static binary shouldn't be that big, these are the numbers I get from

97 https://github.com/TALP-UPC/FreeLing/pull/97 where I've just

added the fix for this issue: |g_defines| is declared in |foma| library but defined in the |main()| function of |foma.c| which have been removed in the copy inside this repository. Static linking detects this issue while dynamic would fail in runtime if used.

I can isolate needed changes, but I would prefer to have a CI working so I can know that my changes are not breaking anything else.

---

Dynamic linking

|⇒ ls -la install/bin total 4960 drwxr-xr-x 10 jgsogo staff 320 18 may 22:16 . drwxr-xr-x 6 jgsogo staff 192 18 may 21:04 .. -rwxr-xr-x 1 jgsogo staff 1028 4 abr 19:49 analyze -rwxr-xr-x 1 jgsogo staff 300880 18 may 22:16 analyzer -rwxr-xr-x 1 jgsogo staff 49344 18 may 22:16 analyzer_client -rwxr-xr-x 1 jgsogo staff 85476 18 may 22:16 convert_model -rwxr-xr-x 1 jgsogo staff 37284 18 may 22:16 crfsuite -rwxr-xr-x 1 jgsogo staff 999 4 abr 19:49 fl_initialize -rwxr-xr-x 1 jgsogo staff 335368 18 may 22:16 threaded_analyzer -rwxr-xr-x 1 jgsogo staff 1708792 18 may 22:16 treeler | |⇒ ls -la install/lib total 26448 drwxr-xr-x 7 jgsogo staff 224 18 may 22:16 . drwxr-xr-x 6 jgsogo staff 192 18 may 21:04 .. -rwxr-xr-x 1 jgsogo staff 165972 18 may 22:16 libcrfsuite.dylib -rwxr-xr-x 1 jgsogo staff 8853324 18 may 22:16 libdynet.dylib -rwxr-xr-x 1 jgsogo staff 304724 18 may 22:16 libfoma.dylib -rwxr-xr-x 1 jgsogo staff 3898584 18 may 22:16 libfreeling.dylib -rwxr-xr-x 1 jgsogo staff 309740 18 may 22:16 libtreeler.dylib |

plus |boost|, |icu|, |zlib|, |eigen| and |bzip2|

Static linking

|⇒ ls -la ./install/bin total 43440 drwxr-xr-x 10 jgsogo staff 320 18 may 22:20 . drwxr-xr-x 6 jgsogo staff 192 18 may 21:36 .. -rwxr-xr-x 1 jgsogo staff 1028 4 abr 19:49 analyze -rwxr-xr-x 1 jgsogo staff 13898276 18 may 22:20 analyzer -rwxr-xr-x 1 jgsogo staff 49344 18 may 22:20 analyzer_client -rwxr-xr-x 1 jgsogo staff 1749752 18 may 22:20 convert_model -rwxr-xr-x 1 jgsogo staff 37284 18 may 22:20 crfsuite -rwxr-xr-x 1 jgsogo staff 999 4 abr 19:49 fl_initialize -rwxr-xr-x 1 jgsogo staff 4632912 18 may 22:20 threaded_analyzer -rwxr-xr-x 1 jgsogo staff 1844120 18 may 22:20 treeler |

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/TALP-UPC/FreeLing/issues/96#issuecomment-630416546, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANH565XLFGJAE4PDQ6F7CDRSGKTBANCNFSM4L6YK6CQ.

[jgsogo]

Hi! I totally understand your concerns and I would think exactly the same if I hadn't used Travis/Appveyor/Github-actions/CircleCI/Azure/... before but in my experience it is the other way around: having those systems enabled helps the maintainers to know that everything keeps working without having to check it with every change. Files for those systems also contain the instructions for developers to compile the library (this is what I have a look to when I need to build a library).

I will submit small changes to improve the build system step by step, that's not a problem (and it is the reason why the big PR is a draft, not to be merged but to talk about it). So, let's go step by step. I'll keep Travis running in my fork (I'll add Appveyor for Windows as soon as I managed to compile it) so we can help some colleagues out there.

Thanks for all your efforts!

[lluisp]

I understand, but you are a developer, so you are able to read those instructions, or use cmake, install a compiler...

The people you are trying to help probably struggle just with installing a compiler.  They need out-of-the box binary packages.

I'd prefer to improve the building system in the line of keeping cmake and easing the build of binary packages. Some things that would be great:   1) not depending on MSVC for windows.   2) including working java and python APIs in the binary packages

On 19/5/20 11:16, Javier G. Sogo wrote:

Hi! I totally understand your concerns and I would think exactly the same if I hadn't used Travis/Appveyor/Github-actions/CircleCI/Azure/... before but in my experience it is the other way around: having those systems enabled helps the maintainers to know that everything keeps working without having to check it with every change. Files for those systems also contain the instructions for developers to compile the library (this is what I have a look to when I need to build a library).

I will submit small changes to improve the build system step by step, that's not a problem (and it is the reason why the big PR is a draft, not to be merged but to talk about it). So, let's go step by step. I'll keep Travis running in my fork (I'll add Appveyor for Windows as soon as I managed to compile it) so we can help some colleagues out there https://twitter.com/JMFradeRue/status/1255747683974885376.

Thanks for all your efforts!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/TALP-UPC/FreeLing/issues/96#issuecomment-630696313, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANH564BSZJNUZ2TJCLYPN3RSJE7NANCNFSM4L6YK6CQ.

[jgsogo]

Yes, those are instructions for developers. Users should have binary packages available out-of-the-box. Any CI should compile the library (instruction for developers) and upload generated binaries (for different distros, windows versions,.... whatever is needed) somewhere so users can download them (FTP, Unix ppa, snap, flatpack, bintray, webpage,...).

What are your ideas for "not depending on MSVC for windows"? What compiler do you want to use?