emcc fails to compile fc-solve properly with -O2 (works fine with -g)

[shlomif]

Hi and happy new year and thanks for all your help on my previous bug report,

sorry to be the harbinger of bad news (= what seems to be a bug in this case) again, but I ran into a place where emcc throws a hard-to-understand Python exception when I use -O2 instead of -g to the opt flags. To reproduce do:

git clone https://shlomif@bitbucket.org/shlomif/fc-solve.git
cd fc-solve/fc-solve/source/
bash ~/emcc-build-O2.bash

Where the emcc-build-O2.bash is:

#!/bin/bash

if ! test -d build ; then
    mkdir build
fi

cd build
cmake -DFCS_WITH_TEST_SUITE= ..
make boards
make -f ../Makefile.to-javascript.mak SRC_DIR=..

(with emcc/clang/etc. in the path).

This gives me:

hlomif/apps/fcs-for-pysol/share/freecell-solver/presets/maliciously-obscure.sh --embed-file /home/shlomif/apps/fcs-for-pysol/share/freecell-solver/presets/micro-finance-improved.sh --embed-file /home/shlomif/apps/fcs-for-pysol/share/freecell-solver/presets/enlightened-ostrich.sh --embed-file /home/shlomif/apps/fcs-for-pysol/share/freecell-solver/presets/micro-finance.sh
WARNING: Linking two modules of different target triples: /home/shlomif/.emscripten_cache/dlmalloc.bc: 'i386-pc-linux-gnu' and 'i386-unknown-linux-gnu'
Stack: Error
    at assertTrue (eval at globalEval (/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/src/compiler.js:105:8), <anonymous>:53:26)
    at parseArbitraryInt (eval at globalEval (/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/src/compiler.js:105:8), <anonymous>:635:3)
    at getLegalLiterals (eval at globalEval (/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/src/compiler.js:105:8), <anonymous>:135:24)
    at eval (eval at globalEval (/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/src/compiler.js:105:8), <anonymous>:331:32)
    at Array.forEach (native)
    at eval (eval at globalEval (/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/src/compiler.js:105:8), <anonymous>:193:23)
    at Array.forEach (native)
    at Object.substrate.addActor.processItem (eval at globalEval (/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/src/compiler.js:105:8), <anonymous>:188:24)
    at Object.Actor.process (eval at globalEval (/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/src/compiler.js:105:8), <anonymous>:248:26)
    at Object.Substrate.solve (eval at globalEval (/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/src/compiler.js:105:8), <anonymous>:184:25)

undefined:54
    throw msg;
          ^
Assertion failed: Arbitrary-sized ints must tails that are of legal size
Traceback (most recent call last):
  File "/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/emscripten.py", line 402, in <module>
    temp_files.run_and_clean(lambda: main(keywords))
  File "/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/tools/shared.py", line 433, in run_and_clean
    return func()
  File "/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/emscripten.py", line 402, in <lambda>
    temp_files.run_and_clean(lambda: main(keywords))
  File "/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/emscripten.py", line 358, in main
    emscript(args.infile, settings, args.outfile, libraries)
  File "/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/emscripten.py", line 228, in emscript
    for func_js, curr_forwarded_data in outputs:
ValueError: need more than 1 value to unpack
Traceback (most recent call last):
  File "/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/emcc", line 1085, in <module>
    final = shared.Building.emscripten(final, append_ext=False, extra_args=extra_args)
  File "/home/shlomif/Download/unpack/prog/llvm-to-js/emscripten/tools/shared.py", line 909, in emscripten
    assert os.path.exists(filename + '.o.js') and len(open(filename + '.o.js', 'r').read()) > 0, 'Emscripten failed to generate .js: ' + str(compiler_output)
AssertionError: Emscripten failed to generate .js: 
make: *** [fc-solve-test.html] Error 1
shlomif@telaviv1:~/TEMP/fc-solve/fc-solve/source$ 

I am using Mageia Linux 3/Cauldron x86-64 with clang-3.2-2.mga3 and if I change "OPT_FLAGS" to "-g" in the Makefile.to-javascript.mak, it works fine.

Thanks again for all your work, and please look into it.

Best regards,

— Shlomi Fish

By the way, I am a packager for Mageia (a free-and-open-source Linux distribution based on Mandriva), and I wonder if you feel that emcc is ready for being packaged for Linux distributions and other operating systems. If this is the case, then it would help if it made use of a solid build system such as CMake or python's setuptools or even GNU Autools. What do you say?

[shlomif]

Replying to myself, I'd like to note that the fc-solve git commit is 9ef34f01cae721a960dda8b9059bc7f64b0750e1 and the emscripten git commit is ada59f0a9d23d8ec19ee6a1326977ddf6e93f5f9

[shlomif]

@kripken : hi!

I should note that after the git clone, you should do:

git checkout 9ef34f01cae721a960dda8b9059bc7f64b0750e1

Because I removed the "-O2" in the master branch, because I went on with preparing this page - http://fc-solve.shlomifish.org/js-fc-solve/text/ - (which is mostly usable but still could use more work). Perhaps I should create a separate git branch for reproducing this bug.

That put aside, I suggest you add your E-mail and/or blog address and/or homepage to https://github.com/kripken like I have on my GitHub user-page. People can always find your E-mail address from the Git commit messages.

Regards,

— Shlomi Fish

[kripken]

Any chance you can update with a full list of the instructions to reproduce this? The last comment says I should do some additional step, but I am not sure when in relation to the other list of instructions. Something as simple as possible so I don't need to exercise any thought when reproducing would be best ;)

[shlomif]

@kripken : hi! Thanks for you response.

OK, here goes. run this using "bash -x reproduce-me.bash" or maybe "set -x" and "source reproduce-me.bash":

#!/bin/bash

test -d fc-solve || git clone https://bitbucket.org/shlomif/fc-solve.git
cd fc-solve
git checkout master
git pull --ff-only origin master
git checkout 9ef34f01cae721a960dda8b9059bc7f64b0750e1
cd fc-solve/source
rm -fr ./build
mkdir build
cd build
cmake -DFCS_WITH_TEST_SUITE= ..
make boards
make -f ../Makefile.to-javascript.mak SRC_DIR=..

I tested it and it appears to reproduce the bug. Sorry for being obscure at first - I'll try to keep it in mind.

Regards,

-- Shlomi Fish

[kripken]

I get some errors,

ack-grep: /home/alon/apps/fcs-for-pysol/share/freecell-solver: No such file or directory
~/Dev/llvmCore-3.2-rc3.install/bin/clang -O2 -I ./build -I . -m32 -std=gnu99 -emit-llvm ../main.c -c -o main.bc
In file included from ../main.c:31:
In file included from /usr/include/string.h:27:
/usr/include/features.h:324:10: fatal error: 'bits/predefs.h' file not found
#include <bits/predefs.h>
         ^
1 error generated.
make: *** [main.bc] Error 1

[shlomif]

Hi @kripken , here is a somewhat more robust (and faster and still self-contained script that you can use instead (but see below for my comments):

#!/bin/bash

test -d fc-solve || git clone https://bitbucket.org/shlomif/fc-solve.git
cd fc-solve
git checkout master
git pull --ff-only origin master
git checkout 9ef34f01cae721a960dda8b9059bc7f64b0750e1
cd fc-solve/source
rm -fr ./build
mkdir build
cd build
cmake -DFCS_BUILD_DOCS= -DFCS_WITH_TEST_SUITE= -DCMAKE_INSTALL_PREFIX="$HOME/apps/fcs-for-pysol" ..
make boards install
make -f ../Makefile.to-javascript.mak SRC_DIR=..

In any case, seems like your system is missing an include that is used by string.h. Freecell Solver only includes standard ISO C headers directly - not "bits/predfs.h" or anything like that. Are you using the 3.2-rc3 version of clang and LLVM? Isn't there already 3.2-FINAL? Does a program that only does an

#include <string.h>

int main()
{  
    return 0;
}

work with a similar invocation? Which distribution and version are you running?

Regards,

-- Shlomi Fish

[shlomif]

@kripken : hi, what's up? Any news? In order to help you (help me - ☺ ), I could use some more information such as what operating system, distribution and version you are running my script on, as well as if you're using a new UNIX user account, and what version of the various emcc dependencies are you using. I think you would also agree that a compiler failing in the middle of compiling "string.h" (with no particularly offending flags), which is a standard ISO C header, is indicative that something is really wrong on the host system, and I am hoping you can investigate and fix this SNAFU.

At the moment I was able to deploy the JS version of my solver on http://fc-solve.shlomifish.org/js-fc-solve/text/ , but without ant optimisations flags, which likely results in both under-performing JS code and one with a considerably larger one. Someone I talked with who used a mobile device (connected to the Internet using a cellular network) and asked to test it, testified that it took a long time to load, and so I'd prefer to remedy as much as I can, as soon as possible.

Sorry if I'm nudging you too much.

Regards,

-- @shlomif .

[kripken]

Sorry for the delay, I've been very busy with other stuff. Taking another look now.

[kripken]

First thing, you should use emcc instead of clang --emit-llvm. The former will use the emscripten headers properly and other stuff that we need.

[kripken]

I built, but there is no main function. You need to have main be in EXPORTED_FUNCTIONS if you want to call it.

[shlomif]

@kripken : hi, thanks for your reply.

OK, after I converted clang --emit-llvm to emcc and replaced the empty OPT_FLAG with -O2, everything is working fine now. Maybe there is a way to fail more gracefully. Thanks for trying to build it, but see the commit at:

https://bitbucket.org/shlomif/fc-solve/commits/ed8fa0181abd56692aca300604d61027a00c28d0

libfreecell-solver.js is a library without a main function, and now the code on this page is working fine with -O2 : http://fc-solve.shlomifish.org/js-fc-solve/text/ . Thanks again.

Regards,

-- Shlomi Fish.

[kripken]

Great, glad it works now :)

Regarding failing more gracefully on that specific problem, I can't think of a good way right now. clang -emit-llvm does generate bitcode, and we can't easily identify it as originating from a non-emcc source.

[juj]

It sounds like the issue was solved, so let's close this. Feel free to reopen or post a new issue if there's something still broken!