Many tests report: buffer overflow detected

svenha commented 4 years ago

For the current hop 3.3.0-pre1, most tests (from make test) fail. Many of these failures show the following message:

*** buffer overflow detected ***: /var/tmp/hop-3.3.0-pre1/bin/hop terminated

The underlying bigloo is 4.3g-2, 64bit, compiled with gcc 9.2.1.

For hop 3.2.0-pre2, the situation is a little better: 40 of 146 tests fail; all of them due to the overflow detection.

(BTW: the colors red and green for ok and failure in the output of make test seem to be switched.) Any help very much appreciated.

manuel-serrano commented 4 years ago

Hi Sven,

I'm actively working on the version 3.3.0-pre1 so fixing this problem is just the right moment. Could you tell me how you have detected these messages (that are not necessarily "real" errors).

svenha commented 4 years ago

Hi Manuel. Thanks for replying.

I'm building hop with a normal bigloo build; bigloo's configure line is:

--disable-pcre --coflags="-O2" --cflags="-O2 -fno-strict-aliasing -ftree-slp-vectorize -ftree-vectorize -funroll-loops"

The make test will show the following type of errors. The first that fails for 3.2.0 is:

5. noserv/cache.js ... *** buffer overflow detected ***: /var/tmp/hop-3.2.0-pre2/bin/hop terminated
./TEST: line 78: 20617 Aborted                 (core dumped) $hop $oflags $* > $TMP/TEST HOPJS.out 2>&1
failure!

Some tests will lead to bigloo processes with more than 8 GB RAM.

svenha commented 4 years ago

@manuel-serrano How can I run a test in isolation?

svenha commented 4 years ago

The RAM explosion is always caused by a bigloo subprocess started by hop process of the individual test. It can grow to 22 GB (virtual an RSS) and more; I have to kill them in htop:

bigloo - -fread-internal-src-file-name /var/tmp/hop-3.2.0-pre2/test/hopjs/noserv/es6-generator.js -fread-internal-src -srfi enable-libuv -srfi hopc -library hopscript -library web -library nodejs -library libuv --force-cc-o -rpath /usr/local/lib/hop/3.2.0 -dload-sym -y -o /home/nn/.config/hop/libs/3.2.0/62fc95454a3077ad7dab93209ce018cc/linux-x86_64/#es6-generator-58be7769eea85ec3c876d6e06412c80a.so -unsafe -safel -O3 -L /usr/local/lib/hop/3.2.0 -fisa -freturn

manuel-serrano commented 4 years ago

Several things here:

there is something wrong going on with none PCRE installation. I'm working on it.
to run a single test: ./TEST -v the-test
if you want to access the intermediate generated Scheme file, use the hopc -t option. For instance, try hopc -t foo.scm foo.js -v4 -- -v2. Using this line you will be able to inspect everything.
the es6-generator.js is expected be to very large. It contains pathological cases of CPS ransformations

All these said, I come back to you when I have fixed the non-pcre problem (the problem is likely to be in Bigloo and not in Hop).

svenha commented 4 years ago

I repeated the hop-3.3.0 build with a bigloo plus pcre (i.e. no --disable-pcre). This worked almost perfectly. Only 1 out of 222 tests failed:

129. noserv/parserFailure1.js (opt) ... ./TEST: line 79: 30189 Aborted                 (core dumped) $hop $oflags $* > $TMP/TEST-HOPJS.out 2>&1
failure

More details from s separate run:

 # ./TEST -v noserv/parserFailure1.js
 Logs in /tmp/TEST-HOPJS-ALL.out
1. noserv/parserFailure1.js (debug)...
/var/tmp/hop-3.3.0-pre1/bin/hop.sh --no-zeroconf noserv/parserFailure1.js --sofile-policy none -g --clear-libs -q --no-server ...ok.
2. noserv/parserFailure1.js (regular) ...
/var/tmp/hop-3.3.0-pre1/bin/hop.sh --no-zeroconf noserv/parserFailure1.js --sofile-policy nte+ --clear-libs -q --no-server ...ok.
3. noserv/parserFailure1.js (opt) ...
/var/tmp/hop-3.3.0-pre1/bin/hop.sh --no-zeroconf noserv/parserFailure1.js -q --no-server ...*** buffer overflow detected ***: /var/tmp/hop-3.3.0-pre1/bin/hop terminated
./TEST: line 79:  7349 Aborted                 (core dumped) $hop $oflags $* > $TMP/TEST-HOPJS.out 2>&1
failure!
catched a failure
*** ERROR:(&pthread-1105 #unspecified):
stack overflow -- #<dynamic-env:3 executed
2 success
1 failure

manuel-serrano commented 4 years ago

I have fixed the non-pcre problem. I still have to test if that's correct now. I will keep you informed.

manuel-serrano commented 4 years ago

I have fixed the non-pcre problem (this untested code was just pure non-sens). It now works. I'm uploading the new Bigloo version (please note the naming schema change for the future versions). Currently it is branded bigloo-unstable.

svenha commented 4 years ago

Thanks for the quick help, @manuel-serrano . The new bigloo and hop version work for me, even with --disable-pcre for bigloo's configure. (But if you have time, please have a look at the one remaining test problem parserFailure1 in optimized mode.)

Do you remember when this bug was introduced? (It might explain another nasty problem that I never had time to track down.)

manuel-serrano commented 4 years ago

No, I don't remember. This was a nasty bug coming from two different reasons.

First, on optimization of the PCRE based implementation has changed the prototytpes of two functions that must identical in the PCRE and non-PCRE implementation. The change had not been reflected to non-PCRE implementation.
The non-PCRE implementation contained obvious bugs that were difficult to detect because of problem 1.

manuel-serrano / hop

Many tests report: buffer overflow detected #74