search

divvun / libdivvun

lib for running gramcheck and other pipelines + cli; modules for CG→spelling, CG→feedback, tagging blanks
https://giellalt.github.io/proof/gramcheck/GrammarCheckerDocumentation.html
GNU General Public License v3.0
9 stars 1 forks source link

run.xml test heisenbug #46

Closed flammie closed 3 years ago

flammie commented 3 years ago

I commented out a test from test/checker/Makefile.am that prevents building automatic packages but doesn't reproduce systematically on gentoo or fresh hirsute docker with:

FROM ubuntu:hirsute

RUN apt-get update &\
export LC_ALL=C.UTF-8 &\
apt-get install git build-essential autoconf automake pkg-config libtool libpugixml-dev libarchive-dev swig libxml2-utils python3 python3-dev zip language-pack-nb curl gawk &\
curl -sS https://apertium.projectjj.com/apt/install-nightly.sh | bash &\
apt-get install libhfst-dev hfst-ospell-dev cg3-dev &\
export LC_ALL=nb_NO.UTF-8 &\
git clone https://github.com/divvun/libdivvun /tmp/libdivvun &\
cd /tmp/libdivvun &\
./autogen.sh &\
./configure &\
make &\
make check

some fails may show up on hirsute docker if you run interactive session and for _ in $(seq 20) ; do make clean ; make check ; done -_-

snomos commented 3 years ago

@unhammer comments?

unhammer commented 3 years ago

Is it just test/checker/run.xml? I can't reproduce on ubuntu 20.04 focal (maybe because different gcc?)

unhammer commented 3 years ago

Also, what's the error output?

flammie commented 3 years ago

It does randomly appear on all(?) Linux targets on tino's build platform, I haven't saved the logs but I think it fails with an old-form diff between empty file and expected.xml.json or so

unhammer commented 3 years ago

mv Dockerfile.txt Dockerfil && docker build -t divvun/hirsute . runs run.xml 20 times (as well as make check) and gives me no errors :-/ Does that exact Docker run error on your end? Dockerfile.txt

unhammer commented 3 years ago

however, there's lots of memory issues to pick from … some related to CG3, some related to HFST, some seem specific to libdivvun:

$ grep -ic cg3 valgrind-out.txt 
3006
$ grep -ic hfst valgrind-out.txt 
3558
$ grep -ic divvun valgrind-out.txt 
9108

valgrind-out.txt

(This was after $ echo ja |valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose --log-file=valgrind-out.txt ~/PREFIX/gtd-gc/bin/divvun-checker --spec test/checker/pipespec.xml --variant smegram after $ ./configure --prefix="$HOME"/PREFIX/gtd-gc --enable-checker CFLAGS="-ggdb3 -O0" CXXFLAGS="-ggdb3 -O0" LDFLAGS="-ggdb3" && make clean && make install)

flammie commented 3 years ago

I tested in an interactive session so maybe there is some difference to dockerfile running? I only tested valgrind with the exact run.xml invocation and that gives 0 errors, I guess it is in theory possible that some unrelated test can break result of run.xml too...

flammie commented 3 years ago

I have zero errors on that too for gentoo:

echo ja |libtool --mode=execute valgrind --track-origins=yes --verbose src/divvun-checker --spec test/checker/pipespec.xml --variant smegram
==891628== Memcheck, a memory error detector
==891628== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==891628== Using Valgrind-3.16.1-36d6727e1d-20200622X and LibVEX; rerun with -h for copyright info
==891628== Command: /home/flammie/github/divvun/libdivvun/src/.libs/divvun-checker --spec test/checker/pipespec.xml --variant smegram
==891628== 
--891628-- Valgrind options:
--891628--    --track-origins=yes
--891628--    --verbose
--891628-- Contents of /proc/version:
--891628--   Linux version 5.4.97-gentoo-x86_64 (root@teraflammie) (gcc version 9.3.0 (Gentoo 9.3.0-r2 p4)) #1 SMP Thu May 6 20:39:14 CEST 2021
--891628-- 
--891628-- Arch and hwcaps: AMD64, LittleEndian, amd64-cx16-lzcnt-rdtscp-sse3-ssse3-avx-avx2-bmi-f16c-rdrand
--891628-- Page sizes: currently 4096, max supported 4096
--891628-- Valgrind library directory: /usr/lib64/valgrind
--891628-- Reading syms from /home/flammie/github/divvun/libdivvun/src/.libs/divvun-checker
--891628-- Reading syms from /lib64/ld-2.32.so
--891628--   Considering /usr/lib/debug/lib64/ld-2.32.so.debug ..
--891628--   .. CRC is valid
--891628-- Reading syms from /usr/lib64/valgrind/memcheck-amd64-linux
--891628--    object doesn't have a symbol table
--891628--    object doesn't have a dynamic symbol table
--891628-- Scheduler: using generic scheduler lock implementation.
--891628-- Reading suppressions file: /usr/lib64/valgrind/default.supp
==891628== embedded gdbserver: reading from /tmp/vgdb-pipe-from-vgdb-to-891628-by-flammie-on-???
==891628== embedded gdbserver: writing to   /tmp/vgdb-pipe-to-vgdb-from-891628-by-flammie-on-???
==891628== embedded gdbserver: shared mem   /tmp/vgdb-pipe-shared-mem-vgdb-891628-by-flammie-on-???
==891628== 
==891628== TO CONTROL THIS PROCESS USING vgdb (which you probably
==891628== don't want to do, unless you know exactly what you're doing,
==891628== or are doing some strange experiment):
==891628==   /usr/lib64/valgrind/../../bin/vgdb --pid=891628 ...command...
==891628== 
==891628== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==891628==   /path/to/gdb /home/flammie/github/divvun/libdivvun/src/.libs/divvun-checker
==891628== and then give GDB the following command
==891628==   target remote | /usr/lib64/valgrind/../../bin/vgdb --pid=891628
==891628== --pid is optional if only one valgrind process is running
==891628== 
--891628-- REDIR: 0x401fc10 (ld-linux-x86-64.so.2:strlen) redirected to 0x580c9f72 (???)
--891628-- REDIR: 0x401f9f0 (ld-linux-x86-64.so.2:index) redirected to 0x580c9f8c (???)
--891628-- Reading syms from /usr/lib64/valgrind/vgpreload_core-amd64-linux.so
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so
--891628--    object doesn't have a symbol table
==891628== WARNING: new redirection conflicts with existing -- ignoring it
--891628--     old: 0x0401fc10 (strlen              ) R-> (0000.0) 0x580c9f72 ???
--891628--     new: 0x0401fc10 (strlen              ) R-> (2007.0) 0x0483bd80 strlen
--891628-- REDIR: 0x401c430 (ld-linux-x86-64.so.2:strcmp) redirected to 0x483ccb0 (strcmp)
--891628-- REDIR: 0x4020230 (ld-linux-x86-64.so.2:mempcpy) redirected to 0x4840710 (mempcpy)
--891628-- Reading syms from /home/flammie/github/divvun/libdivvun/src/.libs/libdivvun.so.0.0.0
--891628-- Reading syms from /usr/lib64/libcg3.so.1
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libarchive.so.13.5.1
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib/libhfst.so.53.0.0
--891628-- Reading syms from /lib64/libpthread-2.32.so
--891628-- Reading syms from /lib64/libdl-2.32.so
--891628--   Considering /usr/lib/debug/lib64/libdl-2.32.so.debug ..
--891628--   .. CRC is valid
--891628-- Reading syms from /usr/lib64/libhfstospell.so.11.0.0
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libpugixml.so.1.11
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/libstdc++.so.6.0.28
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /lib64/libm-2.32.so
--891628--   Considering /usr/lib/debug/lib64/libm-2.32.so.debug ..
--891628--   .. CRC is valid
--891628-- Reading syms from /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/libgcc_s.so.1
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /lib64/libc-2.32.so
--891628--   Considering /usr/lib/debug/lib64/libc-2.32.so.debug ..
--891628--   .. CRC is valid
--891628-- Reading syms from /usr/lib64/libicuuc.so.68.2
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libicuio.so.68.2
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libicui18n.so.68.2
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libcrypto.so.1.1
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /lib64/libacl.so.1.1.2253
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /lib64/liblzma.so.5.2.5
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /lib64/libbz2.so.1.0.8
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /lib64/libz.so.1.2.11
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libxml2.so.2.9.10
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libxml++-2.6.so.2.0.7
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libglibmm-2.4.so.1.3.0
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libgobject-2.0.so.0.6600.7
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libglib-2.0.so.0.6600.7
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libsigc-2.0.so.0.0.0
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libicudata.so.68.2
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libgmodule-2.0.so.0.6600.7
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /usr/lib64/libffi.so.7.1.0
--891628--    object doesn't have a symbol table
--891628-- Reading syms from /lib64/libpcre.so.1.2.12
--891628--    object doesn't have a symbol table
--891628-- REDIR: 0x6ffcb90 (libc.so.6:strlen) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffd7b0 (libc.so.6:memmove) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffcce0 (libc.so.6:strncpy) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffdab0 (libc.so.6:strcasecmp) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffc7c0 (libc.so.6:strcat) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffcd40 (libc.so.6:rindex) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffee00 (libc.so.6:rawmemchr) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x7017070 (libc.so.6:wmemchr) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x7016c20 (libc.so.6:wcscmp) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffd900 (libc.so.6:mempcpy) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffd740 (libc.so.6:bcmp) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffcc70 (libc.so.6:strncmp) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffc870 (libc.so.6:strcmp) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffd870 (libc.so.6:memset) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x7016be0 (libc.so.6:wcschr) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffcbd0 (libc.so.6:strnlen) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffc940 (libc.so.6:strcspn) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffdb00 (libc.so.6:strncasecmp) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffc8e0 (libc.so.6:strcpy) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffdc50 (libc.so.6:memcpy@@GLIBC_2.14) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x7018250 (libc.so.6:wcsnlen) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x7016c60 (libc.so.6:wcscpy) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffcd80 (libc.so.6:strpbrk) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffc820 (libc.so.6:index) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x7003120 (libc.so.6:memrchr) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffdb50 (libc.so.6:strcasecmp_l) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffd700 (libc.so.6:memchr) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x7016d10 (libc.so.6:wcslen) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffcea0 (libc.so.6:strspn) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffda50 (libc.so.6:stpncpy) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffd9f0 (libc.so.6:stpcpy) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffee40 (libc.so.6:strchrnul) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x6ffdba0 (libc.so.6:strncasecmp_l) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x70c89f0 (libc.so.6:__strrchr_avx2) redirected to 0x483b7c0 (rindex)
--891628-- REDIR: 0x70c8bc0 (libc.so.6:__strlen_avx2) redirected to 0x483bc60 (strlen)
--891628-- REDIR: 0x6ff8d20 (libc.so.6:malloc) redirected to 0x4838730 (malloc)
--891628-- REDIR: 0x6ff9390 (libc.so.6:free) redirected to 0x4839960 (free)
--891628-- REDIR: 0x6ff9b10 (libc.so.6:calloc) redirected to 0x483aad0 (calloc)
--891628-- REDIR: 0x6ff9620 (libc.so.6:realloc) redirected to 0x483ad20 (realloc)
--891628-- REDIR: 0x70cbbb0 (libc.so.6:__memcpy_avx_unaligned_erms) redirected to 0x483f770 (memmove)
--891628-- REDIR: 0x70c40f0 (libc.so.6:__strcmp_avx2) redirected to 0x483cbb0 (strcmp)
--891628-- REDIR: 0x70c85d0 (libc.so.6:__strchr_avx2) redirected to 0x483b940 (index)
--891628-- REDIR: 0x70c9070 (libc.so.6:__strcat_avx2) redirected to 0x483b970 (strcat)
--891628-- REDIR: 0x70cc030 (libc.so.6:__memset_avx2_unaligned_erms) redirected to 0x483f660 (memset)
--891628-- REDIR: 0x70c51d0 (libc.so.6:__memcmp_avx2_movbe) redirected to 0x483ee90 (bcmp)
--891628-- REDIR: 0x6ceeef0 (libstdc++.so.6:operator new(unsigned long)) redirected to 0x4838da0 (operator new(unsigned long))
--891628-- REDIR: 0x6ced1b0 (libstdc++.so.6:operator delete(void*)) redirected to 0x4839e60 (operator delete(void*))
--891628-- REDIR: 0x6ceef40 (libstdc++.so.6:operator new[](unsigned long)) redirected to 0x48394c0 (operator new[](unsigned long))
--891628-- REDIR: 0x70cb190 (libc.so.6:__stpncpy_avx2) redirected to 0x483f560 (stpncpy)
--891628-- REDIR: 0x6ced1e0 (libstdc++.so.6:operator delete[](void*)) redirected to 0x483a540 (operator delete[](void*))
--891628-- REDIR: 0x6ced1c0 (libstdc++.so.6:operator delete(void*, unsigned long)) redirected to 0x483a040 (operator delete(void*, unsigned long))
--891628-- REDIR: 0x70c4d20 (libc.so.6:__rawmemchr_avx2) redirected to 0x48402b0 (rawmemchr)
--891628-- REDIR: 0x70cbb90 (libc.so.6:__mempcpy_avx_unaligned_erms) redirected to 0x4840390 (mempcpy)
--891628-- REDIR: 0x6ffd330 (libc.so.6:__GI_strstr) redirected to 0x4840960 (__strstr_sse2)
--891628-- REDIR: 0x7078ec0 (libc.so.6:__memcpy_chk) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
--891628-- REDIR: 0x70cbba0 (libc.so.6:__memcpy_chk_avx_unaligned_erms) redirected to 0x48407f0 (__memcpy_chk)
--891628-- REDIR: 0x70c4a50 (libc.so.6:__memchr_avx2) redirected to 0x483cd30 (memchr)
--891628-- REDIR: 0x70ca0f0 (libc.so.6:__strcpy_avx2) redirected to 0x483bdb0 (strcpy)
--891628-- REDIR: 0x70c8800 (libc.so.6:__strchrnul_avx2) redirected to 0x4840280 (strchrnul)
--891628-- REDIR: 0x7078f80 (libc.so.6:__memmove_chk) redirected to 0x482e1b0 (_vgnU_ifunc_wrapper)
==891628== WARNING: new redirection conflicts with existing -- ignoring it
--891628--     old: 0x070cbba0 (__memcpy_chk_avx_una) R-> (2030.0) 0x048407f0 __memcpy_chk
--891628--     new: 0x070cbba0 (__memcpy_chk_avx_una) R-> (2024.0) 0x04840210 __memmove_chk
--891628-- REDIR: 0x7079260 (libc.so.6:__strcpy_chk) redirected to 0x48402f0 (__strcpy_chk)
--891628-- REDIR: 0x70ca480 (libc.so.6:__strncpy_avx2) redirected to 0x483bf50 (strncpy)
--891628-- REDIR: 0x70c4530 (libc.so.6:__strncmp_avx2) redirected to 0x483c350 (strncmp)
--891628-- REDIR: 0x70cade0 (libc.so.6:__stpcpy_avx2) redirected to 0x483efb0 (stpcpy)
--891628-- REDIR: 0x70cd500 (libc.so.6:__wcslen_avx2) redirected to 0x4840ce0 (wcslen)
--891628-- REDIR: 0x70c8d50 (libc.so.6:__strnlen_avx2) redirected to 0x483bc20 (strnlen)
{"errs":[],"text":"ja"}
==891628== 
==891628== HEAP SUMMARY:
==891628==     in use at exit: 223,356 bytes in 619 blocks
==891628==   total heap usage: 716,230 allocs, 715,611 frees, 73,030,523 bytes allocated
==891628== 
==891628== Searching for pointers to 619 not-freed blocks
==891628== Checked 27,414,048 bytes
==891628== 
==891628== LEAK SUMMARY:
==891628==    definitely lost: 483 bytes in 6 blocks
==891628==    indirectly lost: 0 bytes in 0 blocks
==891628==      possibly lost: 1,352 bytes in 18 blocks
==891628==    still reachable: 221,521 bytes in 595 blocks
==891628==                       of which reachable via heuristic:
==891628==                         newarray           : 1,536 bytes in 16 blocks
==891628==         suppressed: 0 bytes in 0 blocks
==891628== Rerun with --leak-check=full to see details of leaked memory
==891628== 
==891628== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
unhammer commented 3 years ago

… nevermind, all those valgrind errors were because I had LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4 ( github.com/gperftools/gperftools/issues/792#issuecomment-209056616 ). (Wish valgrind would warn about that!)

unhammer commented 3 years ago

valgrind-out.txt with unset LD_PRELOAD, same invocation

unhammer commented 3 years ago

There are 7 places where things get "definitely lost". All lose one block on startup (which I guess is fine if it's stuff that's supposed to be kept while we're running and then get lost on exit?), but there's one that loses a block for every word (here run with about 14 words of input):

==1808994== 672 bytes in 14 blocks are definitely lost in loss record 327 of 358
==1808994==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)                                                    ==1808994==    by 0x5549BD9: hfst_ol::Transducer::lookup_fd[abi:cxx11](char const*, long, double) (in /usr/lib/x86_64-linux-gnu/libhfst.so.53.0.0)
==1808994==    by 0x52D7CB5: hfst::HfstTransducer::lookup_fd(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, double) const (in /usr/lib/
x86_64-linux-gnu/libhfst.so.53.0.0)                                                                                                                                                   
==1808994==    by 0x4A550FF: divvun::Blanktag::proc(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_st
ring<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::v
ector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (blanktag.cpp:46)
==1808994==    by 0x4A5587C: divvun::Blanktag::run(std::istream&, std::ostream&) (blanktag.cpp:83)                                                                                    ==1808994==    by 0x49F328D: divvun::BlanktagCmd::run(std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_stringstream<char, s
td::char_traits<char>, std::allocator<char> >&) const (pipeline.cpp:143)                                                                                                              ==1808994==    by 0x49F66B9: divvun::Pipeline::proc(std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_stringstream<char, std
::char_traits<char>, std::allocator<char> >&) (pipeline.cpp:401)                                                                                                                      ==1808994==    by 0x175899: run(divvun::Pipeline&) (main_checker.cpp:82)
==1808994==    by 0x1760C8: main::{lambda(divvun::Pipeline&)#2}::operator()(divvun::Pipeline&) const (main_checker.cpp:190)
unhammer commented 3 years ago

argh github "in case we fixed 46" does not mean I meant to close it, have to see if projectjj now builds fine again

unhammer commented 3 years ago 

=========================================================
   Divvun gramcheck 0.3.9: test/checker/test-suite.log
=========================================================

# TOTAL: 6
# PASS:  4
# SKIP:  1
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: ./run.xml
===============

+ ../../src/divvun-checker -s pipespec.xml -n smegram
divvun-suggest: WARNING: No <description> for "double-space-before" in xml:lang 'se', trying 'en'
divvun-suggest: WARNING: No <description> for "double-space-before" in any xml:lang
0a1
> {"errs":[["dieđuiguin",20,30,"msyn-valency-loc-com","boasttut sátni",["diehtukorrekt"],"msyn thingy"],["vel",36,39,"double-space-before","double-space-before",[],"double-space-before"]],"text":"seammas ballat ođđa dieđuiguin. Ja  vel."}
diff test/checker/output.xml.json test/checker/expected.xml.json
FAIL run.xml (exit status: 1)

says today's https://apertium.projectjj.com/apt/logs/libdivvun/xenial-amd64.log so the bug is still there. Seems like we're getting zero output. Is there any chance this is not a memory issue?

flammie commented 3 years ago

such a random and non-systematic bug would strongly point to memory issue, but valgrind does not find any in the relevant test... Other possibilities I have thought of would be that the error is in test scripts, race conditions or just shell script bugs perhaps

unhammer commented 3 years ago

It's the same test/checker/run that handles both run.xml and run.archive, but only the xml one fails. The only difference in the shell scripts is that one passes xml as the name of the tests and -s pipespec.xml as arguments, while the other uses archive as test name and -a sme.zcheck as arguments. Nothing is backgrounded in the bash script (and it passes shellcheck). So it seems like the error is in the C++.

flammie commented 3 years ago

probably fixed now in https://github.com/divvun/libdivvun/commit/a1b0fd2074c425fbe9e892e7aa03d724652f25cc