Emitting and loading a DBProgramTrace instead of a TDProgramTrace

[m-carrasco]

Hi,

I'm learning to use polytracker (v4.0.0 - pip), and I'd like to create a DBProgramTrace rather than a TDProgramTrace. However, I cannot successfully generate and load a DBProgramTrace. I could only find docs for TDProgramTrace.

I understand that a DBProgramTrace should be loaded using DBProgramTrace.load . This function expects a .db rather than a .tdag. How could I make the instrumented program emit a .db file? So far, I am only able to emit a .tdag.

I've tried the following:

1. polytracker trace run INSTRUMENTED_BIN INPUT_FILE
2. polytracker trace run INSTRUMENTED_BIN INPUT_FILE -o polytracker.db
3. (REPL) run_trace(INSTRUMENTED_BIN, INPUT_FILE, True, "output")
4. ./INSTRUMENTED_BIN INPUT_FILE

In all these cases, DBProgramTrace.load fails saying that it expects a database.

Let me give you more context. I want to use polytracker to know which input bytes are used in the dataflow after a particular function call. DBProgramTrace seems to offer more functionalities that may help (e.g. access_sequence). Does it make any sense? I would appreciate any feedback you can give me.

I'm sorry if this is already explained elsewhere. Thanks for sharing this project :+1:.

Best regards, Manuel

[surovic]

Hi Manuel!

First of all, thank you for using polytracker! We're always happy to see new users. The less fortunate thing is that DBProgramTrace and all code that requires it is slowly being phased out and as of 4.0.0, I think I can say that it's been obsoleted. We're currently in the middle of a big treewide usability and performance refactor so the functionality you need isn't available in 4.0.0. However, we’re in the process of re-implementing that functionality ASAP.

In the meantime, you may have more luck using an older version of polytracker, for example 3.0.0. I'm not really sure if it's available via pip. If not, you can try building 3.0.0 from source according to the README.md of that particular version. You can let me know if you run into any issues doing that.

Best regards, Marek

[m-carrasco]

@surovic Thanks a lot for your detailed answer! I hope this issue may show a possible use case for future versions.

It looks like version 3.0.1 is available in pip. Otherwise, I will try version v3 by building its Dockerfile directly.

At first glance, would you think my goal could be achieved by the features present in version 3.0.0?

Best regards, Manuel.

[surovic]

At first glance, yes I think it should be possible.

[m-carrasco]

Thanks @surovic! I will close the ticket for now.

[m-carrasco]

Hi @surovic :wave:

Sorry for reopening this issue. I just wanted to ask about some warnings I got using v3.0.1.

> git clone https://github.com/sammycage/lunasvg
> cd lunasvg
> pip install polytracker==3.0.1
> polytracker docker pull
> polytracker docker run
# From now on, we are in the container
> polytracker --version 
3.0.1
> mkdir build && cd build
> make -j4
> cd example
> get-bc -b svg2png
# This is just to provide more context in case it is relevant
> ldd svg2png 
        linux-vdso.so.1 (0x00007ffcdfbef000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f224f285000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f224eee7000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f224eccf000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f224e8de000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f224f4a4000)
> polybuild++ --lower-bitcode -i svg2png.bc -o svg2png.instrumented --libs m pthread
# TO-DO: Download any example.svg
> POLYPATH=/workdir/example.svg ./svg2png.instrumented /workdir/example.svg &> output.log

While executing, I got the following warnings; this is just a sample, output.log is huge:

=2425==WARNING: DataFlowSanitizer: call to uninstrumented function newlocale
==2425==WARNING: DataFlowSanitizer: call to uninstrumented function mbtowc
==2425==WARNING: DataFlowSanitizer: call to uninstrumented function __ctype_get_mb_cur_max
POLYPATH:      /workdir/example.svg
POLYDB:        polytracker.db
POLYFUNC:      0
POLYTRACE:     0
POLYSTART:     0
POLYEND:       116
POLYTTL:        
POLYSAVEINPUT: 1
Warning: Could not resolve the function entry associated with the return from function `_ZN10__cxxabiv112_GLOBAL__N_110construct_Ev` index 6608 to `main` index 121 (expected to be returning from function `_ZN10__cxxabiv112_GLOBAL__N_110construct_Ev` index 6608). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
### fopen, filename is : /workdir/example.svg, fd is 0x55592a3a4fd0 
fopen: adding new taint info!
Warning: Could not resolve the function entry associated with the return from function `_ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC2Ev` index 113 to `_ZN7lunasvg8Document12loadFromFileERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE` index 272 (expected to be returning from function `_ZNSt3__117__compressed_pairINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE5__repES5_EC2INS_18__default_init_tagESA_EEOT_OT0_` index 47). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
### fread, fd is 0x55592a3a4fd0 
### fread, fd is 0x55592a3a4fd0 
==2425==WARNING: DataFlowSanitizer: call to uninstrumented function fseeko
### close, fd is 0x55592a3a4fd0, ret is 0 
Warning: Could not resolve the function entry associated with the return from function `_ZNSt3__113basic_filebufIcNS_11char_traitsIcEEE5closeEv` index 282 to `_ZNSt3__114basic_ifstreamIcNS_11char_traitsIcEEE5closeEv` index 278 (expected to be returning from function `_ZNSt3__113basic_filebufIcNS_11char_traitsIcEEE6setbufEPcl` index 298). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
[...]
==2425==WARNING: DataFlowSanitizer: call to uninstrumented function abs
==2425==WARNING: DataFlowSanitizer: call to uninstrumented function abs
==2425==WARNING: DataFlowSanitizer: call to uninstrumented function abs
[...]
## fopen, filename is : example.svg.png, fd is 0x55592a572430 
### close, fd is 0x55592a572430, ret is 0 
Warning: Could not resolve the function entry associated with the return from function `_ZNKSt3__19basic_iosIcNS_11char_traitsIcEEE5widenEc` index 54 to `_ZNKSt3__19basic_iosIcNS_11char_traitsIcEEE4fillEv` index 28 (expected to be returning from function `_ZNKSt3__15ctypeIcE5widenEc` index 56). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
Generated PNG file : Warning: Could not resolve the function entry associated with the return from function `_ZNSt3__124__put_character_sequenceIcNS_11char_traitsIcEEEERNS_13basic_ostreamIT_T0_EES7_PKS4_m` index 24 to `_ZNSt3__1lsINS_11char_traitsIcEEEERNS_13basic_ostreamIcT_EES6_PKc` index 22 (expected to be returning from function `_ZNSt3__116__pad_and_outputIcNS_11char_traitsIcEEEENS_19ostreambuf_iteratorIT_T0_EES6_PKS4_S8_S8_RNS_8ios_baseES4_` index 29). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
[...]
==2425==WARNING: DataFlowSanitizer: call to uninstrumented function wcrtomb
==2425==WARNING: DataFlowSanitizer: call to uninstrumented function wcrtomb
[...]
==2423==WARNING: DataFlowSanitizer: call to uninstrumented function _setjmp
==2423==WARNING: DataFlowSanitizer: call to uninstrumented function longjmp
==2423==WARNING: DataFlowSanitizer: call to uninstrumented function _setjmp
==2423==WARNING: DataFlowSanitizer: call to uninstrumented function _setjmp
Warning: Could not resolve the function entry associated with the return from function `gray_convert_glyph` index 4648 to `gray_raster_render` index 4646 (expected to be returning from function `gray_record_cell` index 4654). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
Warning: Could not resolve the function entry associated with the return from function `gray_raster_render` index 4646 to `PVG_FT_Raster_Render` index 4645 (expected to be returning from function `gray_record_cell` index 4654). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
Warning: Could not resolve the function entry associated with the return from function `PVG_FT_Raster_Render` index 4645 to `plutovg_rle_rasterize` index 4627 (expected to be returning from function `gray_record_cell` index 4654). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
Warning: Could not resolve the function entry associated with the return from function `plutovg_rle_rasterize` index 4627 to `plutovg_fill_preserve` index 4502 (expected to be returning from function `gray_record_cell` index 4654). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
Warning: Could not resolve the function entry associated with the return from function `plutovg_fill_preserve` index 4502 to `plutovg_fill` index 4501 (expected to be returning from function `gray_record_cell` index 4654). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
Warning: Could not resolve the function entry associated with the return from function `plutovg_fill` index 4501 to `_ZN7lunasvg6Canvas4fillERKNS_4PathERKNS_9TransformENS_8WindRuleENS_9BlendModeEd` index 3378 (expected to be returning from function `gray_record_cell` index 4654). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
Warning: Could not resolve the function entry associated with the return from function `_ZN7lunasvg6Canvas4fillERKNS_4PathERKNS_9TransformENS_8WindRuleENS_9BlendModeEd` index 3378 to `_ZNK7lunasvg8FillData4fillERNS_11RenderStateERKNS_4PathE` index 2746 (expected to be returning from function `gray_record_cell` index 4654). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
^C

I wanted to double-check these warnings, in particular those that seem to refer to the C++ standard library:

_ZNSt3__113basic_filebufIcNS_11char_traitsIcEEE5closeEv which demangled is std::__1::basic_filebuf<char, std::__1::char_traits<char> >::close()

And those that refer to lunasvg's functions, that are at the end:

Could not resolve the function entry associated with the return from function `gray_convert_glyph` index 4648 to `gray_raster_render` index 4646 (expected to be returning from function `gray_record_cell` index 4654).

The polytracker.db file is generated. I just wanted to be safe and ask before assuming that polytracker.db is completely sound.

I apologise if these warnings can be safely ignored!

Best regards, Manuel

[surovic]

Hi @m-carrasco!

We think that the warnings might be related to an interaction between our instrumentation and compiler optimizations. Are you building lunasvg with optimizations turned on?

Bests, Marek

[m-carrasco]

Hi @surovic

Thanks for spending time on this issue :smile:

I've tried the following cmake command but I still get the same warnings:

cmake -DLUNASVG_BUILD_EXAMPLES=ON -DCMAKE_CXX_FLAGS="-O0" -DCMAKE_C_FLAGS="-O0" ..

I also checked the examples of 3.0.1 and couldn't find a particular set of compiler flags. Would you suggest using something different?

Here I share the output of the different commands I previously executed.

root@97885b851cac:/workdir/build# cmake -DLUNASVG_BUILD_EXAMPLES=ON -DCMAKE_CXX_FLAGS="-O0" -DCMAKE_C_FLAGS="-O0" ..
-- The CXX compiler identification is Clang 13.0.0
-- The C compiler identification is Clang 13.0.0
-- Check for working CXX compiler: /polytracker/build/bin/polybuild_script++
-- Check for working CXX compiler: /polytracker/build/bin/polybuild_script++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /polytracker/build/bin/polybuild_script
-- Check for working C compiler: /polytracker/build/bin/polybuild_script -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
You have called ADD_LIBRARY for library lunasvg without any source files. This typically indicates a problem with your CMakeLists.txt file
-- Configuring done
-- Generating done
-- Build files have been written to: /workdir/build

The ADD_LIBRARY warning is only triggered when building inside polytracker's container.

root@97885b851cac:/workdir/build# VERBOSE=1 make -j4
/usr/bin/cmake -H/workdir -B/workdir/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /workdir/build/CMakeFiles /workdir/build/CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/workdir/build'
make -f CMakeFiles/lunasvg.dir/build.make CMakeFiles/lunasvg.dir/depend
make[2]: Entering directory '/workdir/build'
cd /workdir/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /workdir /workdir /workdir/build /workdir/build /workdir/build/CMakeFiles/lunasvg.dir/DependInfo.cmake --color=
Dependee "/workdir/build/CMakeFiles/lunasvg.dir/DependInfo.cmake" is newer than depender "/workdir/build/CMakeFiles/lunasvg.dir/depend.internal".
Dependee "/workdir/build/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/workdir/build/CMakeFiles/lunasvg.dir/depend.internal".
Scanning dependencies of target lunasvg
make[2]: Leaving directory '/workdir/build'
make -f CMakeFiles/lunasvg.dir/build.make CMakeFiles/lunasvg.dir/build
make[2]: Entering directory '/workdir/build'
[ 12%] Building CXX object CMakeFiles/lunasvg.dir/source/element.cpp.o
[ 12%] Building CXX object CMakeFiles/lunasvg.dir/source/property.cpp.o
[ 12%] Building CXX object CMakeFiles/lunasvg.dir/source/lunasvg.cpp.o
[ 12%] Building CXX object CMakeFiles/lunasvg.dir/source/parser.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/lunasvg.cpp.o -c /workdir/source/lunasvg.cpp
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/element.cpp.o -c /workdir/source/element.cpp
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/property.cpp.o -c /workdir/source/property.cpp
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/parser.cpp.o -c /workdir/source/parser.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stKQXH67: Failed to find link section for section 519
objcopy: CMakeFiles/lunasvg.dir/source/stKQXH67: Failed to find link section for section 519
[ 15%] Building CXX object CMakeFiles/lunasvg.dir/source/layoutcontext.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/layoutcontext.cpp.o -c /workdir/source/layoutcontext.cpp
objcopy: CMakeFiles/lunasvg.dir/source/sttNRMce: Failed to find link section for section 823
objcopy: CMakeFiles/lunasvg.dir/source/sttNRMce: Failed to find link section for section 823
[ 18%] Building CXX object CMakeFiles/lunasvg.dir/source/canvas.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/canvas.cpp.o -c /workdir/source/canvas.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stVeygJI: Failed to find link section for section 828
objcopy: CMakeFiles/lunasvg.dir/source/stVeygJI: Failed to find link section for section 828
[ 21%] Building CXX object CMakeFiles/lunasvg.dir/source/clippathelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/clippathelement.cpp.o -c /workdir/source/clippathelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stDyhIzP: Failed to find link section for section 243
objcopy: CMakeFiles/lunasvg.dir/source/stDyhIzP: Failed to find link section for section 243
[ 25%] Building CXX object CMakeFiles/lunasvg.dir/source/defselement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/defselement.cpp.o -c /workdir/source/defselement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stziHW7f: Failed to find link section for section 2229
objcopy: CMakeFiles/lunasvg.dir/source/stziHW7f: Failed to find link section for section 2229
[ 28%] Building CXX object CMakeFiles/lunasvg.dir/source/gelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/gelement.cpp.o -c /workdir/source/gelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stb9hD7m: Failed to find link section for section 5737
objcopy: CMakeFiles/lunasvg.dir/source/stb9hD7m: Failed to find link section for section 5737
[ 31%] Building CXX object CMakeFiles/lunasvg.dir/source/geometryelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/geometryelement.cpp.o -c /workdir/source/geometryelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stWq4aAw: Failed to find link section for section 571
objcopy: CMakeFiles/lunasvg.dir/source/stWq4aAw: Failed to find link section for section 571
objcopy: CMakeFiles/lunasvg.dir/source/stscIH8z: Failed to find link section for section 493
objcopy: CMakeFiles/lunasvg.dir/source/stscIH8z: Failed to find link section for section 493
[ 34%] Building CXX object CMakeFiles/lunasvg.dir/source/graphicselement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/graphicselement.cpp.o -c /workdir/source/graphicselement.cpp
[ 37%] Building CXX object CMakeFiles/lunasvg.dir/source/maskelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/maskelement.cpp.o -c /workdir/source/maskelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/strv1aPZ: Failed to find link section for section 586
objcopy: CMakeFiles/lunasvg.dir/source/strv1aPZ: Failed to find link section for section 586
[ 40%] Building CXX object CMakeFiles/lunasvg.dir/source/markerelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/markerelement.cpp.o -c /workdir/source/markerelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stsWOICh: Failed to find link section for section 223
objcopy: CMakeFiles/lunasvg.dir/source/stsWOICh: Failed to find link section for section 223
[ 43%] Building CXX object CMakeFiles/lunasvg.dir/source/paintelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/paintelement.cpp.o -c /workdir/source/paintelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stbpnANm: Failed to find link section for section 1559
objcopy: CMakeFiles/lunasvg.dir/source/stbpnANm: Failed to find link section for section 1559
[ 46%] Building CXX object CMakeFiles/lunasvg.dir/source/stopelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/stopelement.cpp.o -c /workdir/source/stopelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stDzY6fr: Failed to find link section for section 565
objcopy: CMakeFiles/lunasvg.dir/source/stDzY6fr: Failed to find link section for section 565
[ 50%] Building CXX object CMakeFiles/lunasvg.dir/source/styledelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/styledelement.cpp.o -c /workdir/source/styledelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stKkE4KN: Failed to find link section for section 565
objcopy: CMakeFiles/lunasvg.dir/source/stKkE4KN: Failed to find link section for section 565
[ 53%] Building CXX object CMakeFiles/lunasvg.dir/source/styleelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/styleelement.cpp.o -c /workdir/source/styleelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stYhFuH8: Failed to find link section for section 485
objcopy: CMakeFiles/lunasvg.dir/source/stYhFuH8: Failed to find link section for section 485
[ 56%] Building CXX object CMakeFiles/lunasvg.dir/source/svgelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/svgelement.cpp.o -c /workdir/source/svgelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stDD4Pbd: Failed to find link section for section 216
objcopy: CMakeFiles/lunasvg.dir/source/stDD4Pbd: Failed to find link section for section 216
[ 59%] Building CXX object CMakeFiles/lunasvg.dir/source/symbolelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/symbolelement.cpp.o -c /workdir/source/symbolelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/stzbTRWp: Failed to find link section for section 2030
objcopy: CMakeFiles/lunasvg.dir/source/stzbTRWp: Failed to find link section for section 2030
[ 62%] Building CXX object CMakeFiles/lunasvg.dir/source/useelement.cpp.o
/polytracker/build/bin/polybuild_script++   -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu++1z -o CMakeFiles/lunasvg.dir/source/useelement.cpp.o -c /workdir/source/useelement.cpp
objcopy: CMakeFiles/lunasvg.dir/source/st2JzMZA: Failed to find link section for section 477
objcopy: CMakeFiles/lunasvg.dir/source/st2JzMZA: Failed to find link section for section 477
[ 65%] Building C object CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg.c.o
/polytracker/build/bin/polybuild_script  -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu11 -o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg.c.o   -c /workdir/3rdparty/plutovg/plutovg.c
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stHN0UzK: Failed to find link section for section 8
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stHN0UzK: Failed to find link section for section 8
[ 68%] Building C object CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-paint.c.o
/polytracker/build/bin/polybuild_script  -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu11 -o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-paint.c.o   -c /workdir/3rdparty/plutovg/plutovg-paint.c
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stXOO93T: Failed to find link section for section 8
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stXOO93T: Failed to find link section for section 8
[ 71%] Building C object CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-geometry.c.o
/polytracker/build/bin/polybuild_script  -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu11 -o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-geometry.c.o   -c /workdir/3rdparty/plutovg/plutovg-geometry.c
objcopy: CMakeFiles/lunasvg.dir/source/stP7v05S: Failed to find link section for section 714
objcopy: CMakeFiles/lunasvg.dir/source/stP7v05S: Failed to find link section for section 714
[ 75%] Building C object CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-blend.c.o
/polytracker/build/bin/polybuild_script  -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu11 -o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-blend.c.o   -c /workdir/3rdparty/plutovg/plutovg-blend.c
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stsALT6U: Failed to find link section for section 11
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stsALT6U: Failed to find link section for section 11
objcopy: CMakeFiles/lunasvg.dir/source/stqqu76Y: Failed to find link section for section 485
objcopy: CMakeFiles/lunasvg.dir/source/stqqu76Y: Failed to find link section for section 485
[ 78%] Building C object CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-rle.c.o
/polytracker/build/bin/polybuild_script  -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu11 -o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-rle.c.o   -c /workdir/3rdparty/plutovg/plutovg-rle.c
[ 81%] Building C object CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-dash.c.o
/polytracker/build/bin/polybuild_script  -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu11 -o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-dash.c.o   -c /workdir/3rdparty/plutovg/plutovg-dash.c
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stvzPrh3: Failed to find link section for section 10
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stvzPrh3: Failed to find link section for section 10
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stz4YSZ4: Failed to find link section for section 7
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stz4YSZ4: Failed to find link section for section 7
[ 84%] Building C object CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-ft-raster.c.o
/polytracker/build/bin/polybuild_script  -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu11 -o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-ft-raster.c.o   -c /workdir/3rdparty/plutovg/plutovg-ft-raster.c
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stk2aC18: Failed to find link section for section 11
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stk2aC18: Failed to find link section for section 11
[ 87%] Building C object CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-ft-stroker.c.o
/polytracker/build/bin/polybuild_script  -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu11 -o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-ft-stroker.c.o   -c /workdir/3rdparty/plutovg/plutovg-ft-stroker.c
[ 90%] Building C object CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-ft-math.c.o
/polytracker/build/bin/polybuild_script  -I/workdir/include -I/workdir/source -I/workdir/3rdparty/plutovg  -O0   -std=gnu11 -o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-ft-math.c.o   -c /workdir/3rdparty/plutovg/plutovg-ft-math.c
objcopy: CMakeFiles/lunasvg.dir/source/stHDjHqd: Failed to find link section for section 664
objcopy: CMakeFiles/lunasvg.dir/source/stHDjHqd: Failed to find link section for section 664
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/st6vcYfc: Failed to find link section for section 7
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/st6vcYfc: Failed to find link section for section 7
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stSDQlof: Failed to find link section for section 8
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stSDQlof: Failed to find link section for section 8
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stuPi7qi: Failed to find link section for section 8
objcopy: CMakeFiles/lunasvg.dir/3rdparty/plutovg/stuPi7qi: Failed to find link section for section 8
[ 93%] Linking CXX static library liblunasvg.a
/usr/bin/cmake -P CMakeFiles/lunasvg.dir/cmake_clean_target.cmake
/usr/bin/cmake -E cmake_link_script CMakeFiles/lunasvg.dir/link.txt --verbose=1
/usr/bin/ar qc liblunasvg.a  CMakeFiles/lunasvg.dir/source/lunasvg.cpp.o CMakeFiles/lunasvg.dir/source/element.cpp.o CMakeFiles/lunasvg.dir/source/property.cpp.o CMakeFiles/lunasvg.dir/source/parser.cpp.o CMakeFiles/lunasvg.dir/source/layoutcontext.cpp.o CMakeFiles/lunasvg.dir/source/canvas.cpp.o CMakeFiles/lunasvg.dir/source/clippathelement.cpp.o CMakeFiles/lunasvg.dir/source/defselement.cpp.o CMakeFiles/lunasvg.dir/source/gelement.cpp.o CMakeFiles/lunasvg.dir/source/geometryelement.cpp.o CMakeFiles/lunasvg.dir/source/graphicselement.cpp.o CMakeFiles/lunasvg.dir/source/maskelement.cpp.o CMakeFiles/lunasvg.dir/source/markerelement.cpp.o CMakeFiles/lunasvg.dir/source/paintelement.cpp.o CMakeFiles/lunasvg.dir/source/stopelement.cpp.o CMakeFiles/lunasvg.dir/source/styledelement.cpp.o CMakeFiles/lunasvg.dir/source/styleelement.cpp.o CMakeFiles/lunasvg.dir/source/svgelement.cpp.o CMakeFiles/lunasvg.dir/source/symbolelement.cpp.o CMakeFiles/lunasvg.dir/source/useelement.cpp.o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg.c.o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-paint.c.o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-geometry.c.o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-blend.c.o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-rle.c.o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-dash.c.o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-ft-raster.c.o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-ft-stroker.c.o CMakeFiles/lunasvg.dir/3rdparty/plutovg/plutovg-ft-math.c.o
/usr/bin/ranlib liblunasvg.a
make[2]: Leaving directory '/workdir/build'
[ 93%] Built target lunasvg
make -f example/CMakeFiles/svg2png.dir/build.make example/CMakeFiles/svg2png.dir/depend
make[2]: Entering directory '/workdir/build'
cd /workdir/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /workdir /workdir/example /workdir/build /workdir/build/example /workdir/build/example/CMakeFiles/svg2png.dir/DependInfo.cmake --color=
Dependee "/workdir/build/example/CMakeFiles/svg2png.dir/DependInfo.cmake" is newer than depender "/workdir/build/example/CMakeFiles/svg2png.dir/depend.internal".
Dependee "/workdir/build/example/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/workdir/build/example/CMakeFiles/svg2png.dir/depend.internal".
Scanning dependencies of target svg2png
make[2]: Leaving directory '/workdir/build'
make -f example/CMakeFiles/svg2png.dir/build.make example/CMakeFiles/svg2png.dir/build
make[2]: Entering directory '/workdir/build'
[ 96%] Building CXX object example/CMakeFiles/svg2png.dir/svg2png.cpp.o
cd /workdir/build/example && /polytracker/build/bin/polybuild_script++   -I/workdir/include  -O0   -std=gnu++14 -o CMakeFiles/svg2png.dir/svg2png.cpp.o -c /workdir/example/svg2png.cpp
objcopy: CMakeFiles/svg2png.dir/stLo7j8O: Failed to find link section for section 393
objcopy: CMakeFiles/svg2png.dir/stLo7j8O: Failed to find link section for section 393
[100%] Linking CXX executable svg2png
cd /workdir/build/example && /usr/bin/cmake -E cmake_link_script CMakeFiles/svg2png.dir/link.txt --verbose=1
/polytracker/build/bin/polybuild_script++  -O0  -rdynamic CMakeFiles/svg2png.dir/svg2png.cpp.o  -o svg2png ../liblunasvg.a 
make[2]: Leaving directory '/workdir/build'
[100%] Built target svg2png
make[1]: Leaving directory '/workdir/build'
/usr/bin/cmake -E cmake_progress_start /workdir/build/CMakeFiles 0

From what I read, the objcopy: CMakeFiles/svg2png.dir/stLo7j8O: Failed to find link section for section 393 warnings come from wllvm, but it can be ignored. As I mentioned earlier, I can build and run the final binary.

Again, thanks for the help.

Best regards, Manuel

[m-carrasco]

This is the output of the remaining steps, sorry.

root@97885b851cac:/workdir/build/example# get-bc -b svg2png
Bitcode file extracted to: svg2png.bc.
root@97885b851cac:/workdir/build/example# polybuild++ --lower-bitcode -i svg2png.bc -o svg2png.instrumented --libs m pthread
['opt', '-enable-new-pm=0', '-load', '/polytracker/build/share/polytracker/pass/libPolytrackerPass.so', '-ptrack', '-ignore-list=/polytracker/build/share/polytracker/abi_lists/polytracker_abilist.txt', 'svg2png.bc', '-o', 'svg2png.bc']
IGNORING: /polytracker/build/share/polytracker/abi_lists/polytracker_abilist.txt
Instrumenting: 100% |████████████████████| 6610/6610 [00:00, 188.9 functions/s] 
objcopy: stTf5iPc: Failed to find link section for section 16098
objcopy: stTf5iPc: Failed to find link section for section 16098

POLYPATH=/workdir/example.svg ./svg2png.instrumented /workdir/example.svg &> output.log this log is in the previous comments.

[surovic]

Hey @m-carrasco !

I've done some digging around and it seems that most of the warnings you see are due to exceptions in the C++ standard library. Compiling lunasvg with -DCMAKE_CXX_FLAGS="-O0 -fno-exceptions" got rid of a lot of them. You can keep -O0 for C flags as well, but I haven't found it to do anything.

What I'm left with are warnings when execution is supposed to return from gray_record_cell. I can't really guarantee that the database is sound, but I imagine you could do work with it if you avoid gray_record_cell.

Let me know if this helps or if you run into any other issues.

Best regards, Marek

[m-carrasco]

Hi @surovic

Thanks a lot for your help!

As you mentioned, I can drastically reduce the warnings by disabling exceptions for C++. I can confirm as well the remaining warnings regarding C++ and C functions. I managed to "fix" most of them by tagging them with __attribute__((always_inline)). Not sure if this is a safe thing to do. Two remaining ones still report a warning; possibly, the compiler is not inlining them.

Warning: Could not resolve the function entry associated with the return from function `_ZNK7lunasvg11LayoutShape6renderERNS_11RenderStateE` index 2717 to `_ZNK7lunasvg12LayoutSymbol6renderERNS_11RenderStateE` index 2669 (expected to be returning from function `gray_record_cell` index 4644). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.
Warning: Could not resolve the function entry associated with the return from function `_ZNK7lunasvg12LayoutSymbol6renderERNS_11RenderStateE` index 2669 to `main` index 121 (expected to be returning from function `gray_record_cell` index 4644). This is likely due to either an instrumentation error or non-standard control-flow in the instrumented program.

Regarding your previous comment about optimisations, I noticed that polybuild++ --lower-bitcode -i svg2png.bc -o svg2png.instrumented --libs m pthread invokes opt -O3 svg2png.bc -o svg2png.bc. Could this be another source of problems?

Now that you helped me fix most of the warnings, I will start working on the analysis. Thanks again!

[m-carrasco]

Hi @surovic,

I just wanted to say thanks. I managed to use Polytracker as I needed. I am sharing what I've done in case it is useful to anyone else and to know if you have any feedback on how I used Polytracker.

To reduce the number of warnings (Warning: Could not resolve the function...), I patched the program (commit 585d61eef24510bc0b7fe3d9e768d0675d4b5a6f) in the following way. The patch inlines some problematic functions.

diff --git a/3rdparty/plutovg/plutovg-ft-raster.c b/3rdparty/plutovg/plutovg-ft-raster.c
index 5642c5e..83acc74 100644
--- a/3rdparty/plutovg/plutovg-ft-raster.c
+++ b/3rdparty/plutovg/plutovg-ft-raster.c
@@ -1410,7 +1410,7 @@ PVG_FT_END_STMNT
   }

-  static int
+  static int __attribute__((always_inline))
   gray_convert_glyph( RAS_ARG )
   {
     TBand            bands[40];
@@ -1556,7 +1556,7 @@ PVG_FT_END_STMNT
   }

-  static int
+  static int __attribute__((always_inline))
   gray_raster_render( RAS_ARG_ void* buffer, long buffer_size,
                       const PVG_FT_Raster_Params*  params )
   {
@@ -1608,7 +1608,7 @@ PVG_FT_END_STMNT
     return gray_convert_glyph( RAS_VAR );
   }

-  void
+  void __attribute__((always_inline))
   PVG_FT_Raster_Render(const PVG_FT_Raster_Params *params)
   {
       char stack[PVG_FT_MINIMUM_POOL_SIZE];
diff --git a/3rdparty/plutovg/plutovg-rle.c b/3rdparty/plutovg/plutovg-rle.c
index 857628a..23569d6 100644
--- a/3rdparty/plutovg/plutovg-rle.c
+++ b/3rdparty/plutovg/plutovg-rle.c
@@ -162,7 +162,7 @@ void plutovg_rle_destroy(plutovg_rle_t* rle)
     free(rle);
 }

-void plutovg_rle_rasterize(plutovg_t* pluto, plutovg_rle_t* rle, const plutovg_path_t* path, const plutovg_matrix_t* matrix, const plutovg_rect_t* clip, const plutovg_stroke_data_t* stroke, plutovg_fill_rule_t winding)
+void __attribute__((always_inline)) plutovg_rle_rasterize(plutovg_t* pluto, plutovg_rle_t* rle, const plutovg_path_t* path, const plutovg_matrix_t* matrix, const plutovg_rect_t* clip, const plutovg_stroke_data_t* stroke, plutovg_fill_rule_t winding)
 {
     PVG_FT_Raster_Params params;
     params.flags = PVG_FT_RASTER_FLAG_DIRECT | PVG_FT_RASTER_FLAG_AA;
diff --git a/3rdparty/plutovg/plutovg.c b/3rdparty/plutovg/plutovg.c
index 3b357bc..ed815de 100644
--- a/3rdparty/plutovg/plutovg.c
+++ b/3rdparty/plutovg/plutovg.c
@@ -418,7 +418,7 @@ plutovg_path_t* plutovg_get_path(const plutovg_t* pluto)
     return pluto->path;
 }

-void plutovg_fill(plutovg_t* pluto)
+void __attribute__((always_inline)) plutovg_fill(plutovg_t* pluto)
 {
     plutovg_fill_preserve(pluto);
     plutovg_new_path(pluto);
@@ -454,7 +454,7 @@ void plutovg_paint(plutovg_t* pluto)
     plutovg_blend(pluto, rle);
 }

-void plutovg_fill_preserve(plutovg_t* pluto)
+void __attribute__((always_inline)) plutovg_fill_preserve(plutovg_t* pluto)
 {
     plutovg_state_t* state = pluto->state;
     plutovg_rle_clear(pluto->rle);
diff --git a/source/canvas.cpp b/source/canvas.cpp
index 80e05a3..c30da89 100644
--- a/source/canvas.cpp
+++ b/source/canvas.cpp
@@ -88,7 +88,7 @@ void Canvas::setTexture(const Canvas* source, TextureType type, const Transform&
     plutovg_texture_set_matrix(texture, &matrix);
 }

-void Canvas::fill(const Path& path, const Transform& transform, WindRule winding, BlendMode mode, double opacity)
+void __attribute__((always_inline)) Canvas::fill(const Path& path, const Transform& transform, WindRule winding, BlendMode mode, double opacity)
 {
     auto matrix = to_plutovg_matrix(transform);
     plutovg_matrix_multiply(&matrix, &matrix, &translation);
diff --git a/source/layoutcontext.cpp b/source/layoutcontext.cpp
index 24f4d0f..3800471 100644
--- a/source/layoutcontext.cpp
+++ b/source/layoutcontext.cpp
@@ -82,7 +82,7 @@ LayoutObject* LayoutContainer::addChildIfNotEmpty(std::unique_ptr<LayoutContaine
     return addChild(std::move(child));
 }

-void LayoutContainer::renderChildren(RenderState& state) const
+void __attribute__((always_inline)) LayoutContainer::renderChildren(RenderState& state) const
 {
     for(const auto& child : children)
         child->render(state);
@@ -150,7 +150,7 @@ LayoutSymbol::LayoutSymbol()
 {
 }

-void LayoutSymbol::render(RenderState& state) const
+void __attribute__((always_inline)) LayoutSymbol::render(RenderState& state) const
 {
     BlendInfo info{clipper, masker, opacity, clip};
     RenderState newState(this, state.mode());
@@ -318,7 +318,7 @@ void LayoutSolidColor::apply(RenderState& state) const
     state.canvas->setColor(color);
 }

-void FillData::fill(RenderState& state, const Path& path) const
+void __attribute__((always_inline)) FillData::fill(RenderState& state, const Path& path) const
 {
     if(opacity == 0.0 || (painter == nullptr && color.isNone()))
         return;
@@ -393,7 +393,7 @@ LayoutShape::LayoutShape()
 {
 }

-void LayoutShape::render(RenderState& state) const
+void __attribute__((always_inline)) LayoutShape::render(RenderState& state) const
 {
     if(visibility == Visibility::Hidden)
         return;
diff --git a/source/lunasvg.cpp b/source/lunasvg.cpp
index 4da1304..ee24875 100644
--- a/source/lunasvg.cpp
+++ b/source/lunasvg.cpp
@@ -271,7 +271,7 @@ Matrix Matrix::translated(double tx, double ty)
     return Transform::translated(tx, ty);
 }

-std::unique_ptr<Document> Document::loadFromFile(const std::string& filename)
+std::unique_ptr<Document> __attribute__((always_inline)) Document::loadFromFile(const std::string& filename)
 {
     std::ifstream fs;
     fs.open(filename);
@@ -335,7 +335,7 @@ double Document::height() const
     return root->height;
 }

-void Document::render(Bitmap bitmap, const Matrix& matrix) const
+void __attribute__((always_inline)) Document::render(Bitmap bitmap, const Matrix& matrix) const
 {
     RenderState state(nullptr, RenderMode::Display);
     state.canvas = Canvas::create(bitmap.data(), bitmap.width(), bitmap.height(), bitmap.stride());
@@ -343,7 +343,7 @@ void Document::render(Bitmap bitmap, const Matrix& matrix) const
     root->render(state);
 }

-Bitmap Document::renderToBitmap(std::uint32_t width, std::uint32_t height, std::uint32_t backgroundColor) const
+Bitmap __attribute__((always_inline)) Document::renderToBitmap(std::uint32_t width, std::uint32_t height, std::uint32_t backgroundColor) const
 {
     if(root->width == 0.0 || root->height == 0.0)
         return Bitmap{};

My main goal was to know which byte offsets from the input are used after a certain point in the program. I call this point in the program a breakpoint and represent it by introducing a dummy function call with the same name. Conceptually, my script does the following:

1. Find the E event id associated with the breakpoint() call.
2. Find all labels in accessed_label rows associated with a subsequent event id to E.
3. Last, use the Taint Forest to navigate the label expressions and reach the byte offsets.

While working on this, I found that the API was slow for certain operations. In particular, when I wanted to get the event id associated with the breakpoint (Step 1). That's why I performed SQL queries directly on the database when possible. I understand this is very hacky.

Regarding the instrumentation, using a 600-byte-SVG file, the SUT (svg2png example in lunasvg) took approximately 25 minutes to finish. Without instrumentation, the execution time is almost instant. I recall that I was not running the SUT on a shared folder inside my container. However, I should double-check this to be fully certain.

Thanks for all your help!

Best regards, Manuel.