search

ldn-softdev / jtc

JSON processing utility
MIT License
501 stars 33 forks source link

<ab.json ./jtc -w'<phone>l:' -w'<phone>l:[-1:]' -s / -w'<phone>l:' -l leaks memory #28

Closed D4N closed 4 years ago

D4N commented 4 years ago

The following example from User Guide.md results in a memory leak:

bash $ <ab.json jtc -w'<phone>l:' -w'<phone>l:[-1:]' -s / -w'<phone>l:' -l
"phone": {
   "number": "113-123-2368",
   "type": "mobile"
}
"phone": {
   "number": "223-283-0372",
   "type": "mobile"
}
"phone": {
   "number": "333-638-0238",
   "type": "home"
}

=================================================================
==240775==ERROR: LeakSanitizer: detected memory leaks

Indirect leak of 960 byte(s) in 6 object(s) allocated from:
    #0 0x63c677 in operator new(unsigned long) (/home/dan/projects/github.com/ldn-softdev/jtc/jtc+0x63c677)
    #1 0x855f8c in __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/ext/new_allocator.h:115:27
    #2 0x855ef0 in std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > > >::allocate(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >&, unsigned long) /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/alloc_traits.h:460:20
    #3 0x855e99 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >::_M_get_node() /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/stl_tree.h:584:16
    #4 0x94aa4f in std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >::_M_create_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Jnode>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, Jnode&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/stl_tree.h:634:23
    #5 0x94a407 in std::pair<std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >, bool> std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >::_M_emplace_unique<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Jnode>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, Jnode&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/stl_tree.h:2414:19
    #6 0x7a9df1 in std::pair<std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >, bool> std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Jnode, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Jnode>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, Jnode&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/stl_map.h:577:16
    #7 0x675912 in Json::parse_array_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3074:20
    #8 0x670232 in Json::parse_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3015:29
    #9 0x6734ff in Json::parse_object_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3112:3
    #10 0x670145 in Json::parse_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3014:30
    #11 0x6749ac in Json::parse_array_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3058:3
    #12 0x670232 in Json::parse_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3015:29
    #13 0x6734ff in Json::parse_object_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3112:3
    #14 0x670145 in Json::parse_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3014:30
    #15 0x66d604 in Json::parse(Streamstr::const_iterator&, Json::ParseTrailing) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:2961:2
    #16 0x70798e in Jtc::parsejson(Streamstr::const_iterator&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/jtc.cpp:1392:12
    #17 0x704d88 in run_single_optset(CommonResource&, Streamstr::const_iterator&, Json&, Json&) /home/dan/projects/github.com/ldn-softdev/jtc/jtc.cpp:905:11
    #18 0x7007c9 in run_decomposed_optsets(CommonResource&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/jtc.cpp:888:3
    #19 0x6f595f in main /home/dan/projects/github.com/ldn-softdev/jtc/jtc.cpp:814:6
    #20 0x7ff4d9941041 in __libc_start_main (/lib64/libc.so.6+0x27041)

Indirect leak of 960 byte(s) in 6 object(s) allocated from:
    #0 0x63c677 in operator new(unsigned long) (/home/dan/projects/github.com/ldn-softdev/jtc/jtc+0x63c677)
    #1 0x855f8c in __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/ext/new_allocator.h:115:27
    #2 0x855ef0 in std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > > >::allocate(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >&, unsigned long) /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/alloc_traits.h:460:20
    #3 0x855e99 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >::_M_get_node() /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/stl_tree.h:584:16
    #4 0xa175ef in std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >::_M_create_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&&, Jnode&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/stl_tree.h:634:23
    #5 0xa16f37 in std::pair<std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >, bool> std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >::_M_emplace_unique<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&&, Jnode&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/stl_tree.h:2414:19
    #6 0x7b1121 in std::pair<std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> >, bool> std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Jnode, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode> > >::emplace<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Jnode>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&&, Jnode&&) /usr/bin/../lib/gcc/x86_64-redhat-linux/10/../../../../include/c++/10/bits/stl_map.h:577:16
    #7 0x6742b2 in Json::parse_object_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3120:21
    #8 0x670145 in Json::parse_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3014:30
    #9 0x6749ac in Json::parse_array_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3058:3
    #10 0x670232 in Json::parse_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3015:29
    #11 0x6734ff in Json::parse_object_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3112:3
    #12 0x670145 in Json::parse_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3014:30
    #13 0x6749ac in Json::parse_array_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3058:3
    #14 0x670232 in Json::parse_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3015:29
    #15 0x6734ff in Json::parse_object_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3112:3
    #16 0x670145 in Json::parse_(Jnode&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:3014:30
    #17 0x66d604 in Json::parse(Streamstr::const_iterator&, Json::ParseTrailing) /home/dan/projects/github.com/ldn-softdev/jtc/./lib/Json.hpp:2961:2
    #18 0x70798e in Jtc::parsejson(Streamstr::const_iterator&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/jtc.cpp:1392:12
    #19 0x704d88 in run_single_optset(CommonResource&, Streamstr::const_iterator&, Json&, Json&) /home/dan/projects/github.com/ldn-softdev/jtc/jtc.cpp:905:11
    #20 0x7007c9 in run_decomposed_optsets(CommonResource&, Streamstr::const_iterator&) /home/dan/projects/github.com/ldn-softdev/jtc/jtc.cpp:888:3
    #21 0x6f595f in main /home/dan/projects/github.com/ldn-softdev/jtc/jtc.cpp:814:6
    #22 0x7ff4d9941041 in __libc_start_main (/lib64/libc.so.6+0x27041)

SUMMARY: AddressSanitizer: 1920 byte(s) leaked in 12 allocation(s).
ldn-softdev commented 4 years ago

Hi D4N, thanks for reporting this!

it's a GNU implementation of std::map.emplace() is having this leak (when emplace-constructing a value from r-value reference). MacOS (and presumably Bsd) STL implementation does not have this leak/bug, so it's a bug in the GNU STL implementation only (which I find not the first time).

I'll try working around this issue by dodging std::map.emplace() call and will do it via entry creating and then moving.

ldn-softdev commented 4 years ago

Okay, I've taken a closer look - good thing it's not an implementation error of emplace{} call as I thought before.

After a deeper analysis it turned out to be a harmless side-effect of the swap (-s) operation. Leak indeed was produced but it was harmless: the nesting parts of a JSON tree after swapping with the nested JSON part were meant to be discarded anyways, but instead it was left "lingering in the air".

I'll push a fix soon, together with the fix for the other issue. Thank you again, D4N for catching and reporting those.