child thread json::parse crash

[thiefuniverse]

Hello, I use Json::parse in the child thread and I find that my program will crash at Json::parse in a child thread.

// test_json_main.cpp
//  g++ test_json_main.cpp json11.cpp  -o test_json_main -std=c++11 -lpthread -g

#include <iostream>
#include <thread>
#include "json11.hpp"
#include <unistd.h>

using namespace json11;
using namespace std;

void threadFunc() {
    int counter=0;
    while(counter < 10){
        sleep(1);
        std::cout<< "start parse json: "<<endl;
        std::string err("");
        std::string out("{\"request_id\":\"1512033563.8857\",\"client_track_id\":\"6d368a59d4680e4679359f068a3e7698\",\"flow_control\":{\"sample_rate\":100}}");
        auto json = json11::Json::parse(out, err);
        std::cout<<"get here"<<std::endl;
        counter ++;
    }
}
void sleepExit() {
     std::cout<< "sleep exit start"<<endl;
    //  I want to make main thread wait at here for some time. But I find that if when main thread is sleeping. And child thread which is parsing  will crash.
      sleep(3);
     std::cout<< "sleep exit end"<<endl;
}
int main() {
     atexit(sleepExit);

     std::cout<< "main start"<<endl;
     thread testJson(threadFunc);
     testJson.detach();
     sleep(2);
     std::cout<< "main end"<<endl;
     return 0;
}

And This is my backtrace using gdb with a core dump file.

(gdb) bt
#0  0x00007f6a2c000b00 in ?? ()
#1  0x0000000000402554 in std::_Tuple_impl<0ul, void (*)()>::_M_head (__t=...) at /usr/include/c++/4.9/tuple:241
#2  0x00000000004023e1 in std::thread::thread<void (&)()> (this=0x401e9a <threadFunc()+123>, __f=@0x7f6a33266b90: {void (void)} 0x7f6a33266b90) at /usr/include/c++/4.9/thread:139
#3  0x0000000000402384 in std::__shared_ptr<std::thread::_Impl<std::_Bind_simple<void (*())()> >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (this=0x7f6a33266c10, __in_chrg=<optimized out>)
    at /usr/include/c++/4.9/bits/shared_ptr_base.h:914
#4  0x000000000040a98a in std::_Vector_base<json11::Json, std::allocator<json11::Json> >::_Vector_impl::_M_swap_data (this=0x2f3e9eb85ef0e600, __x=...) at /usr/include/c++/4.9/bits/stl_vector.h:103
#5  0x0000000000409732 in std::_Vector_base<json11::Json, std::allocator<json11::Json> >::_Vector_base(std::_Vector_base<json11::Json, std::allocator<json11::Json> >&&) (this=0x7f6a2c000ba8,
    __x=<unknown type in /home/xiefei/works/prac/testJson11withThread/test_json_main, CU 0xd133, DIE 0x265e0>) at /usr/include/c++/4.9/bits/stl_vector.h:144
#6  0x000000000040833f in __gnu_cxx::operator!=<json11::Json const*, std::vector<json11::Json, std::allocator<json11::Json> > > (
    __lhs=<error reading variable: Cannot access memory at address 0xe8458b48e0758948>, __rhs=...) at /usr/include/c++/4.9/bits/stl_iterator.h:829
#7  0x0000000000406a17 in json11::(anonymous namespace)::JsonParser::parse_json (this=0x7f6a33266dd0, depth=0) at json11.cpp:713
#8  0x0000000000407000 in json11::Json::parse_multi (in=<error reading variable: Cannot access memory at address 0x78>, parser_stop_pos=@0x7f6a33bdee78: 10431281232322004296,
    err="{\"request_id\":\"1512033563.8857\",\"client_track_id\":\"6d368a59d4680e4679359f068a3e7698\",\"flow_control\":{\"sample_rate\":100}}", strategy=(unknown: 32618)) at json11.cpp:757
#9  0x0000000000401f60 in threadFunc () at test_json_main.cpp:21
#10 0x0000000000403627 in __gnu_cxx::__atomic_add_single (__mem=0x1420058, __val=0) at /usr/include/c++/4.9/ext/atomicity.h:74
#11 0x0000000000403562 in std::__get_helper<0ul, void (*)()> (__t=...) at /usr/include/c++/4.9/tuple:746
#12 0x00000000004034d0 in std::_Bind_simple<void (*())()>::operator()() (this=0x4034d0 <std::_Bind_simple<void (*())()>::operator()()+30>) at /usr/include/c++/4.9/functional:1688
#13 0x00007f6a33c09c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007f6a33eda6ba in start_thread (arg=0x7f6a33267700) at pthread_create.c:333
#15 0x00007f6a336783dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

(gdb) bt full
#0  0x00007f6a2c000b00 in ?? ()
No symbol table info available.
#1  0x0000000000402554 in std::_Tuple_impl<0ul, void (*)()>::_M_head (__t=...) at /usr/include/c++/4.9/tuple:241
No locals.
#2  0x00000000004023e1 in std::thread::thread<void (&)()> (this=0x401e9a <threadFunc()+123>, __f=@0x7f6a33266b90: {void (void)} 0x7f6a33266b90) at /usr/include/c++/4.9/thread:139
No locals.
#3  0x0000000000402384 in std::__shared_ptr<std::thread::_Impl<std::_Bind_simple<void (*())()> >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (this=0x7f6a33266c10, __in_chrg=<optimized out>)
    at /usr/include/c++/4.9/bits/shared_ptr_base.h:914
No locals.
#4  0x000000000040a98a in std::_Vector_base<json11::Json, std::allocator<json11::Json> >::_Vector_impl::_M_swap_data (this=0x2f3e9eb85ef0e600, __x=...) at /usr/include/c++/4.9/bits/stl_vector.h:103
No locals.
#5  0x0000000000409732 in std::_Vector_base<json11::Json, std::allocator<json11::Json> >::_Vector_base(std::_Vector_base<json11::Json, std::allocator<json11::Json> >&&) (this=0x7f6a2c000ba8,
    __x=<unknown type in /home/xiefei/works/prac/testJson11withThread/test_json_main, CU 0xd133, DIE 0x265e0>) at /usr/include/c++/4.9/bits/stl_vector.h:144
No locals.
#6  0x000000000040833f in __gnu_cxx::operator!=<json11::Json const*, std::vector<json11::Json, std::allocator<json11::Json> > > (
    __lhs=<error reading variable: Cannot access memory at address 0xe8458b48e0758948>, __rhs=...) at /usr/include/c++/4.9/bits/stl_iterator.h:829
No locals.
#7  0x0000000000406a17 in json11::(anonymous namespace)::JsonParser::parse_json (this=0x7f6a33266dd0, depth=0) at json11.cpp:713
        data = std::vector of length 33285996544, capacity 8755873841336 = {<error reading variable data (Cannot access memory at address 0x0)>
        ch = 58 ':'
#8  0x0000000000407000 in json11::Json::parse_multi (in=<error reading variable: Cannot access memory at address 0x78>, parser_stop_pos=@0x7f6a33bdee78: 10431281232322004296,
    err="{\"request_id\":\"1512033563.8857\",\"client_track_id\":\"6d368a59d4680e4679359f068a3e7698\",\"flow_control\":{\"sample_rate\":100}}", strategy=(unknown: 32618)) at json11.cpp:757
        parser = {str = "{\"request_id\":\"1512033563.8857\",\"client_track_id\":\"6d368a59d4680e4679359f068a3e7698\",\"flow_control\":{\"sample_rate\":100}}", i = 31, err = "", failed = false,
          strategy = json11::STANDARD}
        json_vec = std::vector of length 276919149293893483, capacity -20420258540362602 = {<error reading variable json_vec (Cannot access memory at address 0x1040c708588948)>
#9  0x0000000000401f60 in threadFunc () at test_json_main.cpp:21
        err = ""
        out = "{\"request_id\":\"1512033563.8857\",\"client_track_id\":\"6d368a59d4680e4679359f068a3e7698\",\"flow_control\":{\"sample_rate\":100}}"
        json = {m_ptr = std::shared_ptr (expired, weak 0) 0x7f6a2c000d10}
        counter = 0
#10 0x0000000000403627 in __gnu_cxx::__atomic_add_single (__mem=0x1420058, __val=0) at /usr/include/c++/4.9/ext/atomicity.h:74
No locals.
#11 0x0000000000403562 in std::__get_helper<0ul, void (*)()> (__t=...) at /usr/include/c++/4.9/tuple:746
No locals.
#12 0x00000000004034d0 in std::_Bind_simple<void (*())()>::operator()() (this=0x4034d0 <std::_Bind_simple<void (*())()>::operator()()+30>) at /usr/include/c++/4.9/functional:1688
No locals.
#13 0x00007f6a33c09c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#14 0x00007f6a33eda6ba in start_thread (arg=0x7f6a33267700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f6a33267700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140094101419776, -4939436259746414781, 0, 140737036454079, 140094101420480, 0, 5019260309473427267, 5019260956175990595}, mask_was_saved = 0}},
          priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#15 0x00007f6a336783dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Maybe it's a destruction problem of shared_ptr or my usage problem ? Thanks for your help! :)

[artwyman]

Looks like your test case is going out of its way to run JSON parsing after main has returned. Is that something you need for a real-world use case? Why not join the thread before the process exits?

In general, a lot of code can become unsafe if it's run before main starts, or after main exits (or after a call to exit() or similar). My guess is that this crash is occurring because static variables are destroyed after main exits, and thus can't be safely used anymore. Json11 does depend on some static state, constructed here: https://github.com/dropbox/json11/blob/master/json11.cpp#L238

[thiefuniverse]

Thanks firstly, I think probably it's that problem. I develop a SDK and when user load my lib a thread will parse json all the time. I detach this thread when its created. So sometimes if user's main thread exits my child thread will run to Json::parse function and crash. Do you have some suggestions? I can't control the time when user exits so I can't use join, I think.

[artwyman]

Two possible answers for you:

1) If your SDK is making use of background threads, and you want to support clean termination, then I'd suggest providing your SDK user with an explicit shutdown() function to call to perform cleanup by terminating threads, freeing resources, etc. I'd consider including such a function in any library which managed background threads, regardless of this parsing situation. There's always the potential for unexpected results if a thread isn't terminated cleanly before exit, and an explicit shutdown would also give your user the option to stop their threads if they're unneded for a while. Note that if the process terminates uncleanly (e.g. with a crash or a call to terminate()) your shutdown might not get called, but if the process is already crashing anyway that's likely not a big deal.

2) Building a library to be safe to use during exit puts some additional requirements on it which json11 wasn't designed to meet. Json11 was initially built for use on mobile, where processes mostly don't ever exit (they just get suspended or killed), so this wasn't ever a priority. If you decide you need those requirements you might need to try a different library which has no global data. You could also take a stab at forking and removing the global data requirements from json11, since I believe they're mostly performance/memory optimizations.

[thiefuniverse]

Thanks very much for your suggestions. I tried to use nlohmann/json for my example and it exited normally and I will test it at my SDK (I have no confidence for modifying this lib). I also will consider to add a interface for releasing resource. Really thanks! :smile: