Closed clearday4 closed 3 years ago
Hi @clearday4
What's the step to reproduce this issue ? I think: In a multi-thread program, one thread keeps calling CoreV1API_listNamespacedPod (how often ?), what does the other thread do ?
2 thread call CoreV1API_listNamespacedPod() with fieldselector "status.phase=Running" simultaneously at 10ms intervals. If I add the pthread_mutex_lock back and forth of CoreV1API_listNamespacedPod()+v1_pod_list_free(), the problem is solved.
I think you need to change cJSON_Parse(event_string) =========> const char *parse_end = NULL; cJSON_ParseWithOpts(event_string, &parse_end, 0);
and print out parse_end instead of cJSON_GetErrorPtr()
I think you need to change cJSON_Parse(event_string) =========> const char *parse_end = NULL; cJSON_ParseWithOpts(event_string, &parse_end, 0);
and print out parse_end instead of cJSON_GetErrorPtr()
Where is the code ? In the example ?
@ityuhui It's in a bunch of the generated client library code, eg.
I think we need to change the code generator.
Hi @brendandburns
Described in cJSON official readme,
However it is thread safe under the following conditions:
- cJSON_GetErrorPtr is never used (the return_parse_end parameter of cJSON_ParseWithOpts can be used instead)
We don't need replace cJSON_Parse
in generated code because the generated code does not call cJSON_GetErrorPtr
But the examples and config/authentication plugin code use the function cJSON_GetErrorPtr
, they are not thread safe. e.g.
https://github.com/kubernetes-client/c/blob/master/examples/multi_thread/watch_pod.c#L17 https://github.com/kubernetes-client/c/blob/master/kubernetes/config/authn_plugin/authn_plugin_util.c#L55
I will fix them.
But the core dump described by this issue is not releated to the C client library itself.
Users of the C client library, if they use cJSON directly in their code, should follow the requirements in cJSON official readme https://github.com/DaveGamble/cJSON#thread-safety to ensure the thread safety in multi-threaded program.
@ityuhui thanks for the explanation (and sorry I didn't read into the details of our cJSON usage).
I merged #58 so closing this issue.
Thank you @brendandburns
And thank you @clearday4 for the finding and suggestion.
There is a problem that double free crashes while calling CoreV1API_listNamespacedPod() continuously in the Multi-thread. The direct cause is that the duplicated detection of v1_pod_free()'s address, it seems that cJSON is basically thread unsafe.
https://github.com/DaveGamble/cJSON/issues/221 https://github.com/DaveGamble/cJSON/issues/314
Looking at the above link, If you don't specify return_parse_end in cJSON_ParseWithOpts or use cJSON_GetErrorPtr (), it cause memory violation in the multi-thread.
And we need an additional review of why v1_pod_free() accesses duplicate addresses per thread. We got the following logs using address sanitizer.
================================================================= ==1659378== ERROR: AddressSanitizer: attempting double-free on 0x600800040390:
0 0x7ffff4e60dd9 (/usr/lib/libasan.so+0x15dd9)
0x600800040390 is located 0 bytes inside of 40-byte region [0x600800040390,0x6008000403b8) freed by thread T146 here:
0 0x7ffff4e60dd9 (/usr/lib/libasan.so+0x15dd9)
previously allocated by thread T147 here:
0 0x7ffff4e60ef9 (/usr/lib/libasan.so+0x15ef9)
Thread T147 created by T0 here:
0 0x7ffff4e55a0a (/usr/lib/libasan.so+0xaa0a)
Thread T146 created by T0 here:
0 0x7ffff4e55a0a (/usr/lib/libasan.so+0xaa0a)
==1659378== ABORTING
==2107129== ERROR: AddressSanitizer: attempting double-free on 0x6004003171b0: [0319|10:26:06.251|NRDHCPAD|DHCPAD|0180|INFO|8900][ERR]dal_db_connect fail(-26, Unknown error) [2021-03-19 10:26:06.251][dhcpad_dal.c:180][ERR]dal_db_connect fail(-26, Unknown error)
0 0x7ffff4e60dd9 (/usr/lib64/libasan.so.0+0x15dd9)
0x6004003171b0 is located 0 bytes inside of 4-byte region [0x6004003171b0,0x6004003171b4) freed by thread T5 here:
0 0x7ffff4e60dd9 (/usr/lib64/libasan.so.0+0x15dd9)
previously allocated by thread T5 here:
0 0x7ffff4e60ef9 (/usr/lib64/libasan.so.0+0x15ef9)
Thread T6 created by T0 here:
0 0x7ffff4e55a0a (/usr/lib64/libasan.so.0+0xaa0a)
Thread T5 created by T0 here:
0 0x7ffff4e55a0a (/usr/lib64/libasan.so.0+0xaa0a)
==2107129== ABORTING
We would like to ask you to analyze this issue. Thanks in advance.