Dynamic loading of libcef.so on Linux ARM64 not possible anymore due to TLS size increase

Hethsron commented 6 days ago

Describe the bug Dynamically loading libcef.so into a running process on Linux arm64 fails since CEF 129 with the error message "cannot allocate memory in static TLS block". Since a lot of CEF wrappers depend on the ability to dynamically extract and load libcef.so (known: CEF4Delphi, JCEF, probably more), this issue effectively makes CEF 129+ unusable for these wrappers on Linux arm64.

To Reproduce Either use CEF 129 with for example JCEF on Linux, or use the following synthetic console application to dlopen libcef.so:

#include <stdio.h>
#include <dlfcn.h>

int main (void)
{
  void* g_libcef_handle = dlopen("./libcef.so", RTLD_LAZY | RTLD_LOCAL);
  if (!g_libcef_handle) {
    fprintf(stderr, "dlerror %s\n", dlerror());
  }
  return 0;
}

Compile the source code :

g++ -o main.cpp -ldl

Run :

bash-4.4$ ./a.out 
dlerror ./libcef.so: cannot allocate memory in static TLS block
bash-4.4$

Expected behavior libcef.so can be dynamically loaded into a Linux process.

Screenshots If applicable, add screenshots to help explain your problem.

Versions (please complete the following information):

OS: Linux (all ARM popular distributions)
CEF Version: 129.x (6668)

Background/Analysis The TLS segment size of Chromium 129 increases by over 700 bytes, as can be seen here:

bash-4.4$ readelf -Wl libcef.so | grep -E 'PhysAddr|TLS'
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  TLS            0xc7cc340 0x000000000c7ec340 0x000000000c7ec340 0x000090 0x000430 R   0x40
bash-4.4$

This issue is similar to #3616.

Hethsron commented 6 days ago

@magreenblatt Hi Marshall, do you have a fix for that ?

magreenblatt commented 6 days ago

Does this also impact x64 Intel builds? You can use the tips from #3616 to debug the issue.

Hethsron commented 6 days ago

Does this also impact x64 Intel builds? You can use the tips from #3616 to debug the issue.

Intel x64 versions are not impacted. I will try to debug the issue with these tips and let you as soon as possible a further analysis.

Hethsron commented 5 days ago

@magreenblatt

After following tips, here is my analysis :

In case of libcef.so (M~129), the output shows that the TLS segment has a file size (FileSiz) of 0x000090 and a memory size (MemSiz) of 0x000430. Thus, the segment size in hex is 0x000430 and equal to 1072 bytes.

Earlier versions only had a TLS segment size of 0x0003d8 or 984 bytes. For example, here is the result of the command on libcef.so version M~125 :

readelf -Wl libcef.so | grep -E 'PhysAddr|TLS'
Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
TLS            0xc214fc0 0x000000000c234fc0 0x000000000c234fc0 0x000090 0x0003d8 R   0x40

Here are the two TLS segment results compared side by side:

Property	First Result	Second Result	Difference
Offset	0xc7cc340	0xc214fc0	-
VirtAddr	0x000000000c7ec340	0x000000000c234fc0	-
PhysAddr	0x000000000c7ec340	0x000000000c234fc0	-
FileSiz	0x000090 (144 bytes)	0x000090 (144 bytes)	No difference
MemSiz	0x000430 (1072 bytes)	0x0003d8 (984 bytes)	88 bytes (1072 - 984)
Flags	R	R	No difference
Align	0x40	0x40	No difference

Key Differences:

The memory size (MemSiz) is different:
- First result: 1072 bytes (0x000430)
- Second result: 984 bytes (0x0003d8) The difference in size is 88 bytes. Other fields such as Offset, VirtAddr, PhysAddr, FileSiz, and flags remain the same, except for different starting addresses.

Version used for test

cef_binary_129.0.12+gf09539f+chromium-129.0.6668.101_linuxarm64_client

Workaround Just for the record, this workaround works. But it's not acceptable in our case:

export LD_PRELOAD=<FULL-PATH-TO-libcef.so>

And the following patch is already present in config.h:

diff --git third_party/libxml/linux/config.h third_party/libxml/linux/config.h
index c064071ce1545..65110af9a78f5 100644
--- third_party/libxml/linux/config.h
+++ third_party/libxml/linux/config.h
@@ -171,7 +171,7 @@
/* #undef XML_SOCKLEN_T */

/* TLS specifier */
-#define XML_THREAD_LOCAL _Thread_local
+/* #undef XML_THREAD_LOCAL */

/* Define for Solaris 2.5.1 so the uint32_t typedef from <sys/synch.h>,
<pthread.h>, or <semaphore.h> is not used. If the typedef were allowed, the

Deep Investigation Further investigation found the following usage (.tdata and .tbss are the symbol sections of interest when looking for the actual users of all that TLS space - this article here (A dynamic linker murder mystery) was very helpful in learning interesting tidbits about TLS, and commands to investigate the situation) :

bash-4.4$ objdump -t libcef.so | grep -F '.tdata'
0000000000000000 l       .tdata 0000000000000010              .hidden _ZN4absl13cord_internal17cordz_next_sampleE
0000000000000010 l       .tdata 0000000000000004              _ZN2v88internal12_GLOBAL__N_130current_per_thread_assert_dataE
0000000000000014 l       .tdata 0000000000000004              _ZN4base8internal12_GLOBAL__N_122current_sequence_tokenE
0000000000000018 l       .tdata 0000000000000004              _ZN4base8internal12_GLOBAL__N_118current_task_tokenE
000000000000001c l       .tdata 0000000000000001              _ZN4base8internal12_GLOBAL__N_128current_task_is_thread_boundE
000000000000001d l       .tdata 0000000000000001              _ZN4base8internal12_GLOBAL__N_132task_priority_for_current_threadE
0000000000000020 l       .tdata 0000000000000004              _ZN4base12_GLOBAL__N_119current_thread_typeE
0000000000000028 l       .tdata 0000000000000008              _ZN4base12_GLOBAL__N_111thread_nameE
0000000000000030 l       .tdata 0000000000000004              _ZN4base12_GLOBAL__N_111g_thread_idE
0000000000000034 l       .tdata 0000000000000001              _ZN4base12_GLOBAL__N_116g_is_main_threadE
0000000000000038 l       .tdata 0000000000000004              _ZN15partition_alloc8internal4base12_GLOBAL__N_111g_thread_idE
000000000000003c l       .tdata 0000000000000001              _ZN15partition_alloc8internal4base12_GLOBAL__N_116g_is_main_threadE
0000000000000040 l       .tdata 0000000000000040              .hidden _ZN6google8protobuf8internal15ThreadSafeArena13thread_cache_E
0000000000000080 l       .tdata 0000000000000004              simd_support
0000000000000088 l       .tdata 0000000000000004              simd_features
0000000000000084 l       .tdata 0000000000000004              simd_huffman
000000000000008c l       .tdata 0000000000000004              .hidden _ZN5blink13next_world_idE

bash-4.4$ objdump -t libcef.so | grep -F '.tbss'
0000000000000090 l       .tbss  0000000000000008              .hidden _ZN8perfetto10DataSourceIN4base20perfetto_track_event10TrackEventENS_8internal26TrackEventDataSourceTraitsEE10tls_state_E
0000000000000098 l       .tbss  0000000000000008              _ZN12_GLOBAL__N_117g_isolate_managerE
00000000000000a4 l       .tbss  0000000000000001              _ZGVZN4absl13base_internal12GetCachedTIDEvE9thread_id
00000000000000a0 l       .tbss  0000000000000004              _ZZN4absl13base_internal12GetCachedTIDEvE9thread_id
00000000000000a8 l       .tbss  0000000000000018              _ZZN4absl13cord_internal25cordz_should_profile_slowERNS0_13SamplingStateEE28exponential_biased_generator
00000000000000c0 l       .tbss  0000000000000001              _ZZN4absl12log_internal12_GLOBAL__N_121ThreadIsLoggingStatusEvE17thread_is_logging
00000000000000c1 l       .tbss  0000000000000001              _ZN4ipcz12_GLOBAL__N_125is_thread_within_api_callE
00000000000000c8 l       .tbss  0000000000000008              .hidden _ZN8perfetto10DataSourceIN6webrtc20perfetto_track_event10TrackEventENS_8internal26TrackEventDataSourceTraitsEE10tls_state_E
00000000000000d8 l       .tbss  0000000000000001              _ZGVZN13SkStrikeCache17GlobalStrikeCacheEvE5cache
00000000000000d0 l       .tbss  0000000000000008              _ZZN13SkStrikeCache17GlobalStrikeCacheEvE5cache
00000000000000e0 l       .tbss  0000000000000008              _ZN4dawn6native12_GLOBAL__N_18tlDeviceE
00000000000000e8 l       .tbss  0000000000000008              .hidden _ZN8perfetto10DataSourceIN2v820perfetto_track_event10TrackEventENS_8internal26TrackEventDataSourceTraitsEE10tls_state_E
00000000000000f0 l       .tbss  0000000000000001              _ZN2v88internal12_GLOBAL__N_119tls_singleton_takenE
00000000000000f8 l       .tbss  0000000000000018              _ZN2v88internal12_GLOBAL__N_121tls_singleton_storageE
0000000000000118 l       .tbss  0000000000000008              .hidden _ZN2v88internal18g_current_isolate_E
0000000000000110 l       .tbss  0000000000000008              .hidden _ZN2v88internal34g_current_per_isolate_thread_data_E
0000000000000120 l       .tbss  0000000000000004              _ZN2v88internal12_GLOBAL__N_19thread_idE
0000000000000128 l       .tbss  0000000000000008              _ZN2v88internal12_GLOBAL__N_123current_marking_barrierE
0000000000000130 l       .tbss  0000000000000008              _ZN2v88internal12_GLOBAL__N_136pending_layout_change_object_addressE
0000000000000138 l       .tbss  0000000000000008              _ZN2v88internal12_GLOBAL__N_118current_local_heapE
0000000000000140 l       .tbss  0000000000000008              .hidden _ZN8perfetto10DataSourceIN2v88internal14CodeDataSourceENS2_20CodeDataSourceTraitsEE10tls_state_E
0000000000000148 l       .tbss  0000000000000004              .hidden _ZN2v88internal12trap_handler21g_thread_in_wasm_codeE
0000000000000150 l       .tbss  0000000000000008              _ZN2v88internal4wasm12_GLOBAL__N_123current_code_refs_scopeE
0000000000000158 l       .tbss  0000000000000008              .hidden _ZN2v84base18ContextualVariableINS_8internal8compiler10turboshaft7TracingES5_E4top_E
0000000000000160 l       .tbss  0000000000000008              .hidden _ZN2v84base18ContextualVariableINS_8internal8compiler10turboshaft24TypeInferenceReducerArgsES5_E4top_E
0000000000000168 l       .tbss  0000000000000008              .hidden _ZN7content20media_stream_managerE
0000000000000170 l       .tbss  0000000000000008              .hidden _ZN10openscreen8internal20ScopedTraceOperation7traces_E
0000000000000178 l       .tbss  0000000000000008              .hidden _ZN10openscreen8internal20ScopedTraceOperation10root_node_E
0000000000000180 l       .tbss  0000000000000008              _ZN7content12_GLOBAL__N_114utility_threadE
0000000000000188 l       .tbss  0000000000000008              .hidden _ZN5blink18g_thread_specific_E
0000000000000190 l       .tbss  0000000000000008              _ZN5blink12_GLOBAL__N_114current_threadE
0000000000000198 l       .tbss  0000000000000008              _ZN6webrtc12_GLOBAL__N_121jingle_thread_wrapperE
00000000000001a0 l       .tbss  0000000000000008              _ZN10extensions12_GLOBAL__N_18contextsE
00000000000001a8 l       .tbss  0000000000000008              _ZN10extensions12_GLOBAL__N_119service_worker_dataE
00000000000001b0 l       .tbss  0000000000000010              _ZZN10__cxxabiv112_GLOBAL__N_19__globalsEvE10eh_globals
00000000000001c0 l       .tbss  0000000000000001              _ZN10__cxxabiv112_GLOBAL__N_111dtors_aliveE
00000000000001c8 l       .tbss  0000000000000008              _ZN10__cxxabiv112_GLOBAL__N_15dtorsE
00000000000001d0 l       .tbss  0000000000000008              .hidden _ZN4base8internal20current_notificationE
00000000000001d8 l       .tbss  0000000000000008              _ZN4base12_GLOBAL__N_18delegateE
00000000000001e0 l       .tbss  0000000000000008              _ZN4base12_GLOBAL__N_116run_loop_timeoutE
00000000000001e8 l       .tbss  0000000000000001              _ZN4base8internal12_GLOBAL__N_137current_task_is_running_synchronouslyE
00000000000001f0 l       .tbss  0000000000000008              _ZN4base12_GLOBAL__N_125scoped_defer_task_postingE
00000000000001f8 l       .tbss  0000000000000008              _ZN4base12_GLOBAL__N_120current_pending_taskE
0000000000000208 l       .tbss  0000000000000008              _ZN4base12_GLOBAL__N_125current_long_task_trackerE
0000000000000200 l       .tbss  0000000000000008              _ZN4base12_GLOBAL__N_123current_scoped_ipc_hashE
0000000000000210 l       .tbss  0000000000000008              _ZN4base16sequence_manager12_GLOBAL__N_129thread_local_sequence_managerE
0000000000000218 l       .tbss  0000000000000008              _ZN4base12_GLOBAL__N_122current_default_handleE
0000000000000220 l       .tbss  0000000000000008              _ZN4base12_GLOBAL__N_122current_default_handleE
0000000000000228 l       .tbss  0000000000000004              _ZN4base8internal12_GLOBAL__N_131fizzle_block_shutdown_tasks_refE
0000000000000230 l       .tbss  0000000000000008              _ZN4base8internal12_GLOBAL__N_120current_thread_groupE
0000000000000238 l       .tbss  0000000000000008              _ZN4base12_GLOBAL__N_116hang_watch_stateE
0000000000000240 l       .tbss  0000000000000008              _ZN4base8internal12_GLOBAL__N_117blocking_observerE
0000000000000248 l       .tbss  0000000000000008              _ZN4base8internal12_GLOBAL__N_125last_scoped_blocking_callE
0000000000000250 l       .tbss  0000000000000008              _ZN4base8internal12_GLOBAL__N_130current_sequence_local_storageE
0000000000000258 l       .tbss  0000000000000008              _ZN4base12_GLOBAL__N_110fd_watcherE
0000000000000260 l       .tbss  0000000000000008              _ZN4base11trace_event12_GLOBAL__N_125thread_local_event_bufferE
0000000000000268 l       .tbss  0000000000000001              _ZN4base11trace_event12_GLOBAL__N_126thread_blocks_message_loopE
0000000000000269 l       .tbss  0000000000000001              _ZN4base11trace_event12_GLOBAL__N_124thread_is_in_trace_eventE
0000000000000270 l       .tbss  0000000000000008              _ZZN4base11trace_event8TraceLog27ShouldAddAfterUpdatingStateEcPKhPKcmiNS_9TimeTicksEPNS0_14TraceArgumentsEE19current_thread_name
0000000000000278 l       .tbss  0000000000000001              _ZZN4base7tracing23GetThreadIsInTraceEventEvE24thread_is_in_trace_event
0000000000000280 l       .tbss  0000000000000008              _ZN4mojo8internal12_GLOBAL__N_119g_thread_local_nodeE
0000000000000288 l       .tbss  0000000000000008              _ZN4mojo12_GLOBAL__N_119g_end_to_end_metricE
00000000000002a0 l       .tbss  0000000000000001              __tls_guard
0000000000000290 l       .tbss  0000000000000010              _ZN4mojo12_GLOBAL__N_113g_sub_samplerE
00000000000002a1 l       .tbss  0000000000000001              _ZN4mojo12_GLOBAL__N_126is_in_urgent_message_scopeE
00000000000002c8 l       .tbss  0000000000000001              _ZGVZN6quiche12_GLOBAL__N_118Xoshiro256PlusPlusEvE9rng_state
00000000000002a8 l       .tbss  0000000000000020              _ZZN6quiche12_GLOBAL__N_118Xoshiro256PlusPlusEvE9rng_state
00000000000002d0 l       .tbss  0000000000000008              _ZN4quic12_GLOBAL__N_115current_contextE
00000000000002d8 l       .tbss  0000000000000001              _ZN3IPC12_GLOBAL__N_128off_sequence_binding_allowedE
00000000000002e0 l       .tbss  0000000000000008              _ZN3IPC12_GLOBAL__N_114received_queueE
00000000000002e8 l       .tbss  0000000000000008              .hidden _ZN8perfetto10DataSourceIN7tracing18MetadataDataSourceENS_23DefaultDataSourceTraitsEE10tls_state_E
00000000000002f0 l       .tbss  0000000000000008              .hidden _ZN8perfetto10DataSourceIN7tracing21PerfettoTracedProcess15DataSourceProxyINS1_24TraceEventMetadataSourceEEENS_23DefaultDataSourceTraitsEE10tls_state_E
00000000000002f8 l       .tbss  0000000000000008              .hidden _ZN8perfetto10DataSourceIN7tracing18TriggersDataSourceENS_23DefaultDataSourceTraitsEE10tls_state_E
0000000000000300 l       .tbss  0000000000000001              _ZN10variations12_GLOBAL__N_137in_set_field_trial_group_from_browserE
0000000000000308 l       .tbss  0000000000000008              _ZN6webrtc12_GLOBAL__N_17currentE
0000000000000310 l       .tbss  0000000000000008              _ZN4SkSLL8sMemPoolE
0000000000000318 l       .tbss  0000000000000008              _ZN5skgpu6ganeshL6gCacheE
0000000000000320 l       .tbss  0000000000000008              _ZN2ui12_GLOBAL__N_112event_sourceE
0000000000000328 l       .tbss  0000000000000008              _ZN2gl12_GLOBAL__N_115current_contextE
0000000000000330 l       .tbss  0000000000000008              _ZN2gl12_GLOBAL__N_120current_real_contextE
0000000000000338 l       .tbss  0000000000000008              _ZZN2gl20ThreadLocalCurrentGLEvE10current_gl
0000000000000340 l       .tbss  0000000000000008              _ZN2gl12_GLOBAL__N_115current_surfaceE
0000000000000348 l       .tbss  0000000000000008              .hidden _ZN3re25hooks7contextE
0000000000000350 l       .tbss  0000000000000008              _ZN3gpu12_GLOBAL__N_119current_task_runnerE
000000000000035c l       .tbss  0000000000000001              _ZGVZN3WTF13CurrentThreadEvE4g_id
0000000000000358 l       .tbss  0000000000000004              _ZZN3WTF13CurrentThreadEvE4g_id
000000000000035d l       .tbss  0000000000000001              .hidden _ZN3WTF16g_is_main_threadE
0000000000000360 l       .tbss  0000000000000008              _ZN2v84base12_GLOBAL__N_118thread_stack_startE
0000000000000368 l       .tbss  0000000000000008              .hidden _ZN8perfetto10DataSourceIN7tracing21PerfettoTracedProcess15DataSourceProxyIN22memory_instrumentation15TracingObserverEEENS_23DefaultDataSourceTraitsEE10tls_state_E
0000000000000370 l       .tbss  0000000000000008              _ZN7metrics12_GLOBAL__N_18providerE
0000000000000378 l       .tbss  0000000000000008              _ZN3gpu6webgpu12_GLOBAL__N_114parent_decoderE
0000000000000380 l       .tbss  0000000000000008              _ZN7content12_GLOBAL__N_113child_processE
0000000000000388 l       .tbss  0000000000000008              _ZN7content12_GLOBAL__N_117child_thread_implE
00000000000003c0 l       .tbss  0000000000000008              .hidden _ZGVZN5blink13HeapSizeCache16ForCurrentThreadEvE15heap_size_cache
0000000000000390 l       .tbss  0000000000000030              .hidden _ZZN5blink13HeapSizeCache16ForCurrentThreadEvE15heap_size_cache
00000000000003c8 l       .tbss  0000000000000004              .hidden _ZN5blink24script_forbidden_counterE
00000000000003d0 l       .tbss  0000000000000008              _ZN7content12_GLOBAL__N_113render_threadE
00000000000003d8 l       .tbss  0000000000000008              _ZN7content12_GLOBAL__N_113render_threadE
00000000000003e0 l       .tbss  0000000000000008              _ZN7content12_GLOBAL__N_111worker_dataE
00000000000003f0 l       .tbss  0000000000000001              _ZGVZN13sentencepiece6random18GetRandomGeneratorEvE2mt
00000000000003e8 l       .tbss  0000000000000008              _ZZN13sentencepiece6random18GetRandomGeneratorEvE2mt
00000000000003f8 l       .tbss  0000000000000008              .hidden _ZN8gwp_asan8internal16ThreadLocalStateINS0_29ThreadLocalRandomBitGeneratorEE6state_E
0000000000000400 l       .tbss  0000000000000008              .hidden _ZN8gwp_asan8internal16ThreadLocalStateINS0_13SamplingStateILNS0_15ParentAllocatorE2EEEE6state_E
0000000000000408 l       .tbss  0000000000000008              .hidden _ZN8gwp_asan8internal16ThreadLocalStateINS0_13SamplingStateILNS0_15ParentAllocatorE0EEEE6state_E
0000000000000410 l       .tbss  0000000000000008              .hidden _ZN8gwp_asan8internal16ThreadLocalStateINS0_13SamplingStateILNS0_15ParentAllocatorE1EEEE6state_E
0000000000000418 l       .tbss  0000000000000018              _ZN3std4hash6random11RandomState3new4KEYS29_$u7b$$u7b$constant$u7d$$u7d$28_$u7b$$u7b$closure$u7d$$u7d$3VAL17h022825e67d09fa26E
bash-4.4$

However, please let me know @magreenblatt if anyone knows a better solution :

salvadordf commented 5 days ago

I confirm this issue is present in Ubuntu 24.10 for ARM64 running in a Raspberry Pi 5. The workaround works in this case.

However, the official "Raspberry Pi OS" distribution is not affected by this issue and it's possible to load libcef.so without problems.

All tests were made with CEF 129.0.12 and CEF4Delphi in Lazarus

magreenblatt commented 2 days ago

@Hethsron Thanks for the details. I'm not seeing any large new contributors to MemSiz in your output.

In case of libcef.so (M-129), the output shows [...] a memory size (MemSiz) of 0x000430 [...] libcef.so version M~125 [...] MemSiz 0x0003d8

Can you compare the objdump -t libcef.so | grep -F '.tbss' output more closely between the versions to see where the additional memory usage is coming from? If you can identify a specific new culprit then we can potentially address it. It would be best to compare with M128, or whichever version most recently did not exhibit the problem.

Additionally:

Thus, the segment size in hex is 0x000430 and equal to 1072 bytes.

Whereas, from issue #3616:

The effective surplus allocated by default in x64 Linux (by ld-linux, which is the lib responsible to do this allocation) is sufficient for about 1600 or so bytes.

This suggests that your ARM64 system has a substantially lower threshold than the x64 default. You might want to see if you can adjust that in your system or kernel configuration.

chromiumembedded / cef

Dynamic loading of libcef.so on Linux ARM64 not possible anymore due to TLS size increase #3803