Dynamic loading of libcef.so on Linux ARM64 not possible anymore due to TLS size increase

Hethsron commented 1 month ago

Describe the bug Dynamically loading libcef.so into a running process on Linux arm64 fails since CEF 129 with the error message "cannot allocate memory in static TLS block". Since a lot of CEF wrappers depend on the ability to dynamically extract and load libcef.so (known: CEF4Delphi, JCEF, probably more), this issue effectively makes CEF 129+ unusable for these wrappers on Linux arm64.

To Reproduce Either use CEF 129 with for example JCEF on Linux, or use the following synthetic console application to dlopen libcef.so:

#include <stdio.h>
#include <dlfcn.h>

int main (void)
{
  void* g_libcef_handle = dlopen("./libcef.so", RTLD_LAZY | RTLD_LOCAL);
  if (!g_libcef_handle) {
    fprintf(stderr, "dlerror %s\n", dlerror());
  }
  return 0;
}

Compile the source code :

g++ -o main.cpp -ldl

Run :

bash-4.4$ ./a.out 
dlerror ./libcef.so: cannot allocate memory in static TLS block
bash-4.4$

Expected behavior libcef.so can be dynamically loaded into a Linux process.

Screenshots If applicable, add screenshots to help explain your problem.

Versions (please complete the following information):

OS: Linux (all ARM popular distributions)
CEF Version: 129.x (6668)

Background/Analysis The TLS segment size of Chromium 129 increases by over 700 bytes, as can be seen here:

bash-4.4$ readelf -Wl libcef.so | grep -E 'PhysAddr|TLS'
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  TLS            0xc7cc340 0x000000000c7ec340 0x000000000c7ec340 0x000090 0x000430 R   0x40
bash-4.4$

This issue is similar to #3616.

Hethsron commented 1 month ago

@magreenblatt Hi Marshall, do you have a fix for that ?

magreenblatt commented 1 month ago

Does this also impact x64 Intel builds? You can use the tips from #3616 to debug the issue.

Hethsron commented 1 month ago

Does this also impact x64 Intel builds? You can use the tips from #3616 to debug the issue.

Intel x64 versions are not impacted. I will try to debug the issue with these tips and let you as soon as possible a further analysis.

Hethsron commented 1 month ago

@magreenblatt

After following tips, here is my analysis :

In case of libcef.so (M~129), the output shows that the TLS segment has a file size (FileSiz) of 0x000090 and a memory size (MemSiz) of 0x000430. Thus, the segment size in hex is 0x000430 and equal to 1072 bytes.

Earlier versions only had a TLS segment size of 0x0003d8 or 984 bytes. For example, here is the result of the command on libcef.so version M~125 :

readelf -Wl libcef.so | grep -E 'PhysAddr|TLS'
Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
TLS            0xc214fc0 0x000000000c234fc0 0x000000000c234fc0 0x000090 0x0003d8 R   0x40

Here are the two TLS segment results compared side by side:

Property	First Result	Second Result	Difference
Offset	0xc7cc340	0xc214fc0	-
VirtAddr	0x000000000c7ec340	0x000000000c234fc0	-
PhysAddr	0x000000000c7ec340	0x000000000c234fc0	-
FileSiz	0x000090 (144 bytes)	0x000090 (144 bytes)	No difference
MemSiz	0x000430 (1072 bytes)	0x0003d8 (984 bytes)	88 bytes (1072 - 984)
Flags	R	R	No difference
Align	0x40	0x40	No difference

Key Differences:

The memory size (MemSiz) is different:
- First result: 1072 bytes (0x000430)
- Second result: 984 bytes (0x0003d8) The difference in size is 88 bytes. Other fields such as Offset, VirtAddr, PhysAddr, FileSiz, and flags remain the same, except for different starting addresses.

Version used for test

cef_binary_129.0.12+gf09539f+chromium-129.0.6668.101_linuxarm64_client

Workaround Just for the record, this workaround works. But it's not acceptable in our case:

export LD_PRELOAD=<FULL-PATH-TO-libcef.so>

And the following patch is already present in config.h:

diff --git third_party/libxml/linux/config.h third_party/libxml/linux/config.h
index c064071ce1545..65110af9a78f5 100644
--- third_party/libxml/linux/config.h
+++ third_party/libxml/linux/config.h
@@ -171,7 +171,7 @@
/* #undef XML_SOCKLEN_T */

/* TLS specifier */
-#define XML_THREAD_LOCAL _Thread_local
+/* #undef XML_THREAD_LOCAL */

/* Define for Solaris 2.5.1 so the uint32_t typedef from <sys/synch.h>,
<pthread.h>, or <semaphore.h> is not used. If the typedef were allowed, the

Deep Investigation Further investigation found the following usage (.tdata and .tbss are the symbol sections of interest when looking for the actual users of all that TLS space - this article here (A dynamic linker murder mystery) was very helpful in learning interesting tidbits about TLS, and commands to investigate the situation) :

bash-4.4$ llvm-objdump -C -t libcef.so | grep -F '.tdata'
0000000000000000 l       .tdata 0000000000000000 $d
0000000000000000 l       .tdata 0000000000000010 .hidden absl::cord_internal::cordz_next_sample
0000000000000010 l       .tdata 0000000000000004 v8::internal::(anonymous namespace)::current_per_thread_assert_data
0000000000000010 l       .tdata 0000000000000000 $d
0000000000000014 l       .tdata 0000000000000004 base::internal::(anonymous namespace)::current_sequence_token
0000000000000018 l       .tdata 0000000000000004 base::internal::(anonymous namespace)::current_task_token
000000000000001c l       .tdata 0000000000000001 base::internal::(anonymous namespace)::current_task_is_thread_bound
0000000000000014 l       .tdata 0000000000000000 $d
0000000000000018 l       .tdata 0000000000000000 $d
000000000000001c l       .tdata 0000000000000000 $d
000000000000001d l       .tdata 0000000000000001 base::internal::(anonymous namespace)::task_priority_for_current_thread
000000000000001d l       .tdata 0000000000000000 $d
0000000000000020 l       .tdata 0000000000000004 base::(anonymous namespace)::current_thread_type
0000000000000020 l       .tdata 0000000000000000 $d
0000000000000028 l       .tdata 0000000000000008 base::(anonymous namespace)::thread_name
0000000000000028 l       .tdata 0000000000000000 $d
0000000000000030 l       .tdata 0000000000000004 base::(anonymous namespace)::g_thread_id
0000000000000034 l       .tdata 0000000000000001 base::(anonymous namespace)::g_is_main_thread
0000000000000030 l       .tdata 0000000000000000 $d
0000000000000034 l       .tdata 0000000000000000 $d
0000000000000038 l       .tdata 0000000000000004 partition_alloc::internal::base::(anonymous namespace)::g_thread_id
000000000000003c l       .tdata 0000000000000001 partition_alloc::internal::base::(anonymous namespace)::g_is_main_thread
0000000000000038 l       .tdata 0000000000000000 $d
000000000000003c l       .tdata 0000000000000000 $d
0000000000000040 l       .tdata 0000000000000000 $d
0000000000000040 l       .tdata 0000000000000040 .hidden google::protobuf::internal::ThreadSafeArena::thread_cache_
0000000000000080 l       .tdata 0000000000000004 simd_support
0000000000000088 l       .tdata 0000000000000004 simd_features
0000000000000084 l       .tdata 0000000000000004 simd_huffman
0000000000000080 l       .tdata 0000000000000000 $d
0000000000000084 l       .tdata 0000000000000000 $d
0000000000000088 l       .tdata 0000000000000000 $d
000000000000008c l       .tdata 0000000000000000 $d
000000000000008c l       .tdata 0000000000000004 .hidden blink::next_world_id
0000000000000090 l       .tdata 0000000000000020 std::sys::thread_local::destructors::list::DTORS::h9d4e40d7fcc5ca6c
0000000000000090 l       .tdata 0000000000000000 $d

bash-4.4$ llvm-objdump -C -t libcef.so | grep -F '.tbss' 
00000000000000b0 l       .tbss  0000000000000000 $d
00000000000000b0 l       .tbss  0000000000000008 .hidden perfetto::DataSource<base::perfetto_track_event::TrackEvent, perfetto::internal::TrackEventDataSourceTraits>::tls_state_
00000000000000b8 l       .tbss  0000000000000008 (anonymous namespace)::g_isolate_manager
00000000000000b8 l       .tbss  0000000000000000 $d
00000000000000c4 l       .tbss  0000000000000001 guard variable for absl::base_internal::GetCachedTID()::thread_id
00000000000000c0 l       .tbss  0000000000000004 absl::base_internal::GetCachedTID()::thread_id
00000000000000c0 l       .tbss  0000000000000000 $d
00000000000000c4 l       .tbss  0000000000000000 $d
00000000000000c8 l       .tbss  0000000000000018 absl::cord_internal::cordz_should_profile_slow(absl::cord_internal::SamplingState&)::exponential_biased_generator
00000000000000c8 l       .tbss  0000000000000000 $d
00000000000000e0 l       .tbss  0000000000000001 absl::log_internal::(anonymous namespace)::ThreadIsLoggingStatus()::thread_is_logging
00000000000000e0 l       .tbss  0000000000000000 $d
00000000000000e1 l       .tbss  0000000000000001 ipcz::(anonymous namespace)::is_thread_within_api_call
00000000000000e1 l       .tbss  0000000000000000 $d
00000000000000e8 l       .tbss  0000000000000000 $d
00000000000000e8 l       .tbss  0000000000000008 .hidden perfetto::DataSource<webrtc::perfetto_track_event::TrackEvent, perfetto::internal::TrackEventDataSourceTraits>::tls_state_
00000000000000f8 l       .tbss  0000000000000001 guard variable for SkStrikeCache::GlobalStrikeCache()::cache
00000000000000f0 l       .tbss  0000000000000008 SkStrikeCache::GlobalStrikeCache()::cache
00000000000000f0 l       .tbss  0000000000000000 $d
00000000000000f8 l       .tbss  0000000000000000 $d
0000000000000100 l       .tbss  0000000000000008 dawn::native::(anonymous namespace)::tlDevice
0000000000000100 l       .tbss  0000000000000000 $d
0000000000000108 l       .tbss  0000000000000000 $d
0000000000000108 l       .tbss  0000000000000008 .hidden perfetto::DataSource<v8::perfetto_track_event::TrackEvent, perfetto::internal::TrackEventDataSourceTraits>::tls_state_
0000000000000110 l       .tbss  0000000000000001 v8::internal::(anonymous namespace)::tls_singleton_taken
0000000000000118 l       .tbss  0000000000000018 v8::internal::(anonymous namespace)::tls_singleton_storage
0000000000000110 l       .tbss  0000000000000000 $d
0000000000000118 l       .tbss  0000000000000000 $d
0000000000000130 l       .tbss  0000000000000000 $d
0000000000000138 l       .tbss  0000000000000000 $d
0000000000000138 l       .tbss  0000000000000008 .hidden v8::internal::g_current_isolate_
0000000000000130 l       .tbss  0000000000000008 .hidden v8::internal::g_current_per_isolate_thread_data_
0000000000000140 l       .tbss  0000000000000004 v8::internal::(anonymous namespace)::thread_id
0000000000000140 l       .tbss  0000000000000000 $d
0000000000000148 l       .tbss  0000000000000008 v8::internal::(anonymous namespace)::current_marking_barrier
0000000000000148 l       .tbss  0000000000000000 $d
0000000000000150 l       .tbss  0000000000000008 v8::internal::(anonymous namespace)::pending_layout_change_object_address
0000000000000150 l       .tbss  0000000000000000 $d
0000000000000158 l       .tbss  0000000000000008 v8::internal::(anonymous namespace)::current_local_heap
0000000000000158 l       .tbss  0000000000000000 $d
0000000000000160 l       .tbss  0000000000000000 $d
0000000000000160 l       .tbss  0000000000000008 .hidden perfetto::DataSource<v8::internal::CodeDataSource, v8::internal::CodeDataSourceTraits>::tls_state_
0000000000000168 l       .tbss  0000000000000000 $d
0000000000000168 l       .tbss  0000000000000004 .hidden v8::internal::trap_handler::g_thread_in_wasm_code
0000000000000170 l       .tbss  0000000000000008 v8::internal::wasm::(anonymous namespace)::current_code_refs_scope
0000000000000170 l       .tbss  0000000000000000 $d
0000000000000178 l       .tbss  0000000000000000 $d
0000000000000178 l       .tbss  0000000000000008 .hidden v8::base::ContextualVariable<v8::internal::compiler::turboshaft::Tracing, v8::internal::compiler::turboshaft::Tracing>::top_
0000000000000180 l       .tbss  0000000000000000 $d
0000000000000180 l       .tbss  0000000000000008 .hidden v8::base::ContextualVariable<v8::internal::compiler::turboshaft::TypeInferenceReducerArgs, v8::internal::compiler::turboshaft::TypeInferenceReducerArgs>::top_
0000000000000188 l       .tbss  0000000000000000 $d
0000000000000188 l       .tbss  0000000000000008 .hidden content::media_stream_manager
0000000000000190 l       .tbss  0000000000000000 $d
0000000000000198 l       .tbss  0000000000000000 $d
0000000000000190 l       .tbss  0000000000000008 .hidden openscreen::internal::ScopedTraceOperation::traces_
0000000000000198 l       .tbss  0000000000000008 .hidden openscreen::internal::ScopedTraceOperation::root_node_
00000000000001a0 l       .tbss  0000000000000008 content::(anonymous namespace)::utility_thread
00000000000001a0 l       .tbss  0000000000000000 $d
00000000000001a8 l       .tbss  0000000000000000 $d
00000000000001a8 l       .tbss  0000000000000008 .hidden blink::g_thread_specific_
00000000000001b0 l       .tbss  0000000000000008 blink::(anonymous namespace)::current_thread
00000000000001b0 l       .tbss  0000000000000000 $d
00000000000001b8 l       .tbss  0000000000000008 webrtc::(anonymous namespace)::jingle_thread_wrapper
00000000000001b8 l       .tbss  0000000000000000 $d
00000000000001c0 l       .tbss  0000000000000008 extensions::(anonymous namespace)::contexts
00000000000001c0 l       .tbss  0000000000000000 $d
00000000000001c8 l       .tbss  0000000000000008 extensions::(anonymous namespace)::service_worker_data
00000000000001c8 l       .tbss  0000000000000000 $d
00000000000001d0 l       .tbss  0000000000000010 __cxxabiv1::(anonymous namespace)::__globals()::eh_globals
00000000000001d0 l       .tbss  0000000000000000 $d
00000000000001e0 l       .tbss  0000000000000001 __cxxabiv1::(anonymous namespace)::dtors_alive
00000000000001e8 l       .tbss  0000000000000008 __cxxabiv1::(anonymous namespace)::dtors
00000000000001e0 l       .tbss  0000000000000000 $d
00000000000001e8 l       .tbss  0000000000000000 $d
00000000000001f0 l       .tbss  0000000000000000 $d
00000000000001f0 l       .tbss  0000000000000008 .hidden base::internal::current_notification
00000000000001f8 l       .tbss  0000000000000008 base::(anonymous namespace)::delegate
0000000000000200 l       .tbss  0000000000000008 base::(anonymous namespace)::run_loop_timeout
00000000000001f8 l       .tbss  0000000000000000 $d
0000000000000200 l       .tbss  0000000000000000 $d
0000000000000208 l       .tbss  0000000000000001 base::internal::(anonymous namespace)::current_task_is_running_synchronously
0000000000000208 l       .tbss  0000000000000000 $d
0000000000000210 l       .tbss  0000000000000008 base::(anonymous namespace)::scoped_defer_task_posting
0000000000000210 l       .tbss  0000000000000000 $d
0000000000000218 l       .tbss  0000000000000008 base::(anonymous namespace)::current_pending_task
0000000000000228 l       .tbss  0000000000000008 base::(anonymous namespace)::current_long_task_tracker
0000000000000220 l       .tbss  0000000000000008 base::(anonymous namespace)::current_scoped_ipc_hash
0000000000000218 l       .tbss  0000000000000000 $d
0000000000000220 l       .tbss  0000000000000000 $d
0000000000000228 l       .tbss  0000000000000000 $d
0000000000000230 l       .tbss  0000000000000008 base::sequence_manager::(anonymous namespace)::thread_local_sequence_manager
0000000000000230 l       .tbss  0000000000000000 $d
0000000000000238 l       .tbss  0000000000000008 base::(anonymous namespace)::current_default_handle
0000000000000238 l       .tbss  0000000000000000 $d
0000000000000240 l       .tbss  0000000000000008 base::(anonymous namespace)::current_default_handle
0000000000000240 l       .tbss  0000000000000000 $d
0000000000000248 l       .tbss  0000000000000004 base::internal::(anonymous namespace)::fizzle_block_shutdown_tasks_ref
0000000000000248 l       .tbss  0000000000000000 $d
0000000000000250 l       .tbss  0000000000000008 base::internal::(anonymous namespace)::current_thread_group
0000000000000250 l       .tbss  0000000000000000 $d
0000000000000258 l       .tbss  0000000000000008 base::(anonymous namespace)::hang_watch_state
0000000000000258 l       .tbss  0000000000000000 $d
0000000000000260 l       .tbss  0000000000000008 base::internal::(anonymous namespace)::blocking_observer
0000000000000268 l       .tbss  0000000000000008 base::internal::(anonymous namespace)::last_scoped_blocking_call
0000000000000260 l       .tbss  0000000000000000 $d
0000000000000268 l       .tbss  0000000000000000 $d
0000000000000270 l       .tbss  0000000000000008 base::internal::(anonymous namespace)::current_sequence_local_storage
0000000000000270 l       .tbss  0000000000000000 $d
0000000000000278 l       .tbss  0000000000000001 base::(anonymous namespace)::tls_blocking_disallowed
0000000000000279 l       .tbss  0000000000000001 base::(anonymous namespace)::tls_base_sync_primitives_disallowed
000000000000027a l       .tbss  0000000000000001 base::(anonymous namespace)::tls_cpu_intensive_work_disallowed
000000000000027b l       .tbss  0000000000000001 base::(anonymous namespace)::tls_singleton_disallowed
0000000000000278 l       .tbss  0000000000000000 $d
0000000000000279 l       .tbss  0000000000000000 $d
000000000000027a l       .tbss  0000000000000000 $d
000000000000027b l       .tbss  0000000000000000 $d
0000000000000280 l       .tbss  0000000000000008 base::(anonymous namespace)::fd_watcher
0000000000000280 l       .tbss  0000000000000000 $d
0000000000000288 l       .tbss  0000000000000008 base::trace_event::(anonymous namespace)::thread_local_event_buffer
0000000000000290 l       .tbss  0000000000000001 base::trace_event::(anonymous namespace)::thread_blocks_message_loop
0000000000000291 l       .tbss  0000000000000001 base::trace_event::(anonymous namespace)::thread_is_in_trace_event
0000000000000298 l       .tbss  0000000000000008 base::trace_event::TraceLog::ShouldAddAfterUpdatingState(char, unsigned char const*, char const*, unsigned long, int, base::TimeTicks, base::trace_event::TraceArguments*)::current_thread_name
0000000000000288 l       .tbss  0000000000000000 $d
0000000000000290 l       .tbss  0000000000000000 $d
0000000000000291 l       .tbss  0000000000000000 $d
0000000000000298 l       .tbss  0000000000000000 $d
00000000000002a0 l       .tbss  0000000000000001 base::tracing::GetThreadIsInTraceEvent()::thread_is_in_trace_event
00000000000002a0 l       .tbss  0000000000000000 $d
00000000000002a8 l       .tbss  0000000000000008 mojo::internal::(anonymous namespace)::g_thread_local_node
00000000000002a8 l       .tbss  0000000000000000 $d
00000000000002b0 l       .tbss  0000000000000008 mojo::(anonymous namespace)::g_end_to_end_metric
00000000000002b0 l       .tbss  0000000000000000 $d
00000000000002c8 l       .tbss  0000000000000001 __tls_guard
00000000000002b8 l       .tbss  0000000000000010 mojo::(anonymous namespace)::g_sub_sampler
00000000000002b8 l       .tbss  0000000000000000 $d
00000000000002c8 l       .tbss  0000000000000000 $d
00000000000002c9 l       .tbss  0000000000000001 mojo::(anonymous namespace)::is_in_urgent_message_scope
00000000000002c9 l       .tbss  0000000000000000 $d
00000000000002f0 l       .tbss  0000000000000001 guard variable for quiche::(anonymous namespace)::Xoshiro256PlusPlus()::rng_state
00000000000002d0 l       .tbss  0000000000000020 quiche::(anonymous namespace)::Xoshiro256PlusPlus()::rng_state
00000000000002d0 l       .tbss  0000000000000000 $d
00000000000002f0 l       .tbss  0000000000000000 $d
00000000000002f8 l       .tbss  0000000000000008 quic::(anonymous namespace)::current_context
00000000000002f8 l       .tbss  0000000000000000 $d
0000000000000300 l       .tbss  0000000000000001 IPC::(anonymous namespace)::off_sequence_binding_allowed
0000000000000300 l       .tbss  0000000000000000 $d
0000000000000308 l       .tbss  0000000000000008 IPC::(anonymous namespace)::received_queue
0000000000000308 l       .tbss  0000000000000000 $d
0000000000000310 l       .tbss  0000000000000000 $d
0000000000000310 l       .tbss  0000000000000008 .hidden perfetto::DataSource<tracing::MetadataDataSource, perfetto::DefaultDataSourceTraits>::tls_state_
0000000000000318 l       .tbss  0000000000000000 $d
0000000000000318 l       .tbss  0000000000000008 .hidden perfetto::DataSource<tracing::PerfettoTracedProcess::DataSourceProxy<tracing::TraceEventMetadataSource>, perfetto::DefaultDataSourceTraits>::tls_state_
0000000000000320 l       .tbss  0000000000000000 $d
0000000000000320 l       .tbss  0000000000000008 .hidden perfetto::DataSource<tracing::TriggersDataSource, perfetto::DefaultDataSourceTraits>::tls_state_
0000000000000328 l       .tbss  0000000000000001 variations::(anonymous namespace)::in_set_field_trial_group_from_browser
0000000000000328 l       .tbss  0000000000000000 $d
0000000000000330 l       .tbss  0000000000000008 webrtc::(anonymous namespace)::current
0000000000000330 l       .tbss  0000000000000000 $d
0000000000000338 l       .tbss  0000000000000008 SkSL::sMemPool
0000000000000338 l       .tbss  0000000000000000 $d
0000000000000340 l       .tbss  0000000000000008 skgpu::ganesh::gCache
0000000000000340 l       .tbss  0000000000000000 $d
0000000000000348 l       .tbss  0000000000000008 ui::(anonymous namespace)::event_source
0000000000000348 l       .tbss  0000000000000000 $d
0000000000000350 l       .tbss  0000000000000008 gl::(anonymous namespace)::current_context
0000000000000358 l       .tbss  0000000000000008 gl::(anonymous namespace)::current_real_context
0000000000000350 l       .tbss  0000000000000000 $d
0000000000000358 l       .tbss  0000000000000000 $d
0000000000000360 l       .tbss  0000000000000008 gl::ThreadLocalCurrentGL()::current_gl
0000000000000360 l       .tbss  0000000000000000 $d
0000000000000368 l       .tbss  0000000000000008 gl::(anonymous namespace)::current_surface
0000000000000368 l       .tbss  0000000000000000 $d
0000000000000370 l       .tbss  0000000000000000 $d
0000000000000370 l       .tbss  0000000000000008 .hidden re2::hooks::context
0000000000000378 l       .tbss  0000000000000008 gpu::(anonymous namespace)::current_task_runner
0000000000000378 l       .tbss  0000000000000000 $d
0000000000000384 l       .tbss  0000000000000001 guard variable for WTF::CurrentThread()::g_id
0000000000000380 l       .tbss  0000000000000004 WTF::CurrentThread()::g_id
0000000000000380 l       .tbss  0000000000000000 $d
0000000000000384 l       .tbss  0000000000000000 $d
0000000000000385 l       .tbss  0000000000000000 $d
0000000000000385 l       .tbss  0000000000000001 .hidden WTF::g_is_main_thread
0000000000000388 l       .tbss  0000000000000008 v8::base::(anonymous namespace)::thread_stack_start
0000000000000388 l       .tbss  0000000000000000 $d
0000000000000390 l       .tbss  0000000000000000 $d
0000000000000390 l       .tbss  0000000000000008 .hidden perfetto::DataSource<tracing::PerfettoTracedProcess::DataSourceProxy<memory_instrumentation::TracingObserver>, perfetto::DefaultDataSourceTraits>::tls_state_
0000000000000398 l       .tbss  0000000000000008 metrics::(anonymous namespace)::provider
0000000000000398 l       .tbss  0000000000000000 $d
00000000000003a0 l       .tbss  0000000000000008 gpu::webgpu::(anonymous namespace)::parent_decoder
00000000000003a0 l       .tbss  0000000000000000 $d
00000000000003a8 l       .tbss  0000000000000008 content::(anonymous namespace)::child_process
00000000000003a8 l       .tbss  0000000000000000 $d
00000000000003b0 l       .tbss  0000000000000008 content::(anonymous namespace)::child_thread_impl
00000000000003b0 l       .tbss  0000000000000000 $d
00000000000003b8 l       .tbss  0000000000000000 $d
00000000000003e8 l       .tbss  0000000000000000 $d
00000000000003e8 l       .tbss  0000000000000008 .hidden guard variable for blink::HeapSizeCache::ForCurrentThread()::heap_size_cache
00000000000003b8 l       .tbss  0000000000000030 .hidden blink::HeapSizeCache::ForCurrentThread()::heap_size_cache
00000000000003f0 l       .tbss  0000000000000000 $d
00000000000003f0 l       .tbss  0000000000000004 .hidden blink::script_forbidden_counter
00000000000003f8 l       .tbss  0000000000000008 content::(anonymous namespace)::render_thread
00000000000003f8 l       .tbss  0000000000000000 $d
0000000000000400 l       .tbss  0000000000000008 content::(anonymous namespace)::render_thread
0000000000000400 l       .tbss  0000000000000000 $d
0000000000000408 l       .tbss  0000000000000008 content::(anonymous namespace)::worker_data
0000000000000408 l       .tbss  0000000000000000 $d
0000000000000410 l       .tbss  0000000000000008 AffixMgr::compound_check(std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char>> const&, short, short, short, short, hentry**, hentry**, char, char, int*)::timelimit
0000000000000418 l       .tbss  0000000000000008 AffixMgr::compound_check_morph(char const*, int, short, short, short, short, hentry**, hentry**, char, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char>>&, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char>> const*)::timelimit
0000000000000410 l       .tbss  0000000000000000 $d
0000000000000418 l       .tbss  0000000000000000 $d
0000000000000428 l       .tbss  0000000000000001 guard variable for sentencepiece::random::GetRandomGenerator()::mt
0000000000000420 l       .tbss  0000000000000008 sentencepiece::random::GetRandomGenerator()::mt
0000000000000420 l       .tbss  0000000000000000 $d
0000000000000428 l       .tbss  0000000000000000 $d
0000000000000430 l       .tbss  0000000000000000 $d
0000000000000430 l       .tbss  0000000000000008 .hidden gwp_asan::internal::ThreadLocalState<gwp_asan::internal::ThreadLocalRandomBitGenerator>::state_
0000000000000438 l       .tbss  0000000000000000 $d
0000000000000438 l       .tbss  0000000000000008 .hidden gwp_asan::internal::ThreadLocalState<gwp_asan::internal::SamplingState<(gwp_asan::internal::ParentAllocator)2>>::state_
0000000000000440 l       .tbss  0000000000000000 $d
0000000000000440 l       .tbss  0000000000000008 .hidden gwp_asan::internal::ThreadLocalState<gwp_asan::internal::SamplingState<(gwp_asan::internal::ParentAllocator)0>>::state_
0000000000000448 l       .tbss  0000000000000000 $d
0000000000000448 l       .tbss  0000000000000008 .hidden gwp_asan::internal::ThreadLocalState<gwp_asan::internal::SamplingState<(gwp_asan::internal::ParentAllocator)1>>::state_
0000000000000450 l       .tbss  0000000000000000 $d
0000000000000450 l       .tbss  0000000000000018 std::hash::random::RandomState::new::KEYS::_$u7b$$u7b$constant$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::VAL::hd9d6164a1b6881c6

However, please let me know @magreenblatt if anyone knows a better solution :

salvadordf commented 1 month ago

I confirm this issue is present in Ubuntu 24.10 for ARM64 running in a Raspberry Pi 5. The workaround works in this case.

However, the official "Raspberry Pi OS" distribution is not affected by this issue and it's possible to load libcef.so without problems.

All tests were made with CEF 129.0.12 and CEF4Delphi in Lazarus

magreenblatt commented 1 month ago

@Hethsron Thanks for the details. I'm not seeing any large new contributors to MemSiz in your output.

In case of libcef.so (M-129), the output shows [...] a memory size (MemSiz) of 0x000430 [...] libcef.so version M~125 [...] MemSiz 0x0003d8

Can you compare the objdump -t libcef.so | grep -F '.tbss' output more closely between the versions to see where the additional memory usage is coming from? If you can identify a specific new culprit then we can potentially address it. It would be best to compare with M128, or whichever version most recently did not exhibit the problem.

Additionally:

Thus, the segment size in hex is 0x000430 and equal to 1072 bytes.

Whereas, from issue #3616:

The effective surplus allocated by default in x64 Linux (by ld-linux, which is the lib responsible to do this allocation) is sufficient for about 1600 or so bytes.

This suggests that your ARM64 system has a substantially lower threshold than the x64 default. You might want to see if you can adjust that in your system or kernel configuration.

Hethsron commented 1 month ago

@Hethsron Thanks for the details. I'm not seeing any large new contributors to MemSiz in your output.

In case of libcef.so (M-129), the output shows [...] a memory size (MemSiz) of 0x000430 [...] libcef.so version M~125 [...] MemSiz 0x0003d8

Can you compare the objdump -t libcef.so | grep -F '.tbss' output more closely between the versions to see where the additional memory usage is coming from? If you can identify a specific new culprit then we can potentially address it. It would be best to compare with M128, or whichever version most recently did not exhibit the problem.

@magreenblatt Here is a script I developed to answer this request :

#!/bin/bash

# Array of folder names
folders=(cef125 cef126 cef127 cef128 cef129 cef130 cef131)

# Create a directory to store temporary files for comparisons
temp_dir=$(mktemp -d)
trap "rm -rf $temp_dir" EXIT  # Clean up temp files on exit

# Collect the output of llvm-objdump for each folder
for folder in "${folders[@]}"; do
    if [[ -f "$folder/libcef.so" ]]; then
        echo "Processing $folder/libcef.so"
        # Run llvm-objdump and filter for .tbss, store in a temp file
        llvm-objdump -C -t "$folder/libcef.so" | grep -F '.tbss' > "$temp_dir/$folder.tbss"
    else
        echo "Warning: $folder/libcef.so not found, skipping."
    fi
done

# Compare each version's output line-by-line using diff
for ((i=0; i<${#folders[@]}-1; i++)); do
    for ((j=i+1; j<${#folders[@]}; j++)); do
        folder1=${folders[i]}
        folder2=${folders[j]}
        file1="$temp_dir/$folder1.tbss"
        file2="$temp_dir/$folder2.tbss"

        if [[ -f "$file1" && -f "$file2" ]]; then
            echo "Comparing $folder1 with $folder2:"
            diff -y --suppress-common-lines "$file1" "$file2" || echo "Differences found"
            echo "---------------------------------------"
        fi
    done
done

To get the output, you can execute the following command:

bash-4.4$ chmod +x compare_cef_tbss.sh
bash-4.4$ ./compare_cef_tbss.sh > tbss_comparison.txt

magreenblatt commented 1 month ago

@Hethsron Thanks, that's the first step. Now compare the output to identify what's changed.

FlavionTrack commented 3 weeks ago

I'm testing on Ubuntu 24.10 Concept for Snapdragon X Elite arm64 laptops, I've tested CEF 118, 127, 128 and 130 and for me they all show the "cannot allocate memory in static TLS block" error when dynamically loading libcef.so

Hethsron commented 1 week ago

I'm testing on Ubuntu 24.10 Concept for Snapdragon X Elite arm64 laptops, I've tested CEF 118, 127, 128 and 130 and for me they all show the "cannot allocate memory in static TLS block" error when dynamically loading libcef.so

@magreenblatt I will let you know as soon as possible if i perform to identify what's changed on those versions of CEF.

Hethsron commented 1 week ago

@magreenblatt As promised, I'll get back to you with the results. I used the following CEF versions from spotify :

And when I run my script, I have the following output : tbss_comparison.txt

Can you see the changes for yourself? Thanks in advance for your feedback

Regards

Hethsron

chromiumembedded / cef

Dynamic loading of libcef.so on Linux ARM64 not possible anymore due to TLS size increase #3803