Building/running a ThreadX Module under GCC where there are pointers to global variables

[RobMeades]

We are using ThreadX and ThreadX Modules in our product, have been doing so for many years. So far we have always built under ARMCC. We now need to build the ThreadX Modules under GCC (12.2 release 1).

We have followed the form of your example (i.e. compilation switches -fpie -fno-plt -mno-pic-data-is-text-relative -msingle-pic-base, no particular linker switches) and in general the code runs, however there is a specific case where these compilation flags do NOT work: when taking the address of a global variable. For instance:

char buffer[100];
char* pBuffer = buffer;

int appModMain()
{
    printf("buffer address: %p, buffer address via pBuffer: %p \n", buffer, pBuffer);
}

...results in a printed output of the form:

buffer address:  60C8C25C, buffer address via pBuffer: 1000025C

In other words pBuffer is not being offset correctly. This is clear when you look at the disassembled code:

00000188 <appModMain>:
     188:   b580        push    {r7, lr}
     18a:   af00        add r7, sp, #0
     18c:   4b1a        ldr r3, [pc, #104]  ; (1f8 <appModMain+0x70>)
     18e:   f859 3003   ldr.w   r3, [r9, r3]
     192:   681b        ldr r3, [r3, #0]
     194:   461a        mov r2, r3
     196:   4b19        ldr r3, [pc, #100]  ; (1fc <appModMain+0x74>)
     198:   f859 3003   ldr.w   r3, [r9, r3]
     19c:   4619        mov r1, r3
     19e:   4b18        ldr r3, [pc, #96]   ; (200 <appModMain+0x78>)
     1a0:   f859 3003   ldr.w   r3, [r9, r3]
     1a4:   4618        mov r0, r3
     1a6:   f000 f8bb   bl  320 <printf>
     1aa:   f640 30b8   movw    r0, #3000   ; 0xbb8

When loading-up the variable to put into r2, which is going to be passed to printf() as pBuffer, r3 needs to be adjusted, yet it is not (detail below).

We've tried various permutations of -mpic-data-is-text-relative/-mnopic-data-is-text-relative and -msingle-pic-base/-mnosingle-pic-base but none of them make a difference: we just cannot make a pointer reference to a global variable work. Unfortunately we can't change the source code we are compiling, there's quite a large amount of it and some if it is not ours; and of course this is perfectly valid C code.

Can you suggest how to compile a ThreadX Module under GCC so that it runs correctly?

Detail:

In the disassembled output, the contents of 1f8 and 1fc for the example above are:

     1f8:   000001bc
     1fc:   000001b0

...in the .got section of the disassembled output, these offset are towards the end, here:

Disassembly of section .got:

00006b88 <__ctors_end__>:
...
    6d34:   1002c5a8
    6d38:   1000025c    ; 6b88 + 1b0
    6d3c:   00001e15
    6d40:   1002c5ac
    6d44:   100001d0    ; 6b88 + 1bc

...and 100001d0 in the disassembled output turns out to be:

Disassembly of section .data:

...

100001d0 <pBuffer>:
100001d0:   1000025c

...while 1000025c in the disassembled output turns out to be:

Disassembly of section .bss:

...

1000025c <buffer>:
    ...

So pBuffer is initialised by the compiler to the un-offsetted address of buffer, and the compiler knows that pBuffer it is a pointer to something in the GOT, yet it did not do anything to add the offset when it generated the code.

[RobMeades]

Note: corrected the GCC version above, we are using the latest version, 12.2 release 1.

[RobMeades]

In case it is useful, there is a hint in this Stack Overflow question, where the "failing" code has a similar construct to ours, that there is an initialisation function that should be run as part of glibc which performs offset correction.

Is it possible that this is somehow missing? Note that we do NOT specify -nostdlib.

[RobMeades]

Also in case it is useful, we raised this with GCC in case they could be of help. They have so far suggested that a .reloc section should be present in the image to support this but there is no such section in our full disassembled output attached here.

[RobMeades]

Any views on this? We have concluded, so far, that it does not seem to be possible to compile C code under GCC that will work with ThreadX Modules. It would be really nice if you could tell us we're wrong :-).

[RobMeades]

\<prod \>

[goldscott]

Hi @RobMeades, I'm no longer on the Azure RTOS team, but I know Modules better than anyone. I don't recall ever testing something like this out (a global pointer to a global). I think either the linker settings need to be changed such that it will output a reloc section and/or the GOT needs to be fixed up. Perhaps the compiler libraries that do this aren't linked in because when a module starts, instead of calling cstartup, we call our own function: https://github.com/azure-rtos/threadx/blob/master/ports_module/cortex_m4/gnu/example_build/gcc_setup.s This may need to be modified to fix your issue.

[RobMeades]

@goldscott: thanks for responding, apologies for my delay in replying, I was away. Three (:-)) questions really:

a) am I correct that ThreadX Modules is intended to support compiling modules under GCC? What we're writing here is perfectly valid C code, it just doesn't work in a module. b) if (a) is true, I would hope that ThreadX/Azure/whoever would like to make this work; we have tried, see the analysis and links above, and concluded that it is actually impossible, at least without some script or other to parse output files and then modify the binary somehow or other. We'd like to be proved wrong :-). c) final thing - you are the person this issue is allocated to but, if you're not on the Azure RTOS team, I guess it should be assigned to someone who is. How do we do that?

Just as an FYI, we (u-blox) do pay ThreadZ/Azure/whoever for support.

[TiejunMS]

@RobMeades , since you have paid for Azure support, could you create a support ticket here? Thanks! https://azure.microsoft.com/en-us/support/create-ticket/

[RobMeades]

@TiejunMS: hi, and thanks for the swift response. I am able to log-in to that link but I end up in a very basic Microsoft Azure portal that I expect we all get as employees of a company that use Azure for day-to-day things: I can only ask questions about billing and subscriptions, there is no linkage whatsoever to ThreadX and that support contract.

How do I go about asking [an extremely detailed] technical question about the ThreadX [embedded firmware] through that mechanism?

[TiejunMS]

Here is the step:

1. Select issue type to "Technical" as shown below.

2. Choose "All services" and select "Service type" to "Azure RTOS"

[RobMeades]

Hmm, I don't get that option I'm afraid:

[RobMeades]

Just FYI, we've paid what was ThreadX for support for many many years (10's of them). In the last five years we have also adopted a lot of other Azure stuff for day to day business, so my account will take me to that latter. How do I get to the former?

[RobMeades]

@TiejunMS: just @,ing you in case you didn't receive the above.

[TiejunMS]

Let me check internally. Our team will be on holiday for the next few days. Please expect a delay on response. Thanks for your patience!

[RobMeades]

@TiejunMS: no worries, we will be the same, have a good few days off :-).

[RobMeades]

@TiejunMS: purely FYI, I've asked the guy who is technically responsible for ThreadX in our company to do the same thing and he gets the same options as me.

[RobMeades]

<bump >

[RobMeades]

@TiejunMS [adding your name in case you're not being notified of posts here].

[TiejunMS]

@RobMeades , we will look into this issue and let you know when there is update.

[RobMeades]

@TiejunMS: thanks very much, I will enquire again as to your thoughts in a few weeks.

[wangwen-4220]

Hi, @RobMeades, We create a module sample project with stmcube( 1.10.1) (https://www.st.com/en/development-tools/stm32cubeide.html) which is free to use.

Here is the project which uses the cortex-m4 gnu module port. the module can be loaded and run successfully. we also add similar test code with you mentioned above to see if the load address is correct. https://github.com/wangwen-4220/stm32f4-disco-module

Could you check if it is helpful to resolve your problem?

[RobMeades]

@wangwen-4220: thanks for doing this. I've taken a look at https://github.com/wangwen-4220/stm32f4-disco-module/blob/main/stm32cubeide/sample_threadx_module/sample_threadx_module.c but I can't see where it is doing the test we set out, i.e., within a ThreadX module, creating a global pointer to something and checking that the global pointer has been correctly initialised. So, for instance:

char buffer[100];
char* pBuffer = buffer;

int appModMain()
{
    printf("buffer address: %p, buffer address via pBuffer: %p \n", buffer, pBuffer);
    assert(buffer == pBuffer];
}

Can you show me where that test is now passing? Assuming you can, what did you do to fix it?

[RobMeades]

To be clear, I can see where you have set up the two global values, but I don't see a test.

[wangwen-4220]

We can monitor the global value from debugger windows:

you can build the project and see the gcc compiler and linker options from the console.

[RobMeades]

That's encouraging: of course we aren't using ST, but we can try those compiler options in our situation.

Just in case the debugger is doing something "clever", could you add the assert check and make sure that the executed code agrees?

Which version of GCC is this?

[wangwen-4220]

The git hub project is using the default version integrated in the IDE. I did not change it to use gcc 12.2 release 1, because there will be errors when others try to open the project but not installed the gcc 12.2 tool chain. Actually, I run the projects compiled by different verisons , it seems that there are no difference. I updated the tool chain to use gcc 12.2 at local where i installed gcc 12.2.

[RobMeades]

Good: just to humour me, could you add the assert check and make sure that the executed code agrees with what the debugger displays?

[RobMeades]

Actually, could you attach here a .txt file with the build options etc. in it, just as you get printed into the STM32Cube console window? They kind of trail off the end of the screen in your screenshot so we can't see all of them.

[wangwen-4220]

Good: just to humour me, could you add the assert check and make sure that the executed code agrees with what the debugger displays? Sure,

[wangwen-4220]

Actually, could you attach here a .txt file with the build options etc. in it, just as you get printed into the STM32Cube console window? They kind of trail off the end of the screen in your screenshot so we can't see all of them. cortex-m4-module.txt

[RobMeades]

Very good: let me point those compiler/linker options at the guys who can build the code and see if our outcome is different.

I will get back to you.

[RobMeades]

Some others have had a look at this thread now and ask if you could possibly try two more things:

1. Remove the volatile before pBuffer: we, and the third-party C code that we are building, would not normally/naturally apply the volatile modified to a global variable.
2. Set optimisation to O2: again, we would normally run with optimisation on. I guess you had O0 so that you can see the output more clearly in the debugger but the assert() check passing is the main thing for us.

Thanks!

[wangwen-4220]

after above 2 settings, we can see the results:

[RobMeades]

That's very useful, thanks, we are now investigating the build options at our end.

[RobMeades]

When you get a chance, could you possible attach either the scatter/load file you used, and/or the .lst file output by the build, so that we can see what base address you linked against?

[RobMeades]

FYI, the reason for asking is that, looking at module_manager_entry(), it seems to be calling txm_module_manager_in_place_load(), which will load the ThreadX Module into the same location as it was linked at. In that case there is no relocation, all addresses will inevitably be identical and no adjustment is required.

If that is correct, the real test, i.e. to match what we are doing, would be to call txm_module_manager_memory_load() instead and load the module into a location different to that it was linked at. It would be interesting to see if the assert() passes with that arrangement.

[wangwen-4220]

we will look into it and let you know when there is update.

[wangwen-4220]

Hi, @RobMeades , we updated codes to fix this problem, https://github.com/wangwen-4220/stm32f746-disco-module/tree/main/stm32cubeide/sample_threadx_module/cm7 Would you have a check if it works on your side? sample_threadx_module.ld:

gcc_setup.s:

[RobMeades]

Thanks for the update - we will try this ASAP.

[RobMeades]

Update: sorry for taking so long to get back to you on this - the team doing the work is in Lahore and are currently away for Eid. Before they went on holiday they confirmed that the fix definitely helps but it might not be the whole story, in that statically allocated items still seem to fail the test, however we still need to check that this is not because our linker-file setup is slightly different from yours.

Hopefully we will get back to you next week.

[alizaidiubx]

Hi @wangwen-4220 , We have tested your fix thoroughly, there are still some data sections that needs to be addressed in linker script and in gcc_setup.s. If we access const char * globally from a local scope it won't print that, instead our app got crashed testapp.c is below for reference.

include

include

include

/*

• Structure and enumeration definitons

• */

typedef enum { Field_1 = 1, Field_2, Field_3, Field_4, }ENUM_TEST_t;

typedef struct { char appName; int appId; char appDescription; }App_Test_t;

typedef struct { int test_A; char testChArray[100]; const char text; int test_B; ENUM_TEST_t T1; App_Test_t *app; }GCC_TEST_t;

/ Statically initializing struct globally /

GCC_TEST_t test = {10,"struct member testChArray","const char * text",27,NULL,NULL};

/ Function Declaration/ void displayStructMembers(GCC_TEST_t *t1);

int main() {

printf("\nApplication has started!\n");
printf("\nDisplaying Members of Structure.\n");
displayStructMembers(&test);
printf("Execution successful \n");
return 0;

}

void displayStructMembers(GCC_TEST_t t1) { printf("Reading Members of GCC_TEST Struct: test_A : %u\n",t1->test_A); printf("Reading Members of GCC_TEST Struct: testChArray : %s\n",t1->testChArray); printf("Reading Members of GCC_TEST Struct: const char text : %s\n",t1->text); printf("Reading Members of GCC_TEST Struct: test_B : %u\n",t1->test_B); }

We have analyzed the output map file, and found that there is a data section(.data.rel.local) which was not mapped in linker script. If we add this section in linker script and modify it to *(.data.rel .data.rel.local) our app wont crashes but the data of variables got corrupted. Screenshot is attached for reference

However our expected output should be:

Application has started!  Displaying member of Structure. Reading Members of GCC_TEST Struct: test_A :10 Reading Members of GCC_TEST Struct: testChArray : struct member testChArray Reading Members of GCC_TEST Struct: const char : const char text Reading Members of GCC_TEST Struct: test_B :27 Execution successful

[wangwen-4220]

we will look into it and let you know when there is update. initializing the pointer member in the function will temporality avoid such errors.

[alizaidiubx]

Hi @wangwen-4220 , Its been a long time , is there any update on the above issue ?

[wangwen-4220]

Hi @alizaidiubx, Apologize for long waiting, we have no update on generating a sample project yet. but here is a suggestion for solving the problem: add linker option"-pie" and generate a type called "R_ARM_RELATIVE " in elf file. this will help to resolve the relocation of RO data which is the initialized value of pointer.

[alizaidiubx]

Hi @wangwen-4220 , As suggested i have tried building the application by adding "-pie" linker flag and got some linker error :

: note: this is the location of the previous definition [123/123] Linking C executable hello_world.axf FAILED: cmd.exe /C "cd . && C:\PROGRA~2\GNUARM~1\102021~1.10\bin\AR19DD~1.EXE -g --specs=nosys.specs --specs=nano.specs -mcpu=cortex-m7 -mfloat-abi=hard -mfpu=fpv5-d16 -mthumb -fpic -fno-plt -mno-pic-data-is-text-relative -msingle-pic-base -mcpu=cortex-m7 -mfloat-abi=hard -pie -mfpu=fpv5-d16 -mthumb -fpic -fno-plt -mno-pic-data-is-text-relative -msingle-pic-base -e _txm_module_thread_shell_entry -T C:/Users/azai/Documents/GitHub/dev_gcc_qazi_1/hello_world/../config/GNU/txm_module.ld -o hello_world.axf -Xlinker -Map=output.map CMakeFiles/hello_world.dir/src/hello_world.c.obj CMakeFiles/hello_world.dir/C_/Users/azai/Documents/GitHub/dev_gcc_qazi_1/config/GNU/txm_preamble.S.obj CMakeFiles/hello_world.dir/C_/Users/azai/Documents/GitHub/dev_gcc_qazi_1/config/GNU/gcc_setup.s.obj -o hello_world.axf libubxlib.a ../../cell_ucpu_sdk/r5/lib/lib_cell_ucpu_sdk_gcc.a libmbedtlslib.a ../../cell_ucpu_sdk/r5/threadX5.8/lib/lib_txm_gcc.a && cmd.exe /C "cd /D C:\Users\azai\Documents\GitHub\dev_gcc_qazi_1\hello_world\build && "C:\Program Files (x86)\GNU Arm Embedded Toolchain\10 2021.10\bin\arm-none-eabi-objcopy.exe" -Obinary C:/Users/azai/Documents/GitHub/dev_gcc_qazi_1/hello_world/build/hello_world.axf hello_world.bin"" **c:/progra~2/gnuarm~1/102021~1.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld.exe: section .dynamic LMA [00000000,000000b7] overlaps section .dynsym LMA [00000000,0000011f] c:/progra~2/gnuarm~1/102021~1.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld.exe: hello_world.axf: error: PHDR segment not covered by LOAD segment** collect2.exe: error: ld returned 1 exit status ninja: build stopped: subcommand failed. I have attached output.map file for reference [output.zip](https://github.com/azure-rtos/threadx/files/12657143/output.zip)

[alizaidiubx]

Hi @wangwen-4220 , is there any progress on the above issue? Also i have reported some linker errors on your suggested fix

[wangwen-4220]

Hi, @alizaidiubx ,how about add linker option "--no-dynamic-linker"?

[alizaidiubx]

Hi @wangwen-4220 , I tried by adding the above linker option in my CMakeLists.txt linker flag options, but still getting the below linker error: [123/123] Linking C executable hello_world.axf FAILED: cmd.exe /C "cd . && C:\PROGRA~2\GNUARM~1\102021~1.10\bin\AR19DD~1.EXE -g --specs=nosys.specs --specs=nano.specs -mcpu=cortex-m7 -mfloat-abi=hard -mfpu=fpv5-d16 -mthumb -fpic -fno-plt -mno-pic-data-is-text-relative -msingle-pic-base -mcpu=cortex-m7 -pie -mfloat-abi=hard -mfpu=fpv5-d16 -mthumb -fpic -fno-plt -mno-pic-data-is-text-relative -msingle-pic-base -e _txm_module_thread_shell_entry -T C:/gcc_exm/hello_world/../config/GNU/txm_module.ld -o hello_world.axf -Xlinker -Map=output.map,--no-dynamic-linker CMakeFiles/hello_world.dir/src/hello_world.c.obj CMakeFiles/helloworld.dir/C/gcc_exm/config/GNU/txm_preamble.S.obj CMakeFiles/helloworld.dir/C/gcc_exm/config/GNU/gcc_setup.s.obj -o hello_world.axf libubxlib.a ../../cell_ucpu_sdk/r5/lib/lib_cell_ucpu_sdk_gcc.a libmbedtlslib.a ../../cell_ucpu_sdk/r5/threadX5.8/lib/lib_txm_gcc.a && cmd.exe /C "cd /D C:\gcc_exm\hello_world\build && "C:\Program Files (x86)\GNU Arm Embedded Toolchain\10 2021.10\bin\arm-none-eabi-objcopy.exe" -Obinary C:/gcc_exm/hello_world/build/hello_world.axf hello_world.bin"" c:/progra~2/gnuarm~1/102021~1.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld.exe: section .dynamic LMA [00000000,000000b7] overlaps section .dynsym LMA [00000000,0000011f] c:/progra~2/gnuarm~1/102021~1.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld.exe: hello_world.axf: error: PHDR segment not covered by LOAD segment collect2.exe: error: ld returned 1 exit status ninja: build stopped: subcommand failed.

Linker Flags: "-mcpu=cortex-m7 -pie -mfloat-abi=hard -mfpu=fpv5-d16 -mthumb -fpic -fno-plt -mno-pic-data-is-text-relative -msingle-pic-base -e _txm_module_thread_shell_entry -T ${CMAKE_CURRENT_LIST_DIR}/../config/GNU/txm_module.ld -o ${TARGET_NAME}.axf -Xlinker -Map=output.map,--no-dynamic-linker"

[wangwen-4220]

@alizaidiubx, Not quite sure why still throw out errors. It works when I test using the below linker command,

arm-none-eabi-ld -A cortex-m7 -T sample_threadx_module.ld -pie --no-dynamic-linker txm_module_preamble.o gcc_setup.o sample_threadx_module.o -e _txm_module_thread_shell_entry txm.a -o sample_threadx_module.axf -M > sample_threadx_module.map

[wangwen-4220]

Hi, @alizaidiubx we update sample project (STM32Cube IDE 1.12.1)mostly in 3 parts:1. add -pie linker option. 2. add rel.dyn section relocation in gcc_setup.s. 3. remove last fix in gcc_setup.s https://github.com/wangwen-4220/stm32f746-disco-module/blob/main/stm32cubeide/sample_threadx_module/cm7/gcc_setup.s

the serial port output looks like: