For binary built for esp32 target xtensa-esp32-elf-objdump produces incorrect disassembly, while xtensa-esp32s3-elf-objdump is correct (GCC-247)

michael-sayapin commented 2 years ago

Environment

Module or chip used: ESP32-WROOM-32
IDF version (run git describe --tags to find it): v4.4.1
Build System: idf.py (ninja)
Compiler version (run xtensa-esp32-elf-gcc --version to find it): xtensa-esp32-elf-gcc (crosstool-NG esp-2021r1) 8.4.0
Operating System: Linux
Power Supply: external 5V

Problem Description

I'm building the .elf file with idf.py and following sdkconfig:

CONFIG_IDF_CMAKE=y
CONFIG_IDF_TARGET_ARCH_XTENSA=y
CONFIG_IDF_TARGET="esp32"
CONFIG_IDF_TARGET_ESP32=y
CONFIG_IDF_FIRMWARE_CHIP_ID=0x0000
CONFIG_ESP32_REV_MIN_0=y
# CONFIG_ESP32_REV_MIN_1 is not set
# CONFIG_ESP32_REV_MIN_2 is not set
# CONFIG_ESP32_REV_MIN_3 is not set
CONFIG_ESP32_REV_MIN=0

Here is the real code of one of the tasks:

#define WITH_MUTEX(what, mutex, timeout_tick)                             \
  ({                                                                      \
    __label__ error_macros_with_mutex_exit;                               \
    errors_Error __error = {0};                                           \
    if (pdTRUE != xSemaphoreTake(mutex, (timeout_tick))) {                \
      __error = ERROR(errors_Error_system_tag,                            \
                      errors_System_SYSTEM_TAKE_MUTEX_FAILED, "%s:%u",    \
                      __func__, __LINE__);                                \
      goto error_macros_with_mutex_exit;                                  \
    }                                                                     \
    MDF_LOGD("Mutex %s taken in %s, before %s", #mutex, __func__, #what); \
    __error = (what);                                                     \
    if (pdTRUE != xSemaphoreGive(mutex)) {                                \
      /* highly unlikely, since mutex is successfully taken */            \
      protocol_utils_destroy_error(__error);                              \
      __error = ERROR(errors_Error_system_tag,                            \
                      errors_System_SYSTEM_TAKE_MUTEX_FAILED,             \
                      "xSemaphoreGive %s:%u", __func__, __LINE__);        \
      goto error_macros_with_mutex_exit;                                  \
    }                                                                     \
    MDF_LOGD("Mutex %s given in %s, after %s", #mutex, __func__, #what);  \
  error_macros_with_mutex_exit:                                           \
    __error;                                                              \
  })

...

static void xxx_task(void *pvParameters) {
  char *task_name = pcTaskGetTaskName(NULL);
  MDF_LOGD("%s is running (priority %d)", task_name, uxTaskPriorityGet(NULL));
  errors_Error error = {0};
  while (true) {
    error = WITH_MUTEX(xxx_do_something(), g_mutex, portMAX_DELAY);
    if (0 != error.which_code) {
      protocol_utils_destroy_error(error);
      vTaskDelay(pdMS_TO_TICKS(XXX_TASK_SLEEP_ERROR_MS));
      continue;
    }
    MDF_LOGD("%s free stack memory: %d", task_name,
             uxTaskGetStackHighWaterMark(NULL));
    vTaskDelay(pdMS_TO_TICKS(XXX_TASK_SLEEP_MS));
  }
  MDF_LOGW("%s exited", task_name);
  vTaskDelete(NULL);
}

Expected Behavior

xtensa-esp32s3-elf-objdump -d produces correct disassembly:

400eb64c <xxx_task>:
400eb64c:   00c136          entry   a1, 96
400eb64f:   00a0a2          movi    a10, 0
400eb652:   a9ffa5          call8   4009564c <pcTaskGetName>
400eb655:   ffafb2          movi    a11, -1
400eb658:   9a8b81          l32r    a8, 400d2084 <_stext+0x2064>
400eb65b:   08a8        l32i.n  a10, a8, 0
400eb65d:   a86225          call8   40093c80 <xQueueSemaphoreTake>
400eb660:   201a26          beqi    a10, 1, 400eb684 <xxx_task+0x38>
400eb663:   9a8ba1          l32r    a10, 400d2090 <_stext+0x2070>
400eb666:   55a2b2          movi    a11, 0x255
400eb669:   11b9        s32i.n  a11, a1, 4
400eb66b:   01a9        s32i.n  a10, a1, 0
400eb66d:   9a86f1          l32r    a15, 400d2088 <_stext+0x2068>
400eb670:   9a87e1          l32r    a14, 400d208c <_stext+0x206c>
400eb673:   cd0c        movi.n  a13, 12
400eb675:   dc0c        movi.n  a12, 13
400eb677:   4ab3e5          call8   401361b4 <__error_create>
400eb67a:   a1a9        s32i.n  a10, a1, 40
400eb67c:   b1b9        s32i.n  a11, a1, 44
400eb67e:   c1c9        s32i.n  a12, a1, 48
400eb680:   001006          j   400eb6c4 <xxx_task+0x78>
400eb683:   00              .byte 00
400eb684:   ffc8a5          call8   400eb310 <xxx_do_something>
400eb687:   a1a9        s32i.n  a10, a1, 40
400eb689:   b1b9        s32i.n  a11, a1, 44
400eb68b:   c1c9        s32i.n  a12, a1, 48
400eb68d:   0d0c        movi.n  a13, 0
400eb68f:   0dcd        mov.n   a12, a13
400eb691:   0dbd        mov.n   a11, a13
400eb693:   9a7c81          l32r    a8, 400d2084 <_stext+0x2064>
400eb696:   08a8        l32i.n  a10, a8, 0
400eb698:   a80425          call8   400936dc <xQueueGenericSend>
400eb69b:   251a26          beqi    a10, 1, 400eb6c4 <xxx_task+0x78>
400eb69e:   a1a8        l32i.n  a10, a1, 40
400eb6a0:   b1b8        l32i.n  a11, a1, 44
400eb6a2:   c1c8        l32i.n  a12, a1, 48
400eb6a4:   4cdf65          call8   4013849c <protocol_utils_destroy_error>
400eb6a7:   9a7aa1          l32r    a10, 400d2090 <_stext+0x2070>
400eb6aa:   55a2b2          movi    a11, 0x255
400eb6ad:   11b9        s32i.n  a11, a1, 4
400eb6af:   01a9        s32i.n  a10, a1, 0
400eb6b1:   9a78f1          l32r    a15, 400d2094 <_stext+0x2074>
400eb6b4:   9a76e1          l32r    a14, 400d208c <_stext+0x206c>
400eb6b7:   cd0c        movi.n  a13, 12
400eb6b9:   dc0c        movi.n  a12, 13
400eb6bb:   4aafa5          call8   401361b4 <__error_create>
400eb6be:   a1a9        s32i.n  a10, a1, 40
400eb6c0:   b1b9        s32i.n  a11, a1, 44
400eb6c2:   c1c9        s32i.n  a12, a1, 48
400eb6c4:   0a2182          l32i    a8, a1, 40
400eb6c7:   0b21a2          l32i    a10, a1, 44
400eb6ca:   c198        l32i.n  a9, a1, 48
400eb6cc:   7189        s32i.n  a8, a1, 28
400eb6ce:   81a9        s32i.n  a10, a1, 32
400eb6d0:   9199        s32i.n  a9, a1, 36
400eb6d2:   f48080          extui   a8, a8, 0, 16
400eb6d5:   089c        beqz.n  a8, 400eb6e9 <xxx_task+0x9d>
400eb6d7:   71a8        l32i.n  a10, a1, 28
400eb6d9:   81b8        l32i.n  a11, a1, 32
400eb6db:   09cd        mov.n   a12, a9
400eb6dd:   4cdbe5          call8   4013849c <protocol_utils_destroy_error>
400eb6e0:   d0a7a2          movi    a10, 0x7d0
400eb6e3:   a8f2a5          call8   4009460c <vTaskDelay>
400eb6e6:   ffdac6          j   400eb655 <xxx_task+0x9>
400eb6e9:   95cfa1          l32r    a10, 400d0e28 <_stext+0xe08>
400eb6ec:   a8f1e5          call8   4009460c <vTaskDelay>
400eb6ef:   ffd886          j   400eb655 <xxx_task+0x9>

Actual Behavior

xtensa-esp32-elf-objdump -d produces this disassembly:

400eb64c <xxx_task>:
400eb64c:       00c136          entry   a1, 96
400eb64f:       00a0a2          movi    a10, 0
400eb652:       a9ffa5          call8   4009564c <pcTaskGetName>
400eb655:       ffafb2          movi    a11, -1
400eb658:       9a8b81          l32r    a8, 400d2084 <_stext+0x2064>
400eb65b:       08a8            l32i.n  a10, a8, 0
400eb65d:       a86225          call8   40093c80 <xQueueSemaphoreTake>
400eb660:       201a26          beqi    a10, 1, 400eb684 <xxx_task+0x38>
400eb663:       9a8ba1          l32r    a10, 400d2090 <_stext+0x2070>
400eb666:       55a2b2          movi    a11, 0x255
400eb669:       11b9            s32i.n  a11, a1, 4
400eb66b:       01a9            s32i.n  a10, a1, 0
400eb66d:       9a86f1          l32r    a15, 400d2088 <_stext+0x2068>
400eb670:       9a87e1          l32r    a14, 400d208c <_stext+0x206c>
400eb673:       cd0c            movi.n  a13, 12
400eb675:       dc0c            movi.n  a12, 13
400eb677:       4ab3e5          call8   401361b4 <__error_create>
400eb67a:       a1a9            s32i.n  a10, a1, 40
400eb67c:       b1b9            s32i.n  a11, a1, 44
400eb67e:       c1c9            s32i.n  a12, a1, 48
400eb680:       001006          j       400eb6c4 <xxx_task+0x78>
400eb683:       c8a500          lsi     f0, a5, 0x320
400eb686:       ff              .byte 0xff
400eb687:       a1a9            s32i.n  a10, a1, 40
400eb689:       b1b9            s32i.n  a11, a1, 44
400eb68b:       c1c9            s32i.n  a12, a1, 48
400eb68d:       0d0c            movi.n  a13, 0
400eb68f:       0dcd            mov.n   a12, a13
400eb691:       0dbd            mov.n   a11, a13
400eb693:       9a7c81          l32r    a8, 400d2084 <_stext+0x2064>
400eb696:       08a8            l32i.n  a10, a8, 0
400eb698:       a80425          call8   400936dc <xQueueGenericSend>
400eb69b:       251a26          beqi    a10, 1, 400eb6c4 <xxx_task+0x78>
400eb69e:       a1a8            l32i.n  a10, a1, 40
400eb6a0:       b1b8            l32i.n  a11, a1, 44
400eb6a2:       c1c8            l32i.n  a12, a1, 48
400eb6a4:       4cdf65          call8   4013849c <protocol_utils_destroy_error>
400eb6a7:       9a7aa1          l32r    a10, 400d2090 <_stext+0x2070>
400eb6aa:       55a2b2          movi    a11, 0x255
400eb6ad:       11b9            s32i.n  a11, a1, 4
400eb6af:       01a9            s32i.n  a10, a1, 0
400eb6b1:       9a78f1          l32r    a15, 400d2094 <_stext+0x2074>
400eb6b4:       9a76e1          l32r    a14, 400d208c <_stext+0x206c>
400eb6b7:       cd0c            movi.n  a13, 12
400eb6b9:       dc0c            movi.n  a12, 13
400eb6bb:       4aafa5          call8   401361b4 <__error_create>
400eb6be:       a1a9            s32i.n  a10, a1, 40
400eb6c0:       b1b9            s32i.n  a11, a1, 44
400eb6c2:       c1c9            s32i.n  a12, a1, 48
400eb6c4:       0a2182          l32i    a8, a1, 40
400eb6c7:       0b21a2          l32i    a10, a1, 44
400eb6ca:       c198            l32i.n  a9, a1, 48
400eb6cc:       7189            s32i.n  a8, a1, 28
400eb6ce:       81a9            s32i.n  a10, a1, 32
400eb6d0:       9199            s32i.n  a9, a1, 36
400eb6d2:       f48080          extui   a8, a8, 0, 16
400eb6d5:       089c            beqz.n  a8, 400eb6e9 <xxx_task+0x9d>
400eb6d7:       71a8            l32i.n  a10, a1, 28
400eb6d9:       81b8            l32i.n  a11, a1, 32
400eb6db:       09cd            mov.n   a12, a9
400eb6dd:       4cdbe5          call8   4013849c <protocol_utils_destroy_error>
400eb6e0:       d0a7a2          movi    a10, 0x7d0
400eb6e3:       a8f2a5          call8   4009460c <vTaskDelay>
400eb6e6:       ffdac6          j       400eb655 <xxx_task+0x9>
400eb6e9:       95cfa1          l32r    a10, 400d0e28 <_stext+0xe08>
400eb6ec:       a8f1e5          call8   4009460c <vTaskDelay>
400eb6ef:       ffd886          j       400eb655 <xxx_task+0x9>

Notice that the call to xxx_do_something is missing.

igrr commented 2 years ago

@michael-sayapin could you please attach the ELF file which has this issue?

michael-sayapin commented 2 years ago

@michael-sayapin could you please attach the ELF file which has this issue?

Let me check with the team, as it is a commercial product and I'm not sure I can open the ELF file. I will try to reproduce with a minimal example.

igrr commented 2 years ago

Another option for you to consider is sending it to me (ivan at espressif).

igrr commented 2 years ago

Thank you for sharing the ELF file @michael-sayapin.

Upon a brief look, it seems that there are two issues here:

ESP32 and ESP32-S3 builds of objdump interpret the padding "0x00" byte after the unconditional jump differently. ESP32 version of objdump interprets it along with two other bytes as an "lsi" instruction. The ESP32-S3 version of objdump interprets it as a padding byte and correctly decodes the next 3 bytes as "call8". In theory, ESP32-S3 version should behave the same way ESP32 version does (i.e. incorrectly).

We will investigate why this happens.
Currently, ESP-IDF linker scripts don't add special ELF sections which contain information about such padding bytes, and which can used by the disassembler to avoid issues like this. This fact has been noticed recently by @o-marshmallow and there is a patch available internally for this issue. We haven't merged the patch yet, still investigating one issue which has surfaced when we ran our test suite.

I'm attaching the patch here for you to try. Please try applying this patch to ESP-IDF, link the application again and see if the ESP32 disassembler starts interpreting your ELF file correctly. 17268.patch.txt

michael-sayapin commented 2 years ago

Finally had a moment to try the patch. Turns out, the builder we use is based on IDF 4.2, so I had to modify the patch quite a bit to apply. In the end it applied fully.

However, the disassembly result is the same, and seems like it made it a bit worse.

PS: Both non-patched and patched versions contain correct calls if disassembled with -r flag (relocation entries).

espressif / crosstool-NG