RISCV ULP long wakeup time during deep sleep (IDFGH-13552)

[2opremio]

Answers checklist.

• [X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• [X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• [X] I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

The RISCV ULP in my S3 chip takes an unexpectedly long time (~8ms) to wake up between sleep periods while the main CPU is in deep sleep.

While the Main CPU is awake, the wake up time is 16 times faster (around 0.5ms).

The wake up time during deep sleep seems to depend on the frequency of the RTC clock. In my case I am using a 32K external oscillator (it's linearly shorter when using the internal RC_SLOW oscillator).

To measure the times I created a simple ULP program which goes to sleep immediately after toggling an output GPIO pin.

I measured the time between toggles with a logic analyzer (a Nordic PPK2) which also tells me when the system is in deep sleep.

Here are a couple of screenshots of a transition to deep sleep. The bottom line includes the GPIO toggles.

While the main CPU is awake, I measure 0.540ms between ULP wakeups:

While the system is in deep sleep, I measure ~7.7ms:

Here is the full code:

main.c

#include "esp_log.h"
#include "esp_sleep.h"
#include "ulp_riscv.h"
#include "driver/rtc_io.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"

#define DEBUG_GPIO_NUM GPIO_NUM_5
extern const uint8_t ulp_main_bin_start[] asm("_binary_ulp_main_bin_start");
extern const uint8_t ulp_main_bin_end[] asm("_binary_ulp_main_bin_end");

void app_main(void)
{
    vTaskDelay(pdMS_TO_TICKS(1000));

    esp_sleep_wakeup_cause_t cause = esp_sleep_get_wakeup_cause();

    switch (cause) {
    case ESP_SLEEP_WAKEUP_TIMER:
        ESP_LOGI("test", "Woke up because of timer");
        break;
    default:
        ESP_LOGI("test", "Initializing ULP and GPIO");
        ESP_ERROR_CHECK(rtc_gpio_init(DEBUG_GPIO_NUM));
        ESP_ERROR_CHECK(rtc_gpio_set_direction(DEBUG_GPIO_NUM, RTC_GPIO_MODE_OUTPUT_ONLY));
        esp_err_t err = ulp_riscv_load_binary(ulp_main_bin_start, (ulp_main_bin_end - ulp_main_bin_start));
        ESP_ERROR_CHECK(err);
        // Wakeup immediately
        ESP_ERROR_CHECK(ulp_set_wakeup_period(0, 0));
        ESP_ERROR_CHECK(ulp_riscv_run());
    }

    ESP_ERROR_CHECK(esp_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_ON));
    ESP_ERROR_CHECK(esp_sleep_enable_timer_wakeup(5 * 1000 * 1000));
    esp_deep_sleep_start();
}

ulp_main.c

#include "ulp_riscv_gpio.h"

#define DEBUG_GPIO_NUM GPIO_NUM_5

int main(void)
{
    static bool debug_gpio_output_value = 0;
    debug_gpio_output_value = !debug_gpio_output_value;
    ulp_riscv_gpio_output_level(DEBUG_GPIO_NUM, debug_gpio_output_value);
    return 0;
}

And my sdkconfig: sdkconfig.gz

I reproduced this with ESP IDF 5.3.

[2opremio]

@sudeep-mohanty I think this one could be in your ballpark.

[2opremio]

BTW, when using the internal oscillator (RC_SLOW) as the RTC clock source the time between wakeups is:

• around 0.130ms while the main CPU is active
• around 1.8ms during deep sleep.

[ESP-Marius]

@2opremio Thanks for reporting this. Probably the HW does need some extra time to boot up everything when in deep sleep, but I agree that this difference seems big.

We'll check with the digital team and see what the expected values are here and get back to you.

[2opremio]

I really hope there is a way to improve it, since i need to sample every 6ms with the ULP and I would like to sleep between samples to save energy.

[2opremio]

Also, it would be good to get some explanation as to why and how the ULP boot/halt time depends on the RTC clock.

The TRM doc seems to suggest it only depends on RC_FAST:

Clocked with 17.5 MHz RTC_FAST_CLK

[boarchuz]

This is likely controlled by the various delays in rtc_cntl_reg.h specified in RTC slow clock cycles.

For example, RTC_CNTL_CK8M_WAIT‎ and RTC_CNTL_ULPCP_TOUCH_START_WAIT.

I know from experience that these are set fairly conservatively in ESP-IDF and can be reduced to speed up ULP start time.

If the RTC slow clock in use is slower than the default, I guess it stands to reason that these could be reduced proportionally to ensure consistent behaviour.

[2opremio]

For example, RTC_CNTL_CK8M_WAIT‎ and RTC_CNTL_ULPCP_TOUCH_START_WAIT.

I did try dividing the value of those by 4 (accordingly to the clock frequency), but they didn't make a measurable difference.

i.e. this didn't make a difference:

--- a/main/main.c
+++ b/main/main.c
@@ -1,5 +1,6 @@
 #include <esp_clk_tree.h>
 #include <soc/rtc.h>
+#include <soc/rtc_cntl_reg.h>

 #include "esp_log.h"
 #include "esp_sleep.h"
@@ -59,5 +60,9 @@ void app_main(void)
     print_slow_clock_info();
     ESP_ERROR_CHECK(esp_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_ON));
     ESP_ERROR_CHECK(esp_sleep_enable_timer_wakeup(5 * 1000 * 1000));
+    REG_SET_FIELD(RTC_CNTL_TIMER2_REG, RTC_CNTL_ULPCP_TOUCH_START_WAIT, 4);
+    ESP_LOGI("test", "TOUCH_START_WAIT: %lu",  REG_GET_FIELD(RTC_CNTL_TIMER2_REG, RTC_CNTL_ULPCP_TOUCH_START_WAIT));;
+    REG_SET_FIELD(RTC_CNTL_TIMER1_REG, RTC_CNTL_CK8M_WAIT, 1);
+    ESP_LOGI("test", "CK8M_WAIT: %lu",  REG_GET_FIELD(RTC_CNTL_TIMER1_REG, RTC_CNTL_CK8M_WAIT));
     esp_deep_sleep_start();
 }

Also, I believe that ULPCP_TOUCH_START_WAIT only impacts the FSM ULP and not the RISCV ULP.

[2opremio]

@boarchuz you were actually right, I just wasn't changing the values at the right place. The idf sleep code overrode my changes.

After applying this patch, the time goes down to roughly the same as with the internal oscillator:

diff --git a/components/esp_hw_support/port/esp32s3/include/soc/rtc.h b/components/esp_hw_support/port/esp32s3/include/soc/rtc.h
--- a/components/esp_hw_support/port/esp32s3/include/soc/rtc.h
+++ b/components/esp_hw_support/port/esp32s3/include/soc/rtc.h
@@ -96,7 +96,7 @@ extern "C" {

 #define RTC_CNTL_PLL_BUF_WAIT_DEFAULT  20
 #define RTC_CNTL_XTL_BUF_WAIT_DEFAULT  100
-#define RTC_CNTL_CK8M_WAIT_DEFAULT     20       // Equivalent macro as `CLK_LL_RC_FAST_WAIT_DEFAULT`
+#define RTC_CNTL_CK8M_WAIT_DEFAULT     5        // Equivalent macro as `CLK_LL_RC_FAST_WAIT_DEFAULT`
 #define RTC_CK8M_ENABLE_WAIT_DEFAULT   5        // Equivalent macro as `CLK_LL_RC_FAST_ENABLE_WAIT_DEFAULT`

 /* Various delays to be programmed into power control state machines */
@@ -109,8 +109,8 @@ extern "C" {
 #define RTC_CNTL_CK8M_DFREQ_DEFAULT 100
 #define RTC_CNTL_SCK_DCAP_DEFAULT   255

-#define RTC_CNTL_ULPCP_TOUCH_START_WAIT_IN_SLEEP    (0xFF)
-#define RTC_CNTL_ULPCP_TOUCH_START_WAIT_DEFAULT     (0x10)
+#define RTC_CNTL_ULPCP_TOUCH_START_WAIT_IN_SLEEP    (0x40)
+#define RTC_CNTL_ULPCP_TOUCH_START_WAIT_DEFAULT     (0x04)

Now, what's the reason for having a different ULPCP_TOUCH_START_WAIT during sleep?

Also, what does CK8M_WAIT exactly do?

From the S3 TRM:

Sets the FOSC waiting cycle

What does waiting cycle mean? Is the time to wait for the Fast oscillator to start? When is it started/stopped? Changing its value doesn't seem to help.

It's pretty frustrating to deal with so many poorly documented registers.

@ESP-Marius those parameters should probably be configurable and adapted to the chosen slow clock instead of hardcoded (there doesn't seem to be an API to change them).

[boarchuz]

@2opremio My understanding is that it takes a moment for some blocks to power up, and these timings ensure everything has time to stabilise before it is used. The 8MHz oscillator will be powered down in deep sleep, for example, but the ULP coprocessor is clocked from this, so the ULP must wait until it's ready, else weird things can happen (https://github.com/espressif/esp-idf/issues/8048).

[ESP-Marius]

I think @boarchuz (thanks for helping with solving the issue!) is correct, but unfortunately I dont have any more information available in the TRM either.

there doesn't seem to be an API to change them)

This is probably on purpose since there isnt really any way for users to know what reasonable values for these are. I'll continue following up on this internally and see if I can find someone that can help determine what these actually should be when using an external oscillator.

[2opremio]

My understanding is that it takes a moment for some blocks to power up, and these timings ensure everything has time to stabilise before it is used. The 8MHz oscillator will be powered down in deep sleep, for example

Yep, that makes sense, but doesn't explain why is set so high during deep sleep.

BTW, answering my own question:

What does waiting cycle mean? Is the time to wait for the Fast oscillator to start?

Yep, that seems to be the case.

https://github.com/espressif/esp-idf/blob/c9df77efbf871d4c3ae9fb828778ff8c4ab36804/components/esp_hw_support/port/esp32c3/include/soc/rtc.h#L726

[2opremio]

BTW, setting different values during deep_sleep and awake makes scheduling using the timer a lot more difficult. My scheduling code has been greatly simplified after unifying them.

Also, the default ULPCP_TOUCH_START_WAIT value set by idf during deep sleep (~2ms when using the internal osciallator) seems to be very very conservative. After reducing it, the power consumption of the ULP has also greatly reduced (since it reduces the time the ULP is powered up idle). The main use-case of the ULP is saving power, so using a high default doesn't make sense.

It would be good to understand how far we can push it safely.

So far I have set it to around 250us (which is roughly what @boarchuz suggested at https://github.com/espressif/esp-idf/issues/8048#issuecomment-990350013 )

[2opremio]

I've created this PR to correct the default values: https://github.com/espressif/esp-idf/pull/14453

I still would like to know why there's a difference when deep-sleeping