esp32s3 curve-fitting calibration with continuou ADC reading gives about 10mV difference from the real value (IDFGH-13275)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.1.2-534-gbbd8d13f93

Espressif SoC revision.

ESP32-S3 (QFN56) (revision v0.1)

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

Development Kit.

ESP32-S3-WROOM-1

Power Supply used.

USB

What is the expected behavior?

The calibrated value should be as close to the actual voltage as possible.

What is the actual behavior?

It has about 10mV difference from the actual measurement on average. esp32s3-adc-comparison ADC measurements.ods

Please see the attached spreadsheet for all the measurements.

Steps to reproduce.

Setup a voltage source with multimeter. Feed the voltage into the ADC inputs.
Use the example in examples/peripherals/adc/continuous_read. Change the attenuation level to ADC_ATTEN_DB_6. Initialise the ADC calibration scheme in continuous_adc_init and print the conversion in the app_main while loop.
Read the multi-meter measurements and the "calibrated" value on the console.

Debug Logs.

No response

More Information.

I have got over 20 devices and all behaves the same way.

This is expected behavior and is defined here on pg 58.

This is a good repo that explores the issue, and provides a solution using a LUT.

FYI the esp-idf calibration code is not good.

If you look under esp-idf/components/efuse/{module type}/esp_efuse_rtc_calib.c you'll see many of the values are hardcoded. Specifically out_vol_mv = 850;

IMO the LUT is not a valid choice for an s3 since it doesn't have a DAC you can use to calibrate. I recommend you do some RMS averaging to get the bits up to 14. This will reduce some of the error.

If you need even more accuracy than that, you'll need to lower your attenuation, or rewrite the calibration function to hold more decimals.

/*
    Calibration voltage is 850mV
    Calibration digital is 1003
    Max ADC raw is 409599 (4095.99)(14bit)

    850/1003 = 0.847457627
    0.847457627 * 409599          = 3471178 (3471.178)
    (847457 * 409599)  / 10000000 = 3471175 (3471.175)
    (8474576 * 409599) / 10000000 = 3471178 (3471.178) (No reason to keep more decimals)

*/
#define ADC_CALIBRATION_COEFF_MULTIPLIER 10000000

/*
    Raw values are up to 4095, but through multisampling,
    you can get 4095.50 etc.
    This is 40950. After multiplying with coeffA, you get
    3470338872000
    3470338872000 / 1000 = 3470338872 (3470(mV).338(uV).872(pV))

*/
#define ADC_RAW_14BIT_TO_12BIT 1000

/*
    After calculating ADC to mV you get,
    3470338872 (3470(mV).338(uV).872(pV))

    this value is likely not accurate so we turf the pV
    3470338872 / 1000 = 3470338872 (3470(mV).338(uV))
*/
#define ADC_CALIBRATION_PV_TO_UV 1000

if (esp_efuse_rtc_calib_get_cal_voltage(1, ADC_UNIT_1, ADC_ATTEN_DB_0, &u32CalibrationDigital, &u32CalibrationVoltage) != ESP_OK) { ESP_LOGI(DEBUG_TAG, "esp_efuse_rtc_calib_get_cal_voltage. "); return !ESP_OK; }
coeff_a = ((uint64_t)u32CalibrationVoltage*(uint64_t)ADC_CALIBRATION_COEFF_MULTIPLIER) / u32CalibrationDigital;
if (coeff_a == 0) { ESP_LOGI(DEBUG_TAG, "Invalid Calibration Coefficent. "); return !ESP_OK; }

Here's an example on how to start your calibration function

static void IRAM_ATTR _{}_Cali_Raw_To_Voltage(int raw, uint32_t *mV)  {
    /*
    Max Raw = 409500 (4095.00)
    CoeffA = 8474576
    409500 * 8474576 = 3470338872000
    3470338872000 / 1000 = 3469491414 (3469(mV).491(uV).414(pV))
    3469491414 / 1000 = 3469491 (3469(mV).491(uV))
    */

    uint64_t v_cali_2 = (uint64_t)raw * (*coeff_a);
    v_cali_2 /= ADC_RAW_14BIT_TO_12BIT;
    v_cali_2 /= ADC_CALIBRATION_PV_TO_UV;

   ...

Also, a handheld meter is not accurate enough. I have a few fluke meters here that saw similar results to you. I couldn't accurately do verification until I used a 16bit oscilloscope. A 14bit scope may be enough.

espressif / esp-idf