mec15xxevb_assy6853: 2 GPIO test failures

[yerabolu]

Describe the bug Multiple tests are failing on mec15xxevb_assy6853 board under tests/boards/mec15xxevb_assy6853/qspi/ , i2c_api/ and tests/drivers/gpio/gpio_api_1pin/

To Reproduce Steps to reproduce the behavior:

1. twister --hardware-map ~/mec1.file --device-testing -T tests/boards/mec15xxevb_assy6853/qspi/ , i2c_api and
2. See error

Expected behavior All the tests should pass

Logs and console output

*** Booting Zephyr OS build zephyr-v2.5.0-3177-gfaf68fb213bc  ***
Running test suite test_spi
===================================================================
START - test_spi_device
I: CS control inhibited (no GPIO device)
Assertion failed at WEST_TOPDIR/zephyr/tests/boards/mec15xxevb_assy6853/qspi/src/main.c:94: test_spi_device: (jedec_id == W25Q128_JEDEC_ID is false)
JEDEC ID doesn't match
FAIL - test_spi_device in 0.20 seconds
===================================================================
START - test_spi_sector_erase
PASS - test_spi_sector_erase in 0.1 seconds
===================================================================
START - test_spi_single_write
PASS - test_spi_single_write in 0.1 seconds
===================================================================
START - test_spi_single_read
Assertion failed at WEST_TOPDIR/zephyr/tests/boards/mec15xxevb_assy6853/qspi/src/main.c:273: test_spi_single_read: (memcmp(buffer_tx, safbuf2, BUF_SIZE) == 0 is false)
Buffer read data is different to write data
FAIL - test_spi_single_read in 0.20 seconds
===================================================================
START - test_spi_dual_read
I: CS control inhibited (no GPIO device)
Assertion failed at WEST_TOPDIR/zephyr/tests/boards/mec15xxevb_assy6853/qspi/src/main.c:341: test_spi_dual_read: (memcmp(buffer_tx, safbuf2, BUF_SIZE) == 0 is false)
Buffer read data is different to write data
FAIL - test_spi_dual_read in 0.23 seconds
===================================================================
START - test_spi_quad_write
I: CS control inhibited (no GPIO device)
Assertion failed at WEST_TOPDIR/zephyr/tests/boards/mec15xxevb_assy6853/qspi/src/main.c:449: test_spi_quad_write: ((spi_status2 & SPI_STATUS2_QE) == SPI_STATUS2_QE is false)
Enable QSPI mode failure
FAIL - test_spi_quad_write in 0.22 seconds
===================================================================
START - test_spi_quad_read
Assertion failed at WEST_TOPDIR/zephyr/tests/boards/mec15xxevb_assy6853/qspi/src/main.c:579: test_spi_quad_read: (ret == 0 is false)
Receive fast read data spi_transceive failure: error -35
FAIL - test_spi_quad_read in 0.18 seconds
===================================================================
START - test_spi_octal_read
Assertion failed at WEST_TOPDIR/zephyr/tests/boards/mec15xxevb_assy6853/qspi/src/main.c:645: test_spi_octal_read: (ret == 0 is false)
Send quad read address spi_transceive failure: error -35
FAIL - test_spi_octal_read in 0.30 seconds
===================================================================
Test suite test_spi failed.
===================================================================
PROJECT EXECUTION FAILED

DEBUG   - DEVICE: *** Booting Zephyr OS build zephyr-v2.5.0-2732-g5e1502e816d7  ***
DEBUG   - DEVICE: Running test suite i2c_test
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_i2c_pca95xx
DEBUG   - DEVICE: E: Write error: -60
DEBUG   - DEVICE:
DEBUG   - DEVICE: Assertion failed at WEST_TOPDIR/zephyr/tests/boards/mec15xxevb_assy6853/i2c_api/src/main.c:48: test_i2c_pca95xx: (ret == 0 is false)
DEBUG   - DEVICE: Failed to write data to i2c device
DEBUG   - DEVICE: FAIL - test_i2c_pca95xx in 0.28 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: Test suite i2c_test failed.
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: PROJECT EXECUTION FAILED
DEBUG   - run status: mec15xxevb_assy6853/tests/boards/mec15xxevb_assy6853/i2c_api/boards.mec15xxevb_assy6853.i2c failed

*** Booting Zephyr OS build zephyr-v2.5.0-2839-gb504f8b6a27b  ***
Running test suite gpio_api_1pin_test
===================================================================
START - test_gpio_pin_configure_push_pull
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_pin_configure_push_pull in 0.6 seconds
===================================================================
START - test_gpio_pin_configure_single_ended
Running test on port=GPIO140_176, pin=14
When configured as input test pin value is high
Open Source configuration or Pull Up pin bias is not supported
PASS - test_gpio_pin_configure_single_ended in 0.114 seconds
===================================================================
START - test_gpio_pin_set_get_raw
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_pin_set_get_raw in 0.7 seconds
===================================================================
START - test_gpio_pin_set_get
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_pin_set_get in 0.7 seconds
===================================================================
START - test_gpio_pin_set_get_active_high
Running test on port=GPIO140_176, pin=14
Step 1: Set logical, get logical and physical pin value
Step 2: Set physical, get logical and physical pin value
PASS - test_gpio_pin_set_get_active_high in 0.18 seconds
===================================================================
START - test_gpio_pin_set_get_active_low
Running test on port=GPIO140_176, pin=14
Step 1: Set logical, get logical and physical pin value
Step 2: Set physical, get logical and physical pin value
PASS - test_gpio_pin_set_get_active_low in 0.18 seconds
===================================================================
START - test_gpio_pin_toggle
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_pin_toggle in 0.5 seconds
===================================================================
START - test_gpio_port_set_masked_get_raw
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_port_set_masked_get_raw in 0.7 seconds
===================================================================
START - test_gpio_port_set_masked_get
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_port_set_masked_get in 0.7 seconds
===================================================================
START - test_gpio_port_set_masked_get_active_high
Running test on port=GPIO140_176, pin=14
Step 1: Set logical, get logical and physical port value
Step 2: Set physical, get logical and physical port value
PASS - test_gpio_port_set_masked_get_active_high in 0.20 seconds
===================================================================
START - test_gpio_port_set_masked_get_active_low
Running test on port=GPIO140_176, pin=14
Step 1: Set logical, get logical and physical port value
Step 2: Set physical, get logical and physical port value
PASS - test_gpio_port_set_masked_get_active_low in 0.20 seconds
===================================================================
START - test_gpio_port_set_bits_clear_bits_raw
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_port_set_bits_clear_bits_raw in 0.6 seconds
===================================================================
START - test_gpio_port_set_bits_clear_bits
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_port_set_bits_clear_bits in 0.6 seconds
===================================================================
START - test_gpio_port_set_clr_bits_raw
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_port_set_clr_bits_raw in 0.6 seconds
===================================================================
START - test_gpio_port_set_clr_bits
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_port_set_clr_bits in 0.5 seconds
===================================================================
START - test_gpio_port_toggle
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_port_toggle in 0.5 seconds
===================================================================
START - test_gpio_int_edge_rising
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_int_edge_rising in 0.9 seconds
===================================================================
START - test_gpio_int_edge_falling
Running test on port=GPIO140_176, pin=14
Assertion failed at WEST_TOPDIR/zephyr/tests/drivers/gpio/gpio_api_1pin/src/test_pin_interrupt.c:97: test_gpio_pin_interrupt_edge: (cb_count not equal to cb_count_expected)
Test point 0: Pin interrupt triggered invalid number of times on rising/to active edge
FAIL - test_gpio_int_edge_falling in 0.27 seconds
===================================================================
START - test_gpio_int_edge_both
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_int_edge_both in 0.9 seconds
===================================================================
START - test_gpio_int_edge_to_active
Step 1: Configure pin as active high
Running test on port=GPIO140_176, pin=14
Step 2: Configure pin as active low
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_int_edge_to_active in 0.24 seconds
===================================================================
START - test_gpio_int_edge_to_inactive
Step 1: Configure pin as active high
Running test on port=GPIO140_176, pin=14
Assertion failed at WEST_TOPDIR/zephyr/tests/drivers/gpio/gpio_api_1pin/src/test_pin_interrupt.c:97: test_gpio_pin_interrupt_edge: (cb_count not equal to cb_count_expected)
Test point 0: Pin interrupt triggered invalid number of times on rising/to active edge
FAIL - test_gpio_int_edge_to_inactive in 0.31 seconds
===================================================================
START - test_gpio_int_level_high
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_int_level_high in 0.9 seconds
===================================================================
START - test_gpio_int_level_low
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_int_level_low in 0.9 seconds
===================================================================
START - test_gpio_int_level_active
Step 1: Configure pin as active high
Running test on port=GPIO140_176, pin=14
Step 2: Configure pin as active low
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_int_level_active in 0.24 seconds
===================================================================
START - test_gpio_int_level_inactive
Step 1: Configure pin as active high
Running test on port=GPIO140_176, pin=14
Step 2: Configure pin as active low
Running test on port=GPIO140_176, pin=14
PASS - test_gpio_int_level_inactive in 0.24 seconds
===================================================================
START - test_gpio_pin_toggle_visual
Running test on port=GPIO140_176, pin=14
LED ON
LED OFF
LED ON
LED OFF
PASS - test_gpio_pin_toggle_visual in 6.7 seconds
===================================================================
Test suite gpio_api_1pin_test failed.
===================================================================
PROJECT EXECUTION FAILED

Environment (please complete the following information):

• OS: Fedora 29
• Toolchain: Zephyr SDK 12.4
• Commit SHA: faf68fb213bc

Additional context Tests were passing before, after bisecting tests started failing after this commit: 5e1502e816d7e33cf104ac526f6567f966b169f8

[yerabolu]

@scottwcpg: Any update on this?

[scottwcpg]

SPI test passed on my board. Please check the SPI flash part is a Winbond W25Q128 with JEDEC ID that matches the JEDEC ID hard coded in the test: 0x001840ef. If you are loading the Zephyr test app from the same SPI flash as the test then it should be able to read JEDEC ID. For quad you must make sure the jumpers for SHD_IO2 and SHD_IO3 are correct: SHD_IO2 JP98 connect 8-9 SHD_IO3 JP98 connect 11-12

I2C_API test requires additional jumper settings: JP54 1-2 connected supplies +3.3V to PCA9555 Test uses I2C address 0x26 for PCA9555 requiring: JP53 1-2 connected sets PCA9555 I2C address A0=0 JP53 3-4 unconnected sets PCA9555 I2C address A1=1 JP53 5-6 unconnected sets PCA9555 I2C address A2=1

Have not looked at GPIO test

[scottwcpg]

GPIO falling edge test failure cause by HW firing the interrupt when the GPIO value is low and the interrupt detect mode changed from disabled to falling edge. I would classify this as a HW bug because it is acting like level low triggered. An hack for the test is to add clearing of cb_count and cb_count_expected before the for loop in test_gpio_pin_interrupt_edge. This allows both failing GPIO tests to pass but it is a race condition. It only works if the spurious ISR fires before clearing the variables the second time.

[scottwcpg]

@yerabolu Please review the above debug results

[yerabolu]

@scottwcpg: will look into it and get back to you.

[yerabolu]

@scottwcpg: Verified the jumper settings for SPI and I2C_API. All the settings are as you mentioned above and tests are passing now on my setup as well.

But the GPIO tests are still failing. If the hack causes a race condition, should we look at the test to see if there is another way to test it?

[scottwcpg]

@yerabolu All I can think of is to add delay in the test after enabling the GPIO interrupt, check the globals the ISR uses, if they are set clear them and proceed to the loop. If the delay is several hundred milliseconds that should be longer than the RC time constant of the pin load. But are there cases with other chips if the ISR fires before the test loop a real error should be reported?

[yerabolu]

@yerabolu All I can think of is to add delay in the test after enabling the GPIO interrupt, check the globals the ISR uses, if they are set clear them and proceed to the loop. If the delay is several hundred milliseconds that should be longer than the RC time constant of the pin load. But are there cases with other chips if the ISR fires before the test loop a real error should be reported?

@scottwcpg: Not sure if there are any other cases. Trying to investigate further.

[albertofloyd]

@yerabolu could you clarify if this affects Zephyr 2.5 onwards or only the main branch? I'm asking since traces indicate Booting Zephyr OS build zephyr-v2.5.0-2839-gb504f8b6a27b

Do you have any bisection data? i.e. do we know if anything pointing to any recent changes to gpio/qspi driver or related tests?

[yerabolu]

@yerabolu could you clarify if this affects Zephyr 2.5 onwards or only the main branch? I'm asking since traces indicate Booting Zephyr OS build zephyr-v2.5.0-2839-gb504f8b6a27b

Do you have any bisection data? i.e. do we know if anything pointing to any recent changes to gpio/qspi driver or related tests?

@albertofloyd : It is observed on the main branch as well. Test started failing after this commit 5e1502e.

[albertofloyd]

Thanks @yerabolu, seems this doesn't affect Zephyr 2.5 release but only Zephyr 2.6 and onwards (based solely on that commit inclusion info). @scottwcpg I recall you mention latest a HAL change required a pinmux driver update, is this the HAL update you were referring?

If so is the pinmux driver update in place?

[yerabolu]

@yerabolu All I can think of is to add delay in the test after enabling the GPIO interrupt, check the globals the ISR uses, if they are set clear them and proceed to the loop. If the delay is several hundred milliseconds that should be longer than the RC time constant of the pin load. But are there cases with other chips if the ISR fires before the test loop a real error should be reported?

@scottwcpg: Not sure if there are any other cases. Trying to investigate further.

@scottwcpg: Tried investigating and not sure if this helps - only mps2_an385, mps2_an521 platforms are excluded for this test. And test passes on other platforms. It is failing on Microchip.

[maksimmasalski]

@scottwcpg Any updates about that problem?

[scottwcpg]

@maksimmasalski I added a fix to the GPIO driver in PR 37138 and added you as a reviewer. The fix worked on my MEC152x EVB.

[maksimmasalski]

@maksimmasalski I added a fix to the GPIO driver in PR 37138 and added you as a reviewer. The fix worked on my MEC152x EVB.

Thank you, Scott. Will review it.

[albertofloyd]

@nashif how do we get this backported to v2.6?

@scottwcpg could you confirm what are the actual dependencies for this fix? https://github.com/zephyrproject-rtos/zephyr/pull/37479 mentions PR's 37138 and 37139.

[scottwcpg]

@albertofloyd PR #37479 is the final fix for both MEC152x and MEC172x. Previous PR's using DMB() were not a long term solution.

[maksimmasalski]

@nashif seems like it was not backported to the 2.6. @scottwcpg could you backport it to 2.6? If not, I will backport.