MMU mapping to flash fails for Vaddr range other than Vaddr1 (IDFGH-179)

[nkolban]

This might be a little complex to explain so I'll try and be as clear as possible. If anything is not clear, please contact me and I'll be delighted to assist. ESP_Angus has also sniffed this issue and is (based on a cursory examination) of the mind that this may be a defect.

The premise of the story deals with the low-level MMU functions described in 26.3.2 of the ESP32 Technical Reference (v2.9). Specifically, it deals with mapping flash storage to instruction virtual address areas. If we read the Tech Ref, specifically table 101, we find that there are three address ranges of interest to us. These are:

• V Addr1 - 0x4000 0000 to 0x403F FFFF
• V Addr2 - 0x4040 0000 to 0x407F FFFF
• V Addr3 - 0x4080 0000 to 0x40BF FFFF

Each region is thus 4MBytes in size. We are mapping pages of 64K and hence each region contains 64 pages.

• VAddr1 - 64 - 127
• VAddr2 - 128 - 195
• VAddr3 - 196 - 255

There are two register tables whose base addresses are supplied in the globals DPORT_PRO_FLASH_MMU_TABLE and DPORT_APP_FLASH_MMU_TABLE.

Setting the table entry (which is a 32bit value) to the page number of a flash offset page, causes the corresponding address range to map to flash. For example:

DPORT_PRO_FLASH_MMU_TABLE[96] = 48

Would cause the address range 0x4020 0000 to 0x4020 FFFF to map to offset 0x30 0000 of flash.

What we are finding is that if we map in the V Addr1 range (as described in the tech reference), an attempt to read the corresponding data in the address space does indeed return the content of flash.

However, if we attempt to read from VAddr2 or VAddr3, what we get back is 0x0bad00bad indicating no flash mapping.

Putting this another way, if we use MMU table entries 64-127 it works, but using table entries 128-255 it doesn't.

Here is a sample application that can be used for testing:

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <esp_spi_flash.h>
#include <soc/dport_reg.h>
#include <soc/cpu.h>
#include <rom/cache.h>

// MMU Table Entry

// Test #1
int MMUTableEntry = 96;  // 0x4020 0000
volatile uint32_t *b_instructions = (volatile uint32_t)0x40200000;

// Test #2
//int MMUTableEntry = 128; // 0x4040 0000
//volatile uint32_t *b_instructions = (volatile uint32_t)0x40400000;

// Test #3
//int MMUTableEntry = 192; // 0x4080 0000
//volatile uint32_t *b_instructions = (volatile uint32_t)0x40800000;

extern void spi_flash_disable_interrupts_caches_and_other_cpu();

extern void spi_flash_enable_interrupts_caches_and_other_cpu();

void IRAM_ATTR mapFlash() {

    printf("Setting the flash mapping\n");
    spi_flash_disable_interrupts_caches_and_other_cpu();

    DPORT_PRO_FLASH_MMU_TABLE[MMUTableEntry] = 48;
    DPORT_APP_FLASH_MMU_TABLE[MMUTableEntry] = 48;

    Cache_Flush(0);
    Cache_Flush(1);
    spi_flash_enable_interrupts_caches_and_other_cpu();
}

void app_main(void)
{
    mapFlash();
    printf("%08x %08x %08x %08x\n", b_instructions[0], b_instructions[1], b_instructions[2], b_instructions[3]);
}

[negativekelvin]

Have you ruled out

When the DPORT_PRO_SINGLE_IRAM_ENA bit of register DPORT_PRO_CACHE_CTRL_REG is 1, the MMU enters this special mode for PRO_CPU memory accesses. Similarily, when the DPORT_APP_SINGLE_IRAM_ENA bit of register DPORT_APP_CACHE_CTRL_REG is 1, the APP_CPU accesses memory using this special mode. In this mode, the process and virtual address page supported by each configuration entry of MMU are different. For details please see Table 104 and 105. As shown in these tables, in this special mode V Addr2 and V Addr3 cannot be used to access External Flash.

[nkolban]

Howdy my friend. That is indeed a great thought and to answer your question, no ... I hadn't checked those special bit flags.

I added the following code to my test environment:

printf("DPORT_PRO_SINGLE_IRAM_ENA: %ld\n", DPORT_PRO_CACHE_CTRL_REG & DPORT_PRO_SINGLE_IRAM_ENA);
printf("DPORT_APP_SINGLE_IRAM_ENA: %ld\n", DPORT_APP_CACHE_CTRL_REG & DPORT_APP_SINGLE_IRAM_ENA);

and the result was:

DPORT_PRO_SINGLE_IRAM_ENA: 0
DPORT_APP_SINGLE_IRAM_ENA: 0

This seems to tell me that the values (as returned above) should indicate that we are not in the special mode as described in the tech ref. However, it may still very well be that whatever snipe may be lurking here is related to the notion of Vaddr1 can access flash but Vaddr2 and Vaddr3 regions can't. My reading seems to say that if we are not in special mode, then Vaddr1, Vaddr2 and Vaddr3 regions should work ... but yet experiments seem to show only Vaddr1 working.

[projectgus]

Hi @nkolban,

@negativekelvin is right. With the default configuration, maximum 4MB of SPI flash contents can be mapped into the address space at once. Somehow when I first saw your issue I did some back of the envelope calculations and concluded index 192 was inside this range, but it's clearly not (4MB == 64 pages).

EDIT: Noted this is the default config. As per the TRM, it is possible to use the higher virtual address ranges (ie higher indexes in the DPORT table) - but there are caveats with this which is why we don't support it right now.

Angus

[nkolban]

Howdy guys, I'm still not seeing it ...

I read about a "special mode" which appears to be used to engage PIDs 2-7. However, I am also sensing that this special mode is off.

Is there a restriction that says only 4M of SPI flash can be be mapped at once? I didn't see that in the tech ref ... (note that because I didn't see it, doesn't at all mean it isn't there). On page 536 it states:

Because there are eight address bits in an MMU entry, and the page size for external flash is 64 KB, a maximum of 256 * 64 KB = 16 MB of external flash is supported.

[projectgus]

Because there are eight address bits in an MMU entry, and the page size for external flash is 64 KB, a maximum of 256 * 64 KB = 16 MB of external flash is supported.

This is the physical address range (ie each entry in DPORT table is 0-255). You can map anywhere in the 16MB region of physical external flash into the address space.. But you can't map all 16MB at once.

EDIT: Fixed "256 bits wide" to 0-255, ie 8 bits wide.

[nkolban]

I'm still being a dummy ... here is what I am reading but mis-understanding.

The MMU mapping configuration registers will be referred to as ’entries’ in the rest of this chapter.

Each register is called an "entry".

These two tables are essentially the same, with the sole difference being that the APP_CPU entry numbers are 2048 higher than the corresponding PRO_CPU numbers.

Two tables ... one called DPORT_PRO_FLASH_MMU_TABLE and the other called DPORT_APP_FLASH_MMU_TABLE. Tables are separated by 2048 32bit entries.

Thinking only of the PRO table, I study table 102 in the tech ref.

That seems to show a table composed of 32bit entries from 0 to 1152 (1136+16)

Every configuration entry of MMU maps a virtual address page of a CPU process to a physical address page. An entry is 32 bits wide. Of these, bits 0~7 indicate the physical page the virtual page is mapped to. Bit 8 should be cleared to indicate that the MMU entry is valid; entries with this bit set will not map any physical address to the virtual address.

Can you elaborate on the notion of "each entry in DPORT table is 256 bits wide"?

[projectgus]

Can you elaborate on the notion of a "each entry in DPORT table is 256 bits wide"?

Sorry, I meant to say 8 bits wide - ie 0-255. Fixed my reply.

The DPORT_PRO_FLASH_MMU_TABLE arrays are arrays where the index into the array is a virtual address (0-64 is Vaddr0, etc, etc). The value written to each item in the "array" of registers (ie the entry) lets you set an 8 bit physical page number. So you can set values in the range 0 (physical offset 0) to 255 (physical offset 255 * 65536 = 16711680 = the last page of a 16MB range).

So the comment you quoted is talking about physical address range 0-16MB, not a virtual address range.

Contrary to the short comment I left before, it is possible (via reconfiguring the MMU, as described in the TRM) to map Vaddr2-3 into the IRAM address space, allowing you to have more than 4MB of virtual instruction space mapped at one time. However (as you've found) this isn't supported with the MMU configuration that ESP-IDF uses right now.

See also https://github.com/espressif/esp-idf/issues/1184

[negativekelvin]

@projectgus

via reconfiguring the MMU, as described in the TRM

Don't think it is in there. DPORT_PRO_CMMU_FLASH_PAGE_MODE, DPORT_PRO_CACHE_MASK_x are undocumented. Note that cache_flash_mmu_set returns OK for vaddr2 & 3 but the cache refuses to fetch the data. Register values appears to indicate special mode is not set.