Does not work reliably with pages containing only FF

[mnijweide]

It is not clear to me how to implement dhara_nand_is_free() with an actual flash device. Checking if the page contains all FF seemed to be what was required but that doesn't work reliably when the map is resumed.

Especially there is no way to distinguish between an erased page and a used page containing 0xFF.

If any number of page were written with all 0xFF then can cause find_last_group() and find_head() to initialize the journal with a page that is already used, the next write would then corrupt the data. The pages may be at the end, or in the middle of something.

What would be the correct way to implement dhara_nand_free?

[dlbeer]

Yes, what you've described is correct. You should ensure that the NAND OOB data contains at least one non-0xff byte in order for this to work reliably. You could do this either using an ECC scheme that produces some zero ECC bits for an all-ones page, or by including an explicit reserved non-0xff byte.

[mnijweide]

I see. I'd hoped there was some way to avoid OOB data, the NAND chip I'm using already has built-in ECC so I don't need that.

[dlbeer]

Sorry, it's been a while since I worked on this, and I'd forgotten something: the pages that are checked will, if they've been programmed, contain non-FF bytes. They're checkpoint pages, which have a specific format -- so it is actually fine to test for the page being all-FF.

[mnijweide]

I tried that, but it didn't work. So I 'improved' the dhara_nand_is_free function so that it would only mark a page as free if all following pages are also FF - this vastly improved the stability, but is inefficient at boot time.

But the following still fails:

const int sector_size = 512;

struct dhara_map map;
struct dhara_nand flash_memory;
dhara_error_t err;
uint8_t* page_buf = malloc(sector_size);
uint8_t* sector_buf = malloc(sector_size);

flash_memory.log2_page_size = 9; // 2^9 = 512, the sector size
flash_memory.log2_ppb = 9; // and 2^9 = 512 sectors fit in the 256 kB flash erase block
flash_memory.num_blocks = 4 * 5; // 4 erase blocks = 1 MB, and we have a 5 MB region

dhara_map_init(&map, &flash_memory, page_buf, 64);
dhara_map_resume(&map, &err);
dhara_map_clear(&map);

dhara_sector_t cap = dhara_map_capacity(&map);
cap = cap * 4 / 5; /* Use 80% otherwise this test takes way too long */

for( int run = 0; run < 7; run ++) {
    const int set_a[7] = { 0x00, 0x00, 0x00, 0xff, 0xff, 0xff, 0x00 };
    const int set_b[7] = { 0xff, 0xff, 0x00, 0xff, 0x00, 0x00, 0xff };

    printf("Run: %d - ", run);

    memset( sector_buf, set_a[run], sector_size );

    for (dhara_sector_t i = 0; i < cap - 1; i++) {
        if (i == cap / 2) {
            memset(sector_buf, set_b[run], sector_size );

            /* Clear memory to simulate reboot */
            memset(page_buf, 0xFF, sector_size);
            memset(&map, 0xFF, sizeof(map));

            dhara_map_init(&map, &flash_memory, page_buf, 64);
            dhara_map_resume(&map, &err);
        }
        dhara_map_write(&map, i, sector_buf, &err);
    }

    /* Clear memory to simulate reboot */
    memset(page_buf, 0xFF, sector_size);
    memset(&map, 0xFF, sizeof(map));

    dhara_map_init(&map, &flash_memory, page_buf, 64);
    dhara_map_resume(&map, &err);

    for (dhara_sector_t i = 0; i < cap - 1; i++) {
        dhara_map_read(&map, i, sector_buf, &err);
        uint8_t cmp = i < cap/2 ? set_a[run] : set_b[run];
        for (int j = 0; j < sector_size; j++) {
            if (sector_buf[j] != cmp) {
                printf("Error: %d; sector %d contains %d should be %d\n", run, i, sector_buf[j], cmp);
                break;
            }
        }
    }
}

(the code was copied from test code and I had to remove some parts, so it may not work)

[dlbeer]

The simulator used in the test suite keeps a "next page" counter for each block and uses that for testing whether a page is free. I just modified it check the page's contents instead, and I find that everything works as expected.

I'm sure you've checked already, but are you absolutely sure there's nothing wrong with your NAND flash access layer? If so, is there a minimal test case you can think of that I'd be able to use to reproduce the problem (and fix it)?

[mnijweide]

I'm very sure there is nothing wrong with my NAND flash access layer. I've created a small test case that writes logical sector 0 with FF then reboots and writes sector 1 with 00. Inserted printf() in the nand access layer and this is what I get:

*** boot
dhara_nand_is_free( ..., page = 5884 ) : 0
dhara_nand_is_free( ..., page = 5888 ) : 0
dhara_nand_is_free( ..., page = 6012 ) : 1
dhara_nand_is_free( ..., page = 5948 ) : 1
dhara_nand_is_free( ..., page = 5916 ) : 1
dhara_nand_is_free( ..., page = 5900 ) : 1
dhara_nand_is_free( ..., page = 5892 ) : 0
dhara_nand_is_free( ..., page = 5896 ) : 1
dhara_nand_is_free( ..., page = 5893 ) : 0
dhara_nand_is_free( ..., page = 5894 ) : 0
dhara_nand_is_free( ..., page = 5896 ) : 1             // dhara_map_clear is here 
*** write sector 0 with 0xff
dhara_nand_prog( ..., page = 5896, data[0] = 255, ... )   // data
dhara_nand_prog( ..., page = 5899, data[0] = 68, ... )     // checkpoint page?
*** reboot
dhara_nand_is_free( ..., page = 5884 ) : 0
dhara_nand_is_free( ..., page = 5888 ) : 0
dhara_nand_is_free( ..., page = 6012 ) : 1
dhara_nand_is_free( ..., page = 5948 ) : 1
dhara_nand_is_free( ..., page = 5916 ) : 1
dhara_nand_is_free( ..., page = 5900 ) : 1
dhara_nand_is_free( ..., page = 5892 ) : 0
dhara_nand_is_free( ..., page = 5896 ) : 1
dhara_nand_is_free( ..., page = 5893 ) : 0
dhara_nand_is_free( ..., page = 5894 ) : 0
dhara_nand_is_free( ..., page = 5896 ) : 1            // but this is a user page
*** write sector 1 with 0x 0
dhara_nand_prog( ..., page = 5896, data[0] = 0, ... )     // and this is writing to the user page
dhara_nand_prog( ..., page = 5899, data[0] = 68, ... )
*** reboot
dhara_nand_is_free( ..., page = 5884 ) : 0
dhara_nand_is_free( ..., page = 5888 ) : 0
dhara_nand_is_free( ..., page = 6012 ) : 1
dhara_nand_is_free( ..., page = 5948 ) : 1
dhara_nand_is_free( ..., page = 5916 ) : 1
dhara_nand_is_free( ..., page = 5900 ) : 1
dhara_nand_is_free( ..., page = 5892 ) : 0
dhara_nand_is_free( ..., page = 5896 ) : 0
dhara_nand_is_free( ..., page = 5896 ) : 0
dhara_nand_is_free( ..., page = 5900 ) : 1
dhara_nand_is_free( ..., page = 5897 ) : 0
dhara_nand_is_free( ..., page = 5898 ) : 0
dhara_nand_is_free( ..., page = 5900 ) : 1
sector 0 contains 0 should be 255
sector 1 contains 252 should be 0

And fixing this.... Is see that dhara_nand_is_free is invoked for several user pages. find_last_group() hits only on user pages and find_head() even explicitly iterates through user pages and invokes dhara_nand_free() on them. I think it would be easy to change find_last_group so it hits on checkpoint pages, but as for find_head()?

test case code:

static void test_simple()
{
    const int sector_size = 512;
    const int fill_byte[] = { 0xff, 0x00 };

    struct dhara_nand flash_memory;
    flash_memory.region = REG1; /* Test region */
    flash_memory.log2_page_size = 9; // 2^9 = 512, the sector size
    flash_memory.log2_ppb = 9; // and 2^9 = 512 sectors fit in the 256 kB flash erase block
    flash_memory.num_blocks = 4 * 4; // 4 erase blocks = 1 MB, and we have a 4 MB region

    uint8_t* page_buf = malloc(sector_size);
    uint8_t* sector_buf = malloc(sector_size);

    int r;
    struct dhara_map map;
    dhara_error_t err;

    /* "boot" */
    printf("*** boot\n");
    dhara_map_init(&map, &flash_memory, page_buf, 64);
    r = dhara_map_resume(&map, &err);
    if (r == 0) {
        // there is recognizable data, make sure we start with empty memory
        dhara_map_clear(&map);
    }

    /* cycle */
    for (dhara_sector_t s = 0; s < sizeof(fill_byte) / sizeof(fill_byte[0]); s++) {
        printf("*** write sector %d with 0x%2x\n",s, fill_byte[s]);
        memset( sector_buf, fill_byte[s], sector_size );
        dhara_map_write(&map, s, sector_buf, &err);
        dhara_map_sync(&map, &err);

        /* "reboot" */
        printf("*** reboot\n");
        memset(page_buf, 0xFF, sector_size);
        memset(&map, 0xFF, sizeof(map));
        dhara_map_init(&map, &flash_memory, page_buf, 64);
        dhara_map_resume(&map, &err);
    }

    /* Verify correct data was written */
    for (dhara_sector_t s = 0; s < sizeof(fill_byte) / sizeof(fill_byte[0]); s++) {
        r = dhara_map_read(&map, s, sector_buf, &err);
        for (int i = 0; i < sector_size; i++) {
            if (sector_buf[i] != fill_byte[s]) {
                printf("sector %d contains %d should be %d\n", s, sector_buf[i], fill_byte[s]);
                break;
            }
        }
    }

    free(sector_buf);
    free(page_buf);
}

[dlbeer]

Ah, I see. Sorry, it's been a while since I worked on this -- I think I was expecting that users would be implementing their own OOB scheme and would therefore be able to determine whether a page was programmed by looking at something other than its contents (that's why nand_is_free exists at all).

If you really aren't able to do anything with OOB or query the chip, what I'd suggest in your case is a small change to dhara/journal.c. Add this function:

static int group_is_free(struct dhara_journal *j, dhara_page_t up0)
{
       const unsigned int n = 1 << j->log2_ppc;
       unsigned int i;

       for (i = 0; i < n; i++)
               if (!dhara_nand_is_free(j->nand, up0 + i))
                       return 0;

       return 1;
}

...and then change calls elsewhere to dhara_nand_is_free(j->nand, ...) to group_is_free(j, ...).

I haven't checked that thoroughly, so be careful with it.

[mnijweide]

That works. The only thing I'm concerned about that if reboot occurs after a user page with all FF is written but before the checkpoint page was written, that initialization of the journal will use a page that is already in use. But that is an academical case in my situation.

Anyways, thanks for your time.

[dlbeer]

No problem. It might possibly not be an issue -- I've found on most chips that if you program a page to be all 0xff (including the ECC bytes), it's essentially a no-op and you can freely write the page (i.e. you can go from 1 to 0 at any time, just never from 0 to 1 without an erase).

When I have a bit of spare time I'll look at the fix I've suggested in a bit more detail and perhaps integrate it into the repository properly. In the case where the user is able to implement an ECC-based is_free(), it shouldn't impose much extra cost.

In the meantime, let me know if you do run into any problems with it.

[dlbeer]

After further thought and a bit of testing, this seems to be fine. I've added commit f4a0733752ecd3ce16f6d75d71bce133342c0178.