Closed spikeyamk closed 3 months ago
BrianPugh
commented
6 months ago The line in question, where lfs_block_t is uint32_t.
I haven't seen this before. Theoretically, LittleFS should be able to handle files up to size ~2GB. Not to unnecessarily drag @geky in, but do you have any high level quick thoughts on this? Do we believe a bug lies in this repo (esp_littlefs) or in upstream official littlefs?
spikeyamk
commented
6 months ago The line in question, where
lfs_block_tisuint32_t.I haven't seen this before. Theoretically, LittleFS should be able to handle files up to size ~2GB. Not to unnecessarily drag @geky in, but do you have any high level quick thoughts on this? Do we believe a bug lies in this repo (esp_littlefs) or in upstream official littlefs?
Any idea where I should start digging? I've tried changing a bunch of settings in the esp_little configuration in sdkconfig it didn't really help either
BrianPugh
commented
6 months ago IDK if this is possible in your setup, but can you try something similar with SPI flash? That then may point to if it's a generic problem, or an sd-card specific problem.
spikeyamk
commented
6 months ago IDK if this is possible in your setup, but can you try something similar with SPI flash? That then may point to if it's a generic problem, or an sd-card specific problem.
So, I used the same function as here created this in partitions.csv:
# Name, Type, SubType, Offset, Size, Flags
# Note: if you have increased the bootloader size, make sure to update the offsets to avoid overlap
nvs, data, nvs, 0x9000, 0x6000,
phy_init, data, phy, 0xf000, 0x1000,
factory, app, factory, 0x10000, 2M,
graphics, data, spiffs, , 0xF0000,
And the thing worked fine, it worked up to 512 kB file, but at 1MB file it failed with:
E (656363) esp_littlefs: ./managed_components/joltwallet__littlefs/src/littlefs/lfs.c:689:error: No more free space 0x1a5
But that's a completely different error and says it ran out of space I use an 8MB FLASH ESP32-C6 devkit.
I did this in the same project as my previous test and didn't change anything about it.
BrianPugh
commented
6 months ago With those partitions (assuming the graphics partition, that has a size of 0xF0000 == 983040 bytes) that error makes sense.
So this is actually kind of good news; that means that the error is (probably, most likely) in this repo, esp_littlefs, and is isolated to the SD code. @huming2207 do you think you would be able to look into this?
spikeyamk
commented
6 months ago So, the funny thing is when I do something like this in a loop:
size_t integrity_test(const size_t max_bin_pow) {
for(size_t i = 1; i < (2 << max_bin_pow); i *= 2) {
const std::string name { std::string("jcb").append(std::to_string(i)) };
if(Trielo::trielo<create_test_file>(Trielo::OkErrCode(0), i, name) != 0) {
return i;
}
if(Trielo::trielo<check_test_file>(Trielo::OkErrCode(0), i, name) != 0) {
return i;
}
}
return 0;
}
Where it creates a file and then creates a file twice as large as the iteration before it will fail at filesize 128 kB, but when I only do this once in app_main
extern "C" void app_main() {
if(Trielo::trielo<SD_Card::init>(Trielo::OkErrCode(0)) != ESP_OK) {
return;
}
if(Trielo::trielo<SD_Card::format>(Trielo::OkErrCode(0)) != ESP_OK) {
return;
}
const std::string_view name { "test" };
if(Trielo::trielo<SD_Card::create_test_file>(Trielo::OkErrCode(0), (2 << 18), name) != 0) {
return;
}
if(Trielo::trielo<SD_Card::check_test_file>(Trielo::OkErrCode(0), (2 << 18), name) != 0) {
return;
}
Trielo::trielo<SD_Card::deinit>(Trielo::OkErrCode(0));
}and hardcode the value of the file size and create only one file, it will fail at file size 512 kB with:
assert failed: lfs_bd_prog lfs.c:262 (pcache->block == ((lfs_block_t)-1))
Bare in mind trielo function does nothing it's just a wrapper that prints whether a function failed or succeeded. And it fails inside the function that creates a new file the check_test_file function just reads the entire file and checks whether there's chars from 'a' to 'z'.
Creating and closing a bunch files subsequently seems to make this problem worse.
huming2207
commented
6 months ago Hmmm, this looks like an SPI driver issue to me. See: https://github.com/espressif/esp-idf/issues/7804
Somehow you might just transmitted more than 32KB for a single transaction underneath and cause some sort of corruption? Maybe try SDMMC interface and see if the issue still occurs? I will have a closer look later today.
huming2207
commented
6 months ago By the way, what's the SD card sector size and sector count? @spikeyamk
spikeyamk
commented
6 months ago Hmmm, this looks like an SPI driver issue to me. See: espressif/esp-idf#7804
Somehow you might just transmitted more than 32KB for a single transaction underneath and cause some sort of corruption? Maybe try SDMMC interface and see if the issue still occurs? I will have a closer look later today.
I'm not sure if this helps but I use this in my code?:
const spi_bus_config_t bus_cfg {
.mosi_io_num = Pins::mosi,
.miso_io_num = Pins::miso,
.sclk_io_num = Pins::clk,
.quadwp_io_num = -1,
.quadhd_io_num = -1,
.max_transfer_sz = 4092,
};So, how would it try to transmit more than 32 kB at once I'm not sure.
Alright, I'll try later in the evening when I come back home...
By the way, what's the SD card sector size and sector count? @spikeyamk
So this function: sdmmc_card_print_info gives me:
Name: 00000
Type: SDSC
Speed: 20.00 MHz (limit: 20.00 MHz)
Size: 1910MB
CSD: ver=1, sector_size=512, capacity=3911680 read_bl_len=10
SSR: bus_width=1So I guess the sector size is 512 and total sectors 3911680.
spikeyamk
commented
6 months ago Maybe try SDMMC interface and see if the issue still occurs? I will have a closer look later today.
So it seems like only ESP32S3 and ESP32 supports this and I only have a ESP32C6 board, so I can't do that...
geky
commented
6 months ago Can confirm this is unlikely to be a littlefs issue. littlefs also doesn't support cross-block/cross-sector writes at the moment so https://github.com/espressif/esp-idf/issues/7804 seems a bit unlikely if I understand the integration correctly.
Honestly I'd be a bit suspicious of the c++ stdlib/integration here. What is BUFSIZ? Could it be that the stdlib was compiled with a different BUFSIZ?
spikeyamk
commented
6 months ago Can confirm this is unlikely to be a littlefs issue. littlefs also doesn't support cross-block/cross-sector writes at the moment so espressif/esp-idf#7804 seems a bit unlikely if I understand the integration correctly.
Honestly I'd be a bit suspicious of the c++ stdlib/integration here. What is BUFSIZ? Could it be that the stdlib was compiled with a different BUFSIZ?
BUFSIZ is 128 but honestly it don't matter. I added that thing with char buf[BUFSIZ] with std::setbuf(file, buf) and std::flush(file) as a desparate attempt to fix this and it failed the exact same way before.
huming2207
commented
6 months ago So I guess the sector size is 512 and total sectors 3911680.
This is similar to our 512 byte sector SD card (those XTX SD NAND chip). It works fine for us...
Also, could you please tell me what's the ESP-IDF version are you using? Could you please try v5.2 or newer if you were using the old ones?
spikeyamk
commented
6 months ago So I guess the sector size is 512 and total sectors 3911680.
This is similar to our 512 byte sector SD card (those XTX SD NAND chip). It works fine for us...
Also, could you please tell me what's the ESP-IDF version are you using? Could you please try v5.2 or newer if you were using the old ones?
I'm using ESP-IDF 5.2. I'm gonna try bunch of different versions of ESP-IDF to see if it changes anything
spikeyamk
commented
6 months ago No other version of ESP-IDF seems to fix this. Also I found another error:
E (27560) esp_littlefs: ./managed_components/joltwallet__littlefs/src/littlefs/lfs.c:1366:error: Corrupted dir pair at {0x22dc5, 0x22dc6}
When creating a file with
FILE* file = std::fopen("file_name", "w"); std::close(file);
and then writing 512 bytes to the file and immediately closing it.
Then after few iterations of this in a loop it will fail at creating 74th file like this with the error above.
Also I tried mounting the using the sdspi API to write to SD Card blocks directly and it seems to work fine. I tested the card in Windows for bad blocks.
I also looked into code in this repo and here: write to a partition and write to an sdcard function and the other function completely ignores the lfs_off_t off argument passed to the function. I know that you don't have the granularity of access to blocks you have in SPI flash, but I'm confused and not sure how this is supposed to work.
spikeyamk
commented
6 months ago Also on this line you do some weird thing instead of using the lsf_config_t* c function parameter passed to the function. How do you actually write data to the sectors? Do you immediately push the data into the SD card sectors when std::fwrite is called and when the size of the input buffer is not exactly the sector size it'll leave the rest empty or do you wait for more data to arrive and then write a whole sector out?
BrianPugh
commented
6 months ago I also looked into code in this repo and here: write to a partition and write to an sdcard function and the other function completely ignores the lfs_off_t off
It's not ignored, it's used immediately when computing part_off.
The corrupted dir pair at this point is somewhat expected, because it seems like data is being corrupted somewhere on the way to disk. It's to be expected that the read back data is also bad.
Also on this line you do some weird thing instead of using the lsf_config_t* c function parameter passed to the function
That efs comes from lfs_config_t a few lines above:
esp_littlefs_t * efs = c->context;Ok, so I tried a bunch of different things:
std::fopen, std::fwrite etc. So I wrote this:
#include <cstdint>
#include <cstring>
#include <cstdio>
#include <cassert>
#include <array>
#include <string>namespace LFS { namespace SDMMC { int read(const struct lfs_config c, lfs_block_t block, lfs_off_t off, void buffer, lfs_size_t size) { assert(off == 0); assert(size == c->block_size); assert(block < c->block_count); sdmmc_card_t card = static_cast<sdmmc_card_t>(c->context); assert(sdmmc_read_sectors(card, buffer, block, 1) == ESP_OK); return LFS_ERR_OK; }
int erase(const struct lfs_config *c, lfs_block_t block) {
assert(block < c->block_count);
sdmmc_card_t* card = static_cast<sdmmc_card_t*>(c->context);
assert(sdmmc_erase_sectors(card, block, 1, SDMMC_ERASE_ARG) == ESP_OK);
return LFS_ERR_OK;
}
int prog(const struct lfs_config *c, lfs_block_t block, lfs_off_t off, const void *buffer, lfs_size_t size) {
assert(off == 0);
assert(size == c->block_size);
assert(block < c->block_count);
sdmmc_card_t* card = static_cast<sdmmc_card_t*>(c->context);
assert(sdmmc_write_sectors(card, buffer, block, 1) == ESP_OK);
return LFS_ERR_OK;
}
int sync(const struct lfs_config *c) {
(void) c;
return LFS_ERR_OK;
}
int test(sdmmc_card_t* card) {
const lfs_size_t block_size { 512 };
const lfs_size_t block_count { static_cast<lfs_size_t>(card->csd.capacity) };
static uint8_t read_buffer[block_size];
static uint8_t prog_buffer[block_size];
static uint8_t lookahead_buffer[block_size];
const lfs_config config {
.context = static_cast<void*>(card),
.read = read,
.prog = prog,
.erase = erase,
.sync = sync,
#ifdef LFS_THREADSAFE
.lock = nullptr,
.unlock = nullptr,
#endif
.read_size = block_size,
.prog_size = block_size,
.block_size = block_size,
.block_count = block_count,
.block_cycles = -1,
.cache_size = block_size,
.lookahead_size = block_size,
.compact_thresh = static_cast<lfs_size_t>(-1),
.read_buffer = static_cast<void*>(read_buffer),
.prog_buffer = static_cast<void*>(prog_buffer),
.lookahead_buffer = static_cast<void*>(lookahead_buffer),
.name_max = 0,
.file_max = 0,
.attr_max = 0,
.metadata_max = 0,
.inline_max = static_cast<lfs_size_t>(-1),
};
lfs_t lfs;
if(Trielo::trielo<lfs_format>(Trielo::OkErrCode<int>(LFS_ERR_OK), &lfs, &config) != LFS_ERR_OK) {
return 1;
}
// mount the filesystem
if(Trielo::trielo<lfs_mount>(Trielo::OkErrCode<int>(LFS_ERR_OK), &lfs, &config) != LFS_ERR_OK) {
return 2;
}
static const std::array<uint8_t, 512> message {
0x52, 0x20, 0xeb, 0x24, 0xe8, 0x1f, 0xa5, 0x66,
0x00, 0x16, 0xc7, 0x65, 0x42, 0x5c, 0x3c, 0x71,
0x52, 0x6c, 0x2c, 0x0f, 0x36, 0xc2, 0x92, 0x52,
0xc1, 0x4f, 0x86, 0x0c, 0x72, 0x69, 0x06, 0x95,
0x54, 0x92, 0x55, 0x3e, 0xf4, 0xaf, 0x12, 0x2e,
0x39, 0xc0, 0xf6, 0x9a, 0x9b, 0x19, 0x91, 0xc9,
0xb3, 0xbd, 0xd0, 0x85, 0xbc, 0x1a, 0xed, 0xb5,
0xce, 0x09, 0x1a, 0x68, 0x75, 0xd9, 0x42, 0xdc,
0x18, 0xf7, 0xed, 0x27, 0xcb, 0x8a, 0x22, 0xb6,
0x8b, 0xf9, 0x0c, 0x7b, 0x01, 0x0a, 0x57, 0x66,
0x75, 0x45, 0x72, 0x48, 0xf6, 0xdd, 0x72, 0x16,
0x9e, 0x99, 0x87, 0x1f, 0x12, 0x1e, 0x61, 0x20,
0x66, 0xf9, 0x9b, 0x08, 0x1c, 0x3e, 0x8d, 0xa5,
0xfd, 0x2a, 0x4c, 0xbe, 0xd7, 0x44, 0xfc, 0x30,
0x87, 0x7a, 0x15, 0xd4, 0x18, 0x53, 0x35, 0xb3,
0xad, 0x59, 0x2d, 0xb0, 0xf1, 0xaf, 0x4a, 0xff,
0x06, 0x3b, 0x95, 0x75, 0xef, 0xb2, 0x1b, 0x2f,
0x56, 0x31, 0x09, 0x93, 0x8d, 0x1a, 0x50, 0x68,
0xcd, 0x81, 0x16, 0x47, 0x6d, 0xa0, 0x5f, 0x04,
0xc9, 0x49, 0x5c, 0x60, 0xc9, 0x7a, 0xf4, 0x3d,
0x9c, 0x73, 0xea, 0x66, 0x87, 0x4d, 0xfb, 0xb6,
0x7a, 0xc3, 0x56, 0x6f, 0xb9, 0x58, 0x49, 0x8c,
0x6b, 0x92, 0x14, 0xbc, 0x9f, 0x78, 0x5c, 0x63,
0x97, 0xb1, 0x41, 0xd2, 0x19, 0x71, 0x11, 0xc0,
0x15, 0xa2, 0x1f, 0x4b, 0x76, 0x80, 0x2b, 0xf2,
0xf6, 0xe1, 0x28, 0xcd, 0x0d, 0x3f, 0xca, 0xe8,
0x9c, 0x90, 0x51, 0x34, 0xc5, 0xdc, 0x0f, 0x6d,
0xc6, 0xb0, 0x88, 0x47, 0x13, 0x4b, 0xac, 0x8d,
0x52, 0x95, 0xf1, 0xec, 0xed, 0xe0, 0x99, 0x9b,
0x3d, 0xde, 0x14, 0xd3, 0x8a, 0xb3, 0xd9, 0x4f,
0x23, 0x2e, 0xb2, 0x66, 0xec, 0x98, 0xa8, 0x09,
0x0d, 0x7d, 0xc8, 0xf3, 0x61, 0x3d, 0x39, 0x6a,
0x00, 0xc2, 0x0e, 0x65, 0x3d, 0xda, 0x3d, 0xce,
0x82, 0xc8, 0x12, 0x37, 0x30, 0xfb, 0xd6, 0x3b,
0x47, 0xbb, 0xf9, 0x68, 0xce, 0x2d, 0x27, 0x0e,
0xc7, 0xd3, 0x8f, 0x47, 0x89, 0xeb, 0x21, 0xfb,
0x0e, 0xd4, 0x47, 0xd3, 0xcf, 0x54, 0xa6, 0xac,
0x38, 0xed, 0xf2, 0xa5, 0xf6, 0x41, 0xe0, 0xae,
0xa1, 0xa9, 0xfe, 0xf1, 0x23, 0x9c, 0x47, 0xf4,
0x24, 0xa6, 0x74, 0x57, 0xd5, 0x8b, 0xe9, 0xa4,
0x9a, 0x75, 0xda, 0x28, 0xdd, 0xf1, 0x2e, 0x84,
0x14, 0x07, 0x2f, 0x47, 0xb8, 0xb8, 0xbb, 0x41,
0xf4, 0x1a, 0x2b, 0x7d, 0xb2, 0x32, 0x23, 0x6e,
0xc9, 0x80, 0x9e, 0x10, 0x07, 0x3d, 0x6d, 0xaa,
0x5b, 0x08, 0x61, 0x79, 0xfd, 0xd4, 0xdf, 0x7a,
0x3b, 0xb7, 0x54, 0x9b, 0x84, 0xb0, 0x2c, 0xff,
0xce, 0x23, 0x84, 0x53, 0x59, 0x33, 0xff, 0xa4,
0xe6, 0x7b, 0x54, 0xd4, 0x81, 0x5e, 0xad, 0xb9,
0x9a, 0x60, 0x64, 0x3e, 0x93, 0x5e, 0xec, 0x2e,
0xbb, 0xac, 0x02, 0x57, 0xd6, 0x33, 0xdc, 0xea,
0xbe, 0xe4, 0xb1, 0x56, 0x19, 0x7d, 0xa0, 0x16,
0x0c, 0xed, 0x2a, 0x2b, 0xd4, 0x38, 0x1d, 0x75,
0x5d, 0x35, 0x07, 0xf9, 0x19, 0xc3, 0x5e, 0x1c,
0xa5, 0x94, 0x08, 0xa1, 0x8e, 0x0e, 0x73, 0xda,
0x39, 0x32, 0x92, 0xb9, 0x62, 0x62, 0xa8, 0x68,
0xb0, 0xa5, 0xf8, 0x18, 0x1d, 0x2e, 0x63, 0x4c,
0x1b, 0xea, 0xe9, 0xc0, 0x80, 0xf6, 0xec, 0x7d,
0x50, 0x96, 0x3f, 0x5c, 0x82, 0x1d, 0xe8, 0x44,
0x5f, 0x01, 0x3b, 0x20, 0xea, 0x58, 0xc6, 0x56,
0x43, 0x93, 0x2c, 0xe9, 0x7d, 0x7e, 0xf5, 0x6f,
0x8d, 0x06, 0x36, 0x75, 0xc6, 0x2b, 0x1e, 0xd0,
0x74, 0xf9, 0x51, 0x86, 0x8a, 0x75, 0xdc, 0x3c,
0x76, 0xe8, 0xfe, 0xd7, 0x52, 0xd6, 0x83, 0x71,
0xb4, 0x21, 0x32, 0x77, 0x5e, 0xc0, 0x5b, 0x11,
};
for(size_t i = 0; i < 10; i++) {
const std::string name { std::string("test").append(std::to_string(i)) };
lfs_file_t write_file;
Trielo::trielo<lfs_file_open>(Trielo::OkErrCode<int>(LFS_ERR_OK), &lfs, &write_file, name.c_str(), LFS_O_RDWR | LFS_O_CREAT);
const lfs_ssize_t ret_write { lfs_file_write(&lfs, &write_file, static_cast<const void*>(message.data()), message.size()) };
if(ret_write != static_cast<lfs_ssize_t>(message.size())) {
std::printf("lfs_file_write(&lfs, &write_file, static_cast<const void*>(message.data()), message.size()) != static_cast<lfs_ssize_t>(message.size())\n");
std::printf("ret_write: %ld\n", ret_write);
return -1 * static_cast<int>(i);
}
Trielo::trielo<lfs_file_close>(Trielo::OkErrCode<int>(LFS_ERR_OK), &lfs, &write_file);
lfs_file_t read_file;
Trielo::trielo<lfs_file_open>(Trielo::OkErrCode<int>(LFS_ERR_OK), &lfs, &read_file, name.c_str(), LFS_O_RDONLY);
static std::array<uint8_t, message.size()> read_message;
read_message.fill(0);
const lfs_ssize_t ret_read { lfs_file_read(&lfs, &read_file, static_cast<void*>(read_message.data()), read_message.size()) };
if(read_message != message) {
std::printf("read_message != message\n");
std::printf("ret_read: %ld\n", ret_read);
return -1 * static_cast<int>(i);
}
Trielo::trielo<lfs_file_close>(Trielo::OkErrCode<int>(LFS_ERR_OK), &lfs, &read_file);
}
if(Trielo::trielo<lfs_unmount>(Trielo::OkErrCode<int>(LFS_ERR_OK), &lfs) != LFS_ERR_OK) {
return 3;
}
return 0;
}
}}
Although this code seems to work at first, it all falls apart when trying to use it to create larger files or multiple more than 100 files with size of 512 bytes. The occurence of the failures seems to somewhat deterministic. So, I though maybe there's bad sectors inside the SD card? Or it's some kind of other hardware failure?
2. I put the SD card into my laptop I took 2GB of random data from `/dev/urandom` saved it into a file then wrote it into each sector of the SD Card with `dd` command and then dumped the content of the entire SD card into a file again with `dd` command and ran `diff` command to compare the two files and they were identical so,...
3. Last I wrote this to write to the sectors of the SD card directly:esp_err_t SD_Card::check_all_blocks() { static const std::array<uint8_t, 512> urazr { 0xac, 0x72, 0x1b, 0x29, 0x76, 0x1c, 0x09, 0x56, 0xd6, 0xc6, 0xe9, 0xb4, 0x4e, 0x84, 0x01, 0xdc, 0x4e, 0x77, 0x62, 0x53, 0xa2, 0xb4, 0x2e, 0x75, 0xe6, 0x69, 0x65, 0x69, 0x70, 0x53, 0xc8, 0x81, 0x43, 0xb5, 0x7b, 0xe5, 0xc8, 0x81, 0xba, 0x2a, 0xa7, 0x80, 0x07, 0x46, 0x39, 0x26, 0x4f, 0xd7, 0xf2, 0x1b, 0xf1, 0xd4, 0x0d, 0xae, 0x90, 0x37, 0x96, 0xa6, 0x2b, 0x50, 0x76, 0x82, 0x78, 0x77, 0x8a, 0x86, 0x4c, 0x5d, 0xb5, 0xa7, 0x6c, 0x63, 0xe1, 0xf9, 0x43, 0xda, 0x51, 0xac, 0x73, 0xdf, 0xfb, 0x4b, 0xc0, 0x30, 0x5b, 0x89, 0x37, 0x85, 0x1c, 0x3d, 0xc8, 0xd5, 0xbc, 0x21, 0xc2, 0x51, 0x3b, 0x21, 0xc6, 0xed, 0xe0, 0x3b, 0x07, 0x4b, 0xe7, 0x88, 0x0b, 0xa0, 0xaf, 0x3e, 0x84, 0xb4, 0x83, 0xe6, 0xb0, 0xb4, 0xf3, 0xcf, 0x38, 0x3b, 0xce, 0xbb, 0x2f, 0x87, 0x17, 0x25, 0x68, 0x7a, 0x29, 0x6b, 0x82, 0xe1, 0x1d, 0x03, 0x98, 0x10, 0x5d, 0x2e, 0x12, 0xeb, 0xa4, 0x89, 0xc2, 0x8b, 0x4c, 0xe4, 0x73, 0x94, 0x18, 0x66, 0xaf, 0xcc, 0x10, 0x3c, 0x8d, 0x01, 0xc0, 0x0c, 0x9c, 0xab, 0x17, 0xb3, 0x00, 0x60, 0xb7, 0xcd, 0x74, 0xbe, 0xe0, 0x34, 0x0c, 0xf3, 0x8b, 0x95, 0xfd, 0xd5, 0xb9, 0x95, 0x80, 0xaa, 0x05, 0x66, 0x18, 0x6a, 0xc5, 0x5d, 0x5c, 0xff, 0x86, 0x2d, 0xc2, 0x08, 0x6a, 0x31, 0xb3, 0x9e, 0xe7, 0x2f, 0xf2, 0x71, 0xc4, 0x91, 0xb9, 0x20, 0x43, 0xfc, 0xcb, 0xf0, 0x0c, 0xcb, 0xce, 0xd9, 0x8b, 0xf3, 0xfb, 0x8b, 0xba, 0xbd, 0x64, 0x87, 0xab, 0x9f, 0x77, 0x17, 0xec, 0x3f, 0x97, 0x78, 0xcc, 0xe2, 0x86, 0x65, 0x63, 0xa7, 0x82, 0x47, 0xa5, 0xa6, 0x40, 0x0d, 0xfb, 0x82, 0xf4, 0xe7, 0x1a, 0x02, 0xba, 0x26, 0x32, 0x96, 0x8a, 0x5e, 0xc0, 0x76, 0x26, 0x4d, 0x6e, 0x36, 0x1c, 0x26, 0xc8, 0x05, 0x61, 0xf3, 0xe5, 0xdf, 0x49, 0x80, 0xa8, 0xb0, 0x12, 0xf7, 0x4c, 0xb2, 0xe2, 0x62, 0xbb, 0x34, 0x2b, 0x30, 0xc2, 0x47, 0x47, 0x2c, 0x36, 0xa7, 0x7f, 0x97, 0xd6, 0x23, 0xea, 0x73, 0xf2, 0x68, 0x57, 0x7d, 0x78, 0x79, 0xd4, 0x5f, 0xb5, 0xfe, 0x0b, 0x87, 0x37, 0x26, 0x3e, 0xdd, 0x28, 0xe5, 0xf3, 0xe6, 0xea, 0xae, 0x1b, 0xc7, 0x31, 0x4b, 0x27, 0x3b, 0xd9, 0x9d, 0x5e, 0x85, 0x19, 0xeb, 0xe8, 0x9e, 0x99, 0xb6, 0x7e, 0x54, 0xc3, 0xac, 0xc7, 0x22, 0x17, 0x23, 0xea, 0x47, 0x70, 0x4c, 0xde, 0xe3, 0x70, 0x8e, 0x61, 0x4a, 0x06, 0x4f, 0xcd, 0x91, 0x0b, 0x6e, 0x3b, 0x85, 0x99, 0xa0, 0x6b, 0xa7, 0x48, 0x45, 0x52, 0x84, 0x9a, 0xea, 0xde, 0xc9, 0xe6, 0x05, 0x32, 0xeb, 0x16, 0x32, 0xb9, 0xb3, 0x43, 0x5c, 0xe4, 0x59, 0x50, 0x91, 0x0f, 0xc3, 0x34, 0x4b, 0xa0, 0x06, 0xe6, 0xe6, 0xbf, 0xa9, 0x98, 0x56, 0xb6, 0x19, 0x8c, 0xa1, 0xfe, 0xca, 0xf5, 0x5d, 0x17, 0xd7, 0xe5, 0xbf, 0x25, 0x96, 0xbf, 0x9e, 0x71, 0x2e, 0x70, 0x1b, 0x9b, 0x50, 0x9a, 0x92, 0x29, 0x76, 0xb5, 0xea, 0xd6, 0xe5, 0xb4, 0x35, 0x10, 0xbc, 0x1d, 0x0c, 0xe6, 0xc7, 0x5d, 0xfb, 0x93, 0xb1, 0xe7, 0xab, 0x50, 0xac, 0x82, 0xe5, 0xf3, 0x31, 0x09, 0xcc, 0x78, 0x72, 0x73, 0x7b, 0x20, 0x53, 0x4f, 0x2a, 0xb7, 0x44, 0x4b, 0xe7, 0xa2, 0xc0, 0xee, 0xac, 0x27, 0xc6, 0x3c, 0x8c, 0xdf, 0x4c, 0x30, 0x41, 0xf2, 0x80, 0x09, 0x41, 0xc7, 0xcb, 0xce, 0x8c, 0xff, 0x75, 0x29, 0xcb, 0x42, 0x23, 0xe5, 0x3c, 0x11, 0x14, 0x5d, 0x1b, 0xc8, 0x8a, 0x0a, 0x55, 0xeb, 0x37, 0x6c, 0xa5, 0x9b, 0xbc, 0x2b, 0x3a, 0x10, 0xc3, 0xe7, 0x02, 0x31, 0xba, 0xa7, 0x29, 0x27, 0x66, };
std::shared_ptr<std::optional<size_t>> done { std::make_shared<std::optional<size_t>>(0) };
std::thread([](const std::shared_ptr<std::optional<size_t>> done, const size_t block_count) {
const int64_t sleep_s { 3 };
for(
size_t i = 0, current = done->value(), before = current;
done->has_value();
[&i, ¤t, &before, block_count, &done]() {
before = current;
current = done->value_or(block_count);
if(i == 9) {
i = 0;
} else {
i++;
}
}()
) {
const float average_speed_s {
(
static_cast<float>(
current > before ?
current - before
: before - current
)
)
/ static_cast<float>(sleep_s)
};
const std::chrono::seconds eta { static_cast<int64_t>(block_count / average_speed_s) };
const std::chrono::hours eta_h { std::chrono::duration_cast<std::chrono::hours>(eta) };
const std::chrono::minutes eta_m { std::chrono::duration_cast<std::chrono::minutes>(eta) - std::chrono::duration_cast<std::chrono::minutes>(eta_h) };
const std::chrono::seconds eta_s { eta - std::chrono::duration_cast<std::chrono::seconds>(eta_h) - std::chrono::duration_cast<std::chrono::seconds>(eta_m) };
std::cout
<< "SD_Card::check_all_blocks: done: '" << done->value_or(block_count) << "' "
<< "out of: '" << block_count << "' blocks "
<< "average speed: '" << average_speed_s << "' blocks/s "
<< "ETA: '" << eta_h << ' ' << eta_m << ' ' << eta_s << "' "
<< std::endl;
std::this_thread::sleep_for(std::chrono::seconds(sleep_s));
}
}, done, card.csd.capacity).detach();
for(
;
done->value() < card.csd.capacity;
done->emplace(done->value() + 1)
) {
if(sdmmc_write_sectors(&card, urazr.data(), done->value(), 1) != ESP_OK) {
std::printf("sdmmc_write_sectors(&card, urazr.data(), %zu, 1) != ESP_OK)\n", done->value());
done->reset();
return ESP_FAIL;
}
static std::array<uint8_t, urazr.size()> read_buf;
read_buf.fill(0);
if(sdmmc_read_sectors(&card, read_buf.data(), done->value(), 1) != ESP_OK) {
std::printf("sdmmc_read_sectors(&card, read_buf.data(), %zu, 1) != ESP_OK)\n", done->value());
done->reset();
return ESP_FAIL;
}
if(read_buf != urazr) {
std::printf("read_buf != urazr, i: %zu\n", done->value());
done->reset();
return ESP_FAIL;
}
}
done->reset();
return ESP_OK;}
And it's super slow, we're talking 128 sectors/s ~ 128*512 bytes/s. But it seems to work. It's supposed to take 9 hours to complete?
Honestly, I don't know what I'm doing and I'm running out of ideas, but I suppose the VFS works fine?? since littlefs fails even when I'm interacting with it directly? Sometimes it fails when writing to the file, but sometimes writing to the file succeeds, but when I try to verify the written data by reading it it fails.Although this code seems to work at first, it all falls apart when trying to use it to create larger files or multiple more than 100 files with size of 512 bytes. The occurence of the failures seems to somewhat deterministic. So, I though maybe there's bad sectors inside the SD card? Or it's some kind of other hardware failure?
Hmmm that sounds like a low-level issue to me, it could be Espressif's SPI and SD card driver or hardware connections (e.g. wiring is too long).
I've also seen occasionally some SD card over SPI may behave differently compared to SD over SDMMC mode. I remembered I've read a Chinese article long time ago that someone managed to recover a semi-bricked SD card data with lower-speed SPI, while he couldn't read out anything by normal USB card readers over SDMMC. Maybe in your case this could be somewhat similar? Have you tried any other SD cards? Particularly some newer ones?
BrianPugh
commented
3 months ago closing due to lack of activity.
I'm doing something like this to create a test file with ESP32-C6 and 2GB SD card over single lane SPI. I've tried adding the std::fflush and std::setbuf for the FILE* stream and it doesn't seem to help at all. What is the actual maximum file size? And can you create files like 1MB in size and larger? Or not?
Logs: