Open waihong84 opened 6 years ago
waihong84
commented
6 years ago It is better to call lfs_file_sync every time a successful write. It is becoming more stable after adding this statement. I will perform more tests and reports more findings.
FreddieChopin
commented
6 years ago Quite possible that this is related to #75 (skip to the last comment at the bottom). If you already have a lot of data on the file system, the whole-system-scan - which is executed during the first modifying operation (like creating a file or a directory) - can take pretty long time, as the code really attempts to read everything that you have in the filesystem. On embedded system "a lot" can be quite a small number - with slow SPI transfers I saw this initial whole-system-scan to take 1-2 minutes on a file system which had just 2 MB of data.
geky
commented
6 years ago Sorry for the late reply. I think @FreddieChopin is right, it's not stuck but just taking a long time.
How many files and how large are the files on your filesystem?
You can try increasing the block size. Multiplying the block size by n divides the scan cost by n. It also forces each file to be a multiple of the block size, but on an SD card this is less of a concern. There is no RAM cost for this. (Most SD cards are formatted with 8KB FAT clusters for similar reasons).
It looks like you can pull this off in RIOT OS by changing this part of your init function:
#define PAGESPERBLOCK 16 // For 8KB blocks on a SD card with 512B erase units
/* erasing whole sectors is handled internally by the card so you can
delete single blocks (i.e. pages) */
dev->pages_per_sector = PAGESPERBLOCK;
dev->sector_count = ((cardInfo.BlockNbr * cardInfo.BlockSize) / BLOCKSIZE) / PAGESPERBLOCK;
/* sdcard_spi always uses the fixed block size of SD-HC cards */
dev->page_size = cardInfo.BlockSize;Hi @geky , First of all, thanks for your response. I will try with your solution and update the result. To answer your questions,
How many files and how large are the files on your filesystem?
It can be in the range of 16MB - 32MB of size, and may need around 30 files, although not all the files are at the same size. I am trying to work on a simple database on a file system, and it needs to be power safe.
In the meantime, I did some performance test on LittleFs.
Test Procedures
Result (Only one file is shown here, the remaining 3 files show similar result, just the spike is getting taller, meaning more time is needed for the execution)
Observation
The file reading time on different offset is consistent. It takes no more than 30ms to fetch 512 bytes of data. Code routine for the test TEST_LEVEL defined as 4
static void tests_littlefs(void *pvParameters)
{
int fd[TEST_LEVEL]; // 4 open files
char fileName[32];
uint32_t testIndex = 0;
uint32_t i;
for (i = 0; i < TEST_LEVEL; i++)
{
sprintf(fileName, "/sdcard/%lu.txt", i + 1);
fd[i] = vfs_open(fileName, O_RDWR, 0);
if (fd[i] <= 0)
{
if (fd[i] == -ENOENT)
{
fd[i] = vfs_open(fileName, O_RDWR | O_CREAT, 0);
if (fd[i] <= 0)
// Error, shouldn't happen
osThreadTerminate(NULL);
}
else
{
// Error, shouldn't happen
osThreadTerminate(NULL);
}
}
}
srand(time(NULL));
DEBUG(("Files are opened.\r\n"));
for (;;)
{
testIndex = 0;
// Test file read
// TestFileSeekRead(fd, &testIndex);
// Test file write and read to compare
TestFileSeekWriteValidate(fd);
TestFileSeekRead(fd);
osThreadTerminate(NULL);
}
}static void TestFileSeekWriteValidate(int *fd) { off_t res; TickType_t tickBefore; uint8_t buff[512]; uint8_t readBuff[512]; char fileName[FILENAME_MAX]; struct stat fileStat; off_t remainByte; off_t currLoc; off_t bytesToExecute; off_t bytesReturn; uint32_t i, j;
for (i = 0; i < sizeof(buff); i++)
{
buff[i] = (uint8_t) (i);
}
for (i = 0; i < TEST_LEVEL; i++)
{
res = (off_t) vfs_fstat(fd[i], &fileStat);
assert_param(res >= 0);
DEBUG(("\r\nFile %d.txt, Size = %d\r\n", (i + 1), fileStat.st_size));
remainByte = fileStat.st_size;
currLoc = 0;
if (remainByte == 0)
{
DEBUG(("write\r\n"));
DEBUG(("pos, lseek, write, lseek, read\r\n"));
// newly created file
// write around 32MB of file 62500 loop
// Write around 1MB of file, 2000 loops
// Write around 6MB of file, 11720 loops
// 15625 * 512 == 8 MB
for (j = 0; j < NUM_LOOP_OF_512; j++, currLoc += sizeof(buff))
{
DEBUG(("%d, ", currLoc));
bytesToExecute = sizeof(buff);
// lseek
tickBefore = osKernelSysTick();
res = vfs_lseek(fd[i], currLoc, SEEK_SET);
assert_param(res == currLoc);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
// write
tickBefore = osKernelSysTick();
bytesReturn = vfs_write(fd[i], buff, bytesToExecute);
assert_param(bytesReturn == bytesToExecute);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
// lseek
tickBefore = osKernelSysTick();
res = vfs_lseek(fd[i], currLoc, SEEK_SET);
assert_param(res == currLoc);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
// read
tickBefore = osKernelSysTick();
bytesReturn = vfs_read(fd[i], readBuff, bytesToExecute);
assert_param(bytesReturn == bytesToExecute);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
// compare
tickBefore = osKernelSysTick();
assert_param(memcmp(buff, readBuff, sizeof(buff)) == 0);
DEBUG(("%lu,\r\n", osKernelSysTick() - tickBefore));
}
}
// test file modification, write to the temp file, then replace it with original file
{
DEBUG(("modify\r\n"));
res = (off_t) vfs_fstat(fd[i], &fileStat);
assert_param(res >= 0);
DEBUG(("\r\nFile %d.txt, Size = %d\r\n", (i + 1), fileStat.st_size));
remainByte = fileStat.st_size;
DEBUG(("currLoc, lseek, modify, write file, lseek, read file, compare, unlink, rename and reopen\r\n"));
currLoc = 0;
// Create a temporary file
int tempFileFd = vfs_open("/sdcard/temp.txt", O_CREAT | O_RDWR, 0);
assert_param(tempFileFd > 0);
while (remainByte > 0)
{
if (remainByte > 512)
bytesToExecute = 512;
else
bytesToExecute = remainByte;
for (j = 0; j < sizeof(buff); j++)
{
buff[j] = (uint8_t) sizeof(buff) - j - 1;
}
DEBUG(("%lu,", currLoc));
// Seek current file
tickBefore = osKernelSysTick();
res = vfs_lseek(fd[i], currLoc, SEEK_SET);
assert_param(res == currLoc);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
// Randomly check whether need to update the line, otherwise read the content from original and write to temp file
if (Rand(1, 5, 2))
{
DEBUG(("1,"));
// Write temp file
tickBefore = osKernelSysTick();
bytesReturn = vfs_write(tempFileFd, buff, bytesToExecute);
assert_param(bytesReturn == bytesToExecute);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
// Read Temp file
tickBefore = osKernelSysTick();
vfs_lseek(tempFileFd, currLoc, SEEK_SET);
bytesReturn = vfs_read(tempFileFd, readBuff, bytesToExecute);
assert_param(bytesReturn == bytesToExecute);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
// Compare Temp file
tickBefore = osKernelSysTick();
assert_param(memcmp(buff, readBuff, sizeof(buff)) == 0);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
}
else
{
DEBUG(("0,"));
// read current file
bytesReturn = vfs_read(fd[i], buff, bytesToExecute);
assert_param(bytesReturn == bytesToExecute);
// and write into temp file
tickBefore = osKernelSysTick();
bytesReturn = vfs_write(tempFileFd, buff, bytesToExecute);
assert_param(bytesReturn == bytesToExecute);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
// Read Temp file
tickBefore = osKernelSysTick();
vfs_lseek(tempFileFd, currLoc, SEEK_SET);
bytesReturn = vfs_read(tempFileFd, readBuff, bytesToExecute);
assert_param(bytesReturn == bytesToExecute);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
// Compare Temp file
tickBefore = osKernelSysTick();
assert_param(memcmp(buff, readBuff, sizeof(buff)) == 0);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
}
remainByte -= bytesReturn;
currLoc += bytesReturn;
if (remainByte != 0)
DEBUG(("\r\n"));
}
// Close temp file
vfs_close(tempFileFd);
sprintf(fileName, "/sdcard/%s", vfs_getopenfilename(fd[i]));
vfs_close(fd[i]);
tickBefore = osKernelSysTick();
vfs_unlink(fileName);
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
// rename
tickBefore = osKernelSysTick();
vfs_rename("/sdcard/temp.txt", fileName);
// reopen file
fd[i] = vfs_open(fileName, O_RDWR, 0);
assert_param(fd[i] > 0);
DEBUG(("%lu,\r\n", osKernelSysTick() - tickBefore));
}
}}
static void TestFileSeekRead(int *fd) { off_t res; TickType_t tickBefore; uint8_t buff[512]; uint32_t i, j, k, l, m; off_t offset[TEST_LEVEL];
DEBUG(("File Offset Read\r\n"));
DEBUG(("read1, read2, read3, read4, offset1, offset2, offset3, offset4\r\n"));
for (i = 0; i < TEST_LEVEL; i++)
{
for (j = 0; j < TEST_LEVEL; j++)
{
for (k = 0; k < TEST_LEVEL; k++)
{
for (l = 0; l < TEST_LEVEL; l++)
{
offset[0] = GetFileAddressOffset((eFileReadLevel) i);
tickBefore = osKernelSysTick();
// Read 1.txt
res = vfs_lseek(fd[0], offset[0], SEEK_SET);
assert_param(res == offset[0]);
res = vfs_read(fd[0], buff, sizeof(buff));
assert_param(res == sizeof(buff));
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
offset[1] = GetFileAddressOffset((eFileReadLevel) j);
tickBefore = osKernelSysTick();
// Read 2.txt
res = vfs_lseek(fd[1], offset[1], SEEK_SET);
assert_param(res == offset[1]);
res = vfs_read(fd[1], buff, sizeof(buff));
assert_param(res == sizeof(buff));
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
offset[2] = GetFileAddressOffset((eFileReadLevel) k);
tickBefore = osKernelSysTick();
// Read 3.txt
res = vfs_lseek(fd[2], offset[2], SEEK_SET);
assert_param(res == offset[2]);
res = vfs_read(fd[2], buff, sizeof(buff));
assert_param(res == sizeof(buff));
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
offset[3] = GetFileAddressOffset((eFileReadLevel) l);
tickBefore = osKernelSysTick();
// Read 4.txt
res = vfs_lseek(fd[3], offset[3], SEEK_SET);
assert_param(res == offset[3]);
res = vfs_read(fd[3], buff, sizeof(buff));
assert_param(res == sizeof(buff));
DEBUG(("%lu,", osKernelSysTick() - tickBefore));
// now get the offset
for (m = 0; m < TEST_LEVEL; m++)
{
DEBUG(("%lu,", offset[m]));
}
// New line
DEBUG(("\r\n"));
}
}
}
}}
Hi,
I did another same test on 32F746GDISCOVERY, but with the following changes.
#define PAGESPERBLOCK 16 // For 8KB blocks on a SD card with 512B erase units
/* erasing whole sectors is handled internally by the card so you can
delete single blocks (i.e. pages) */
dev->pages_per_sector = PAGESPERBLOCK;
dev->sector_count = ((cardInfo.BlockNbr * cardInfo.BlockSize) / BLOCKSIZE) / PAGESPERBLOCK;
/* sdcard_spi always uses the fixed block size of SD-HC cards */
dev->page_size = cardInfo.BlockSize;
Hi Guys,
Currently I am working on a project that has FTP Server included. The board that I am using is STM32L496G-Disco, LittleFs on SD Card, and W5500 from Wiznet for the tcp/ip interface.
I have ported the virtual file system on top of LittleFs from RIOT OS, it seems works. The file creation, writing and directory creation are working fine if without any power interruption.
The code is always stuck at at lfs_traverse whenever an attempt for new directory or new file creation, after the system is restarted during a writing process is in progress. The system is executing the while(true) loop in
lfs_traversewithout breaking out. Kindly refer below on the snapshot.Also, please refer below on the sd card lower level calling,
I will try to look at the library to check is there a way to printout the debug log.