`os.listdir()` Too Slow

Cursor-S commented 2 years ago

CircuitPython version

Adafruit CircuitPython 7.2.5 on 2022-04-06; Raspberry Pi Pico with rp2040

Code/REPL

import os
import time
import board
import storage
import busio as io
from adafruit_sdcard import SDCard
from digitalio import DigitalInOut

SD_PATH = "/sd"

spi = io.SPI(board.GP2, board.GP3, board.GP0)
cs = DigitalInOut(board.GP1)
sdcard = SDCard(spi, cs)
vfs = storage.VfsFat(sdcard)
storage.mount(vfs, SD_PATH)

# list out all the directories in the card
dirs = []
for item in os.listdir(SD_PATH):
    start_time = time.monotonic()
    try:
        os.listdir(SD_PATH + "/" + item)
        dirs.append(item)
    except:
        pass
    print(time.monotonic() - start_time)  # print the time used for os.listdir()

print(dirs)

Behavior

The results are correct but some of the os.listdir() seems that has used more than 0.4s according to print(time.monotonic() - start_time)

Description

Too slow using os.listdir() and try except to pick out all the directories while I have a lot of files on my card, it takes over 30s in all
I've tried to change the SD card but didn't work
I've tried to print out the time taken by each os.listdir() and I noticed that some is quick as 0.1s but some are slow as 0.4s+
Or maybe there can be some way quicker to pick out all the directories?

Additional information

No response

anecdata commented 2 years ago

Can you give more detail on the number of directories and files? How many directories at the root, and what's the average and range in number of files in those directories? You might try not to listdir on files that aren't directories (os.stat[6] will be zero for directories, and non-zero for non-zero-length files, I believe).

Cursor-S commented 2 years ago

There are 160 files and 5 directories on the card. One of the directories is empty and the largest one includes 120 files. On average, 20 files.

The file system of the card is FAT32

Actually, it's mostly not slow while listdir on directories. But on files, listdir takes longer to raise an exception.

Checking os.stat[6] is a little bit quicker, but I'd like to know how can I tell the difference between empty files and directories for both of them returns a 0 on os.stat[6] :D

Thanks.

Neradoc commented 2 years ago

I use this to test if a path is a directory:

import os

IS_DIR = 0x4000
IS_FILE = 0x8000

def is_dir(file):
    return os.stat(file)[0] & IS_DIR != 0

I don't know if there is some unavoidable overhead, but you can try speeding up the bus speed. The default speed in the library is 1.32MHz, this sets it to 10MHz:

sdcard = SDCard(spi, cs, baudrate=10_000_000)

dhalbert commented 2 years ago

Besides speeding up the bus, could you also try using sdcardio instead of adadfruit_sdcard? It is a native implementation of the Python code inadafruit_sdcard.

adafruit / circuitpython