Build with filesystem/

[dcmcshan]

I am unfamiliar with your build approach (but very impressed by it). I am wondering if there is an easy way to package up a filesystem in your build world. Previously, I hacked the Makefile in ports/esp32 and used mklittlefs to package everything up. I could probably just do that here, but thought you might have something clever...

Thanks as always!

[kdschlosser]

ya lost me on this one... what are you trying to "package"?

The binary that gets output for the esp32 is a single bin file. if you are wanting to bake your python source files into the firmware that is able to be done pretty easily.

you need to make a manifest.py file for your project and in it you will have something along these lines...

freeze('path/to/file', 'filename.py')
freeze('path/to/file', 'filename.py')
freeze('', 'filename.py')

the paths are relitive to the location of the manifest file. that is the reason why I say to create the file in your project folder that holds the python source files you want to bake.

ONce you are done with that then when you build add the following onto your build command

FROZEN_MANIFEST=/absolute/path/to/manifest.py

replacing the path and filename so it points to the one you made.

You can only bake python source files into the firmware so if you want to also get images put into the firmware you are going to need to use this code to convert an image file into a bytearray...

import binascii
import os
import sys

try:
    input_file = sys.argv[1]
except IndexError:
    print('No file or directory given using current working directory')
    input_file = os.getcwd()

def run(in_file):
    output_file = os.path.splitext(in_file)[0] + '.py'

    ext = os.path.splitext(in_file)[1][1:]

    with open(input_file, 'rb') as f:
        data = f.read()

    data = binascii.hexlify(data)
    data = ['    ' + str(data[i: min(i + 74, len(data))])[1:] for i in range(0, len(data), 74)]
    data = '\n'.join(data)

    output = f'''\
import binascii

_{ext} = bytearray(binascii.unhexlify(
{data}
))
{ext} = memoryview(_{ext})
'''

    with open(output_file, 'w') as f:
        f.write(output)

if os.path.isdir(input_file):
    for file in os.listdir(input_file):
        file_ext = os.path.splitext(file)[1][1:]
        if file_ext not in ('bmp', 'jpg', 'gif', 'png'):
            continue

        file = os.path.join(input_file, file)
        print('found file:', file)
        run(file)

else:
    file_ext = os.path.splitext(input_file)[1][1:]
    if file_ext not in ('bmp', 'jpg', 'gif', 'png'):
        raise RuntimeError('supported image files are bmp, jpg, gif and png')
    print('found file:', input_file)
    run(input_file)

print('DONE!!')

That will convert images into modules that can be imported...

3 ways to use the script..

specify the file

python3 script.py /path/to/image.png

specify a directory that has multiple image files in it

python3 script.py /path/to/images

using the current working directory

cd /path/to/images
python3 /path/to/script.py

The generated modules will be located in the same folder as the image.

to use the image file you import the module and the name of the image is going to be the extension of the image. so if you had an image file names my_car.png you would run the following code to collect the image...

import my_car

print(my_car.png)

Yeah, I know... The way I did the naming is pretty slick...

[kdschlosser]

I forgot to mention that freezing the source files doesn't preserve the directory structure. it will be baked into the firmware as if all of the files are in the root folder of the ESP32.

The build script will automatically adjust the partition size to fit the firmware.

If you plan on using OTA firmware updates I highly recommend using the --partition-size build argument and set a size that is quite a bit larger than your initial project. You don't want to have to mess around with trying to resize the partitions as part of an OTA update.

To enable the OTA updates you need to add --ota to the build command. This will create 2 application partitions that are the same size. One is used for the new firmware and the other is for the old firmware. You need to have 2 application partitions to do OTA updates.

OTA updates allow you to update all of the firmware, that includes the MicroPython version and LVGL version that is being used. Optionally you can do OTA updates for your project only and that doesn't require 2 application partitions. This is able to be done with one and you can use a network socket from inside of python to handle the receiving of the data.

[dcmcshan]

Yeah, I definitely want OTA partitions. My hope was to prepopulate with directory structure. As I say, currently I use mklittefs to make a custom image that I can then add to the bin.

[dcmcshan]

Adding --ota to build fails:

Partition ota_0 invalid: Offset 0x12000 is not aligned to 0x10000
[1001/1967] Generating /Users/danielmcshan/GitHub/lvgl_micropython/lib/micropython/mpy-cross/build/mpy-cross              
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.
makeversionhdr.py: Warning: No git repo or tag info available, falling back to mpconfig.h version info.
[1005/1967] Generating ../../genhdr/qstr.i.last                                                                        
ninja: build stopped: subcommand failed.
-e See https://github.com/micropython/micropython/wiki/Build-Troubleshooting
ninja failed with exit code 1, output of the command is in the /Users/danielmcshan/GitHub/lvgl_micropython/lib/micropython/ports/esp32/build-ESP32_GENERIC_S3-SPIRAM_OCT/log/idf_py_stderr_output_20840 and /Users/danielmcshan/GitHub/lvgl_micropython/lib/micropython/ports/esp32/build-ESP32_GENERIC_S3-SPIRAM_OCT/log/idf_py_stdout_output_20840
make: *** [all] Error 1

I created a separate issue for this.

[dcmcshan]

What I probably need to do is figure a way to just take on a vfs.img to the binary in the right place...

[dcmcshan]

I have wasted a little time trying to build a vfs FAT filesystem and overlay it onto the .bin firmware that lvgl_micropython builds. With little success... the filesystems is detected as corrupt despite passing fsck locally. Is there an easy way to provide your build script with an external VFS that it can just build the .bin with? I've done this with regular micropython hacking the Makefile, but you have a more elegant and sophisticated system, I'm afraid to hack at it :)

[dcmcshan]

Here is how I am creating my FAT32 filesystem. When I did this before with stock micropython, I used littlefs. Is there a way in your make.py to use littlefs? Maybe FAT32 isn't a good answer?

#!/bin/bash

set -e  # Exit immediately if a command exits with a non-zero status

MCT_DIR="../MCT"
VFS_IMG="mct.img"
MAX_SIZE=$((1024 * 2048))  # 2MB (2097152 bytes)
SECTOR_SIZE=1024   # Changed to 1024 bytes
SECTOR_COUNT=2048  # Added sector count

# Function to check FAT image integrity
check_fat_image() {
    local image_file="$1"

    echo "Checking FAT image integrity..."

    if ! command -v fsck.vfat &> /dev/null; then
        echo "Error: fsck.vfat is not installed. Please install dosfstools."
        return 1
    fi

    # Run fsck.vfat on the image file with verbose output
    if ! fsck.vfat -vn "$image_file"; then
        echo "FAT image check failed. The filesystem may be corrupted."
        return 1
    fi

    echo "FAT image check passed."
    return 0
}

echo "Creating VFS image with size: $MAX_SIZE bytes"

# Remove existing image file if it exists
rm -f "$VFS_IMG"

# Create a raw image file
dd if=/dev/zero of="$VFS_IMG" bs=$SECTOR_SIZE count=$SECTOR_COUNT

# Format the image as FAT32
if ! mkfs.vfat -F 32 -S $SECTOR_SIZE "$VFS_IMG"; then
    echo "Error: Failed to format the image as FAT32"
    exit 1
fi
echo "Image created and formatted successfully"

# Create directories and copy files to the image using mmd and mcopy
echo "Copying files from $MCT_DIR to $VFS_IMG"
find "$MCT_DIR" -type d ! -path '*/.*' ! -path '*/__*' -print0 | 
    while IFS= read -r -d '' dir; do
        rel_dir="${dir#$MCT_DIR/}"
        if [ -n "$rel_dir" ]; then
            mmd -i "$VFS_IMG" "::$rel_dir"
        fi
    done

find "$MCT_DIR" -type f ! -name '.*' ! -path '*/.*' ! -path '*/__*' -print0 | 
    while IFS= read -r -d '' file; do
        rel_path="${file#$MCT_DIR/}"
        mcopy -i "$VFS_IMG" "$file" "::$rel_path"
    done
echo "Files copied successfully"

echo "VFS image created successfully: $VFS_IMG"
VFS_IMG_SIZE=$(stat -f%z "$VFS_IMG")
echo "Final VFS image size: $VFS_IMG_SIZE bytes"

# Print diagnostic information
echo "VFS image details:"
file "$VFS_IMG"
hexdump -C -n 512 "$VFS_IMG"

# Check the created FAT image
if ! check_fat_image "$VFS_IMG"; then
    echo "Warning: FAT image integrity check failed."
    exit 1
fi

echo "FAT image integrity check passed."

[dcmcshan]

I continue to struggle getting a good fat filesytem to overlay. I am using esp-idf fatfsgen and it does load and somewhat work, but the files are all lowercase and I cannot change directories to load files into /io or whatever.

This approach worked well enough with littlefs. Is there a build option to switch vfs from fat to littlefs if I can't get this working?