search

esp8266 / arduino-esp8266fs-plugin

Arduino plugin for uploading files to ESP8266 file system
GNU General Public License v2.0
706 stars 214 forks source link

Need to automate creating and uploading a SPIFFS image from the CLI #51

Open scottchiefbaker opened 5 years ago

scottchiefbaker commented 5 years ago

I'm working on polishing a Makefile for automating compilation and uploading to an ESP8266 board. Using the Arduino IDE I'm able to build and upload a SPIFFS image per these instructions. I am unable to find a way to automate this from the command line.

What commands does the IDE use to compile and upload a SPIFFS image? I found reference to this chunk of code in another issue as a starting point. Is there a way to enable debug mode in the IDE and spit out the exact commands it runs? Once I can figure out the exact commands I can recreate them in my Makefile.

scottchiefbaker commented 5 years ago

I believe I was able to reverse engineer the commands. I'm leaving them here just for searchability if someone else has this issue. Can someone confirm that what I found are the appropriate settings for 1MB, 2MB, and 3MB SPIFFS?

I've integrated it in to my Arduino Makefile when you call make spiffs

Arduino Makefile

1MB Spiffs

mkspiffs -c ~/Arduino/ds18b20/data/ -p 256 -b 8192 -s 1028096 /tmp/out.spiffs
esptool -cd nodemcu -cb 460800 -cp /dev/ttyUSB0 -ca 0x300000 -cf /tmp/out.spiffs

2MB Spiffs

mkspiffs -c ~/Arduino/ds18b20/data/ -p 256 -b 8192 -s 2076672 /tmp/out.spiffs
esptool -cd nodemcu -cb 460800 -cp /dev/ttyUSB0 -ca 0x200000 -cf /tmp/out.spiffs

3MB Spiffs

mkspiffs -c ~/Arduino/ds18b20/data/ -p 256 -b 8192 -s 3125248 /tmp/out.spiffs
esptool -cd nodemcu -cb 460800 -cp /dev/ttyUSB0 -ca 0x100000 -cf /tmp/out.spiffs
bertrandlo commented 5 years ago

@scottchiefbaker i try the command for 1MB but the error message was shown 

"error: Image size should be a multiple of block size"

do you get the same result ??

scottchiefbaker commented 5 years ago

No the commands are working for me:

mkspiffs -c /home/bakers/Arduino/NodeWebServerGZ/data/ --page 256 --block 8192 -s 1028096 /tmp/NodeWebServerGZ.spiffs

Gives me a .spiffs files and no errors.

canadaduane commented 4 years ago

This is my working configuration (Mac OS X Mojave, esp8266 with 2MB filesystem):

(Side note: Has esptool changed to esptool.py and did its arguments change as well? I couldn't find -cd, -cb, -cp, -ca options.)

# Compile https://github.com/igrr/mkspiffs
mkspiffs -c ./firmware/tictactoe/data \
  --size 2072576 \
  --page 256 \
  --block 8192 -- \
  spiffs.bin

# Install via `brew install esptool`
esptool.py --chip esp8266 \
  --port /dev/cu.usbserial-1410 \
  --baud 460800 \
  --before default_reset \
  --after hard_reset \
  write_flash 0x200000 spiffs.bin

# Configured from parameters shown in Arduino IDE:
#
# data    : /Users/duane/Dropbox/tictactoe/firmware/tictactoe/data
# size    : 2024
# page    : 256
# block   : 8192
# /player.txt
# upload  : /var/folders/bl/vmhly6rn6kbd7hphg614m4b80000gn/T/arduino_build_276330/tictactoe.spiffs.bin
# address  : 0x200000
# reset    : --before default_reset --after hard_reset
# port     : /dev/cu.usbserial-1410
# speed    : 921600
# python   : python
# uploader : /Users/duane/Library/Arduino15/packages/esp8266/hardware/esp8266/2.6.1/tools/upload.py

Notes:

wolfen351 commented 4 years ago

I also encountered the "error: Image size should be a multiple of block size" error.

I adjusted the 2Mb command like this:

calc 8192*256 2097152

So I tried that number in the command:

~/mkspiffs -c data/ -p 256 -b 8192 -s 2097152 filesystem.spiffs And it created the filesystem.

neinseg commented 3 years ago

Here is an example of a Makefile that automatically extracts the SPIFFS properties from arduino's build settings:


sketch          := foo.ino
boardconfig     := esp8266:esp8266:generic:eesz=4M1M

ARDUINO_CLI ?= arduino-cli
MKSPIFFS    ?= mkspiffs
BC          ?= bc

all: build

.PHONY: build
build:
        $(ARDUINO_CLI) compile --fqbn $(boardconfig) $(sketch)

.ONESHELL:
filesystem.bin:
        PROPS=$$($(ARDUINO_CLI) compile --fqbn $(boardconfig) --show-properties)
        BUILD_SPIFFS_BLOCKSIZE=$$(echo "$$PROPS"|grep "^build.spiffs_blocksize"|cut -d= -f2)
        BUILD_SPIFFS_END=$$(echo "$$PROPS"|grep "^build.spiffs_end"|cut -d= -f2)
        BUILD_SPIFFS_PAGESIZE=$$(echo "$$PROPS"|grep "^build.spiffs_pagesize"|cut -d= -f2)
        BUILD_SPIFFS_START=$$(echo "$$PROPS"|grep "^build.spiffs_start"|cut -d= -f2)
        BUILD_SPIFFS_SIZE=$$(echo "ibase=16;$${BUILD_SPIFFS_END:2}-$${BUILD_SPIFFS_START:2}"|bc -q)
        echo BUILD_SPIFFS_SIZE $$BUILD_SPIFFS_SIZE
        $(MKSPIFFS) -c data -p $$BUILD_SPIFFS_PAGESIZE -b $$BUILD_SPIFFS_BLOCKSIZE -s $$BUILD_SPIFFS_SIZE $@

.PHONY: flash
flash: filesystem.bin

.PHONY: flash-fs
flash-fs:

.PHONY: clean
        rm -rf build
        rm -f filesystem.bin
weshouman commented 3 years ago

Thanks @neinseg for the Makefile
I've had to do the following modifications to make it work

And the following modifications to support ESP32 instead of ESP8266

sketch      := foo.ino
CORE        :=esp32:esp32
# Flashing on  an "ESP32 Dev Module" board
boardconfig :="${CORE}:esp32"

ARDUINO_CLI ?= arduino-cli
MKSPIFFS    ?= mkspiffs
BC          ?= bc

PARTITION_TABLE=/root/.arduino15/packages/esp32/hardware/esp32/1.0.6/tools/partitions/default.csv

DEVICE :=/dev/ttyUSB0

.PHONY: build
build:
    $(ARDUINO_CLI) compile --fqbn $(boardconfig) $(sketch)

.PHONY: flash
flash:
    $(ARDUINO_CLI) upload -p ${DEVICE} --fqbn ${boardconfig} ${sketch} 

.PHONY: filesystem.bin
.ONESHELL:
filesystem.bin:
    PROPS=$$($(ARDUINO_CLI) compile --fqbn $(boardconfig) --show-properties)
    BUILD_SPIFFS_BLOCKSIZE=4096
    BUILD_SPIFFS_PAGESIZE=256
    BUILD_SPIFFS_START_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d, -f4 | xargs)
    BUILD_SPIFFS_START=$$(echo "ibase=16;$${BUILD_SPIFFS_START_HEX:2}"|bc -q)
    echo "BUILD_SPIFFS_START $$BUILD_SPIFFS_START_HEX ($$BUILD_SPIFFS_START)"
    BUILD_SPIFFS_SIZE_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d, -f5 | xargs)
    BUILD_SPIFFS_SIZE=$$(echo "ibase=16;$${BUILD_SPIFFS_SIZE_HEX:2}"|bc -q)
    echo "BUILD_SPIFFS_SIZE  $$BUILD_SPIFFS_SIZE_HEX ($$BUILD_SPIFFS_SIZE)"
    $(MKSPIFFS) -c data -p $$BUILD_SPIFFS_PAGESIZE -b $$BUILD_SPIFFS_BLOCKSIZE -s $$BUILD_SPIFFS_SIZE $@

.PHONY: flash-fs
.ONESHELL:
flash-fs: filesystem.bin
    BUILD_SPIFFS_START_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d, -f4 | xargs)
    python /root/.arduino15/packages/esp32/tools/esptool_py/3.0.0/esptool.py --chip esp32 \
      --port ${DEVICE} \
      --baud 460800 \
      --before default_reset \
      --after hard_reset \
      write_flash $${BUILD_SPIFFS_START_HEX} filesystem.bin

.PHONY: clean
clean:
    rm -rf build
    rm -f filesystem.bin
arifainchtein commented 1 year ago

Hello, I am trying to do exactly the same, but i keep getting problems I have a custom made pcb using Esp32 WROOM D32. If i use the Arduino IDE i can upload the files and everything works fine.

However, when i use the commands to upload and i run the code, i always get the error:

SPIFFS mount failed -10025

My script has two commands. The first one creates the partition:

/home/ari/.arduino15/packages/esp32/tools/mkspiffs/0.2.3/mkspiffs -c /home/ari/Data/DigitalStables/Projects/pancho-tank-flow/data/ --page 256 --block 8192 --size 2686976 /tmp/out.spiffs The second command uploads the file:

/home/ari/.arduino15/packages/esp32/tools/esptool_py/4.2.1/esptool.py --chip esp32 \ --port /dev/ttyUSB0 \ --baud 921600 \ --before default_reset \ --after hard_reset \ write_flash 0x00290000 /tmp/out.spiffs The size parameters i got from seeing the arduino ide 1 console as i do the upload.

Any suggestions would be very much appreciated,

dlarue commented 1 year ago

@arifainchtein did you figure out was was missing? I'm trying to use IDE v2.0.3 and just found out there's no SPIFFS upload tool so I too am resorting to commandline spiffs image building and esptool.py uploading but get the same mount failed -10025 error.

dlarue commented 1 year ago

I could not figure out the makefile enough to get it to generate the proper values so I went to bash and ran the commands then hardcoded the makefile variables. From there filesystem.bin and flash-fs both worked with the source compiled in the IDE v2.0.3.

build and flash both ran from the makefile but the result was a crashing upload similar to what I see when the IDE is set to flash freq of 80MHz instead of 40.

My IDE settings for the esp-wroom-32 module is:

CPU Freq=240MHz Core Debug Level=NONE Flash Freq=40MHz Flash Mode=QIO Flash Size=4MB Partition Scheme=Default 4MB with spiffs(1.2MB app/1.5MB spiffs)

Makefile: `sketch := foo.ino CORE :=esp32:esp32 boardconfig :="${CORE}:esp32"

ARDUINO_CLI ?= arduino-cli MKSPIFFS ?= mkspiffs BC ?= bc

PARTITION_TABLE=~/.arduino15/packages/esp32/hardware/esp32/2.0.5/tools/partitions/default.csv

DEVICE :=/dev/ttyUSB0

.PHONY: build build: $(ARDUINO_CLI) compile --fqbn $(boardconfig) $(sketch)

.PHONY: flash flash: $(ARDUINO_CLI) upload -p ${DEVICE} --fqbn ${boardconfig} ${sketch}

.PHONY: filesystem.bin .ONESHELL: filesystem.bin: PROPS=$$($(ARDUINO_CLI) compile --fqbn $(boardconfig) --show-properties) BUILD_SPIFFS_BLOCKSIZE=4096 BUILD_SPIFFS_PAGESIZE=256 BUILD_SPIFFS_START_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d"," -f4 | xargs) BUILD_SPIFFS_START_HEX=cat $(PARTITION_TABLE) BUILD_SPIFFS_START_HEX="0x290000" BUILD_SPIFFS_START=$$(echo "ibase=16;$${BUILD_SPIFFS_START_HEX:2}"|bc -q) BUILD_SPIFFS_START=0 echo "BUILD_SPIFFS_START $$BUILD_SPIFFS_START_HEX ($$BUILD_SPIFFS_START)" BUILD_SPIFFS_SIZE_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d, -f5 | xargs) BUILD_SPIFFS_SIZE_HEX="0x170000" BUILD_SPIFFS_SIZE=$$(echo "ibase=16;$${BUILD_SPIFFS_SIZE_HEX:2}"|bc -q) BUILD_SPIFFS_SIZE=1507328 echo "BUILD_SPIFFS_SIZE $$BUILD_SPIFFS_SIZE_HEX ($$BUILD_SPIFFS_SIZE)" $(MKSPIFFS) -c data --page $$BUILD_SPIFFS_PAGESIZE --block $$BUILD_SPIFFS_BLOCKSIZE --size $$BUILD_SPIFFS_SIZE $@

.PHONY: flash-fs .ONESHELL: flash-fs: filesystem.bin BUILD_SPIFFS_START_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d, -f4 | xargs) python ~/.arduino15/packages/esp32/tools/esptool_py/4.2.1/esptool.py --chip esp32 \ --port ${DEVICE} \ --baud 460800 \ --before default_reset \ --after hard_reset \ write_flash $${BUILD_SPIFFS_START_HEX} filesystem.bin

.PHONY: clean clean: rm -rf build rm -f filesystem.bin `

arifainchtein commented 1 year ago

@dlarue I sort of gave up trying to automate that part of the build and I upload manually using the Arduino IDE 1.8. A bit of a pain but it works. Would love to automate that process and will revisit this from time to time as other priorities allow. Will share here if I have succcess

dlarue commented 1 year ago

I'd posted about this process to an esp32-arduino forum and was told that LittleFS was the supported filesystem now and for about 2 years SPIFFS is in holding pattern. There seems to be some advantage of a real hierarchical filesystem, a bit faster for only a little bit more memory. I've not tried LittleFS yet but might be something to look into.

ssshake commented 1 year ago

Any idea how to do this make file for an esp8266? For example this partition table csv does not exist for the esp8266.

/esp32/hardware/esp32/2.0.5/tools/partitions/default.csv

So I'm not sure how to adapt the filesystem.bin command.

dlarue commented 1 year ago

from what I can figure out, you can create your own default.csv if you set it to the data you need for the esp8266.

From: https://github.com/esp8266/Arduino/issues/2320#issuecomment-236485462 you see this: mkspiffs -p 256 -b 8192 -s $((0x3FB000 - 0x300000)) -c path/to/files/dir/ spiffs-image.bin

and since the default.csv looks like this maybe you can modify it? Or do what's mentioned above.

Name, Type, SubType, Offset, Size, Flags

nvs, data, nvs, 0x9000, 0x5000, otadata, data, ota, 0xe000, 0x2000, app0, app, ota_0, 0x10000, 0x140000, app1, app, ota_1, 0x150000,0x140000, spiffs, data, spiffs, 0x290000,0x170000,

And then there is this discussion and examples: https://github.com/esp8266/arduino-esp8266fs-plugin/issues/51

Deepakkumar2102 commented 1 year ago

@canadaduane Hi i am generating same spiffs image as you mention, but it not working for me .