File System Boot Problems EIO / EPERM

[pacmac]

After a long pause, I decided today to take a look at the esp32 and updated & re-compiled using the latest.

esptool.py v2.0-beta3
Connecting........_____.....___
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Erasing flash (this may take a while)...
Chip erase completed successfully in 10.9s
Hard resetting...
@@ Updating Firmware....
esptool.py v2.0-beta3
Connecting.....
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Compressed 990352 bytes to 569786...
Wrote 990352 bytes (569786 compressed) at 0x00000000 in 14.4 seconds (effective 550.2 kbit/s)...
Hash of data verified.

I am able to flash the devices (ESPWROOM & ESP-THING) and are able to get a repl prompt, however I am unable to access the file systems on either device, and during boot I get the following errors on both devices:

>>> ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0x00
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0008,len:8
load:0x3fff0010,len:3432
load:0x40078000,len:9492
load:0x40080000,len:252
entry 0x40080034
I (1285) cpu_start: Pro cpu up.
I (1285) cpu_start: Single core mode
I (1287) heap_alloc_caps: Initializing. RAM available for dynamic allocation:
I (1300) heap_alloc_caps: At 3FFAE2A0 len 00001D60 (7 KiB): DRAM
I (1320) heap_alloc_caps: At 3FFD5258 len 0000ADA8 (43 KiB): DRAM
I (1341) heap_alloc_caps: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (1362) heap_alloc_caps: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (1384) heap_alloc_caps: At 400922DC len 0000DD24 (55 KiB): IRAM
I (1405) cpu_start: Pro cpu start user code
I (1464) cpu_start: Starting scheduler on PRO CPU.
Traceback (most recent call last):
  File "_boot.py", line 12, in <module>
  File "inisetup.py", line 28, in setup
  File "inisetup.py", line 6, in check_bootsec
  File "flashbdev.py", line 13, in readblocks
OSError: [Errno 5] EIO
OSError: [Errno 1] EPERM
OSError: [Errno 1] EPERM
MicroPython v1.8.7-955-ga64dc827 on 2017-06-10; ESP32 module with ESP32
Type "help()" for more information.
>>> import os
>>> os.listdir()
[]
>>> 

As I have not done this for a while, I guess that I may have done something wrong, but I have been through it a few times and can't seem to find any problem ?

If I use the pre-compiled version, everything is fine, but I am then unable to use my frozen code ?!

Any suggestions ??

[pacmac]

I am pretty sure i know the cause , my firmware.bin file is 6k smaller than the precompiled version, and i notice that on 8266 the file is named firmware-combined.bin

So how can i create a firmware-combined file instead of just a firmware.bin file ?

[nickzoic]

There's a Make file rule which combines three parts into one for easier writing ... I havent got the code in front of me but maybe that's enough of a hint, I'll fill in the details tonight :-). Also might be worth clearing whole flash before flashing newer version.

On Sat, Jun 10, 2017, at 15:05, pacmac wrote:

I am pretty sure i know the cause , my firmware.bin file is 6k smaller than the precompiled version, and i notice that on 8266 the file is named firmware-combined.bin> So how can i create a firmware-combined file instead of just a firmware.bin file ?> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub[1], or mute the thread[2].>

-- Nick Moore nick@zoic.org 0409 656 267

Links:

1. https://github.com/micropython/micropython-esp32/issues/109#issuecomment-307543116
2. https://github.com/notifications/unsubscribe-auth/ABazRn9Prp8lXu7VHoFioMDin4mBdKzhks5sCiQsgaJpZM4N1_qP

[pacmac]

Thanks, thought there may be, i am also not at my computer now but i guess this is defined in the 8266 makefile so i can copy ?!6

[pacmac]

OK, the 6kb difference was wrong, I was comparing different verssions.

I have completely rebuilt the entire build setup several times, and have tried building several different recent commits, every one fails during boot with the same error that appears to be related to the file system.

esp32-20170610-v1.8.7-964-g62d40e8b.bin (995,792 bytes)
firmware.bin (995,792 bytes)

As you can see, the file size of the file I compiled is identical to the pre-compiled one, the pre-compiled does not have an error, the one I compiled does.

I have spent the whole day on this and tried everything I can think of. As a sanity check, could someone else try a build / deploy and see if they are successful as I really don't know what to do next:

Traceback (most recent call last):
  File "_boot.py", line 10, in <module>
  File "inisetup.py", line 28, in setup
  File "inisetup.py", line 6, in check_bootsec
  File "flashbdev.py", line 13, in readblocks
OSError: [Errno 5] EIO
OSError: [Errno 1] EPERM
OSError: [Errno 1] EPERM
MicroPython v1.8.7-964-g62d40e8b on 2017-06-10; ESP32 module with ESP32
Type "help()" for more information.
>>> import os
>>> os.listdir()
[]
>>> import flashbdev
>>> import os
>>> os.VfsFat.mkfs(flashbdev.bdev)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "flashbdev.py", line 18, in writeblocks
OSError: [Errno 5] EIO
>>> 

[dpgeorge]

The "combined" firmware image is called esp32/build/firmware.bin. Did you try completely erasing the flash before copying across the firmware?

[pacmac]

Hi Damien, spent all weekend on this.

1. I always do a full erase before I re-flash

2. I am using identical flash settings and procedure for the pre-compiled file and the one I compiled.

3. Both files are exactly the same size in bytes.

4. The pre-compiled file and my file produce exactly the same output when flashing, erase time is approx 10 seconds using both files and flash time is the same.

5. I have completely rebuilt the entire build folders several times (esp-idf, xtensa, micropython-esp32) just to be sure I have not messed something up, the result is exactly the same.

6. No problem at all with compiling and flashing the ESP8266, but the ESP32 refuses to work.

7. I have tested on both ESP32-Thing and on ESP-WROOM - the error is the same on both devices.

8. I am sure that my flash settings are correct as if they were not, then the pre-compiled version would not work, but it is only the version that i compiled that shows the error after booting (see above).

[dpgeorge]

Both files are exactly the same size in bytes.

But are they exactly the same in contents, every byte?

The esp32 builds provided at micropython.org/download are compiled using the following version of the compiler: xtensa-esp32-elf-gcc (crosstool-NG crosstool-ng-1.22.0-61-gab8375a) 5.2.0. It uses 4MB for the flash size and 40MHz for the flash SPI speed.

Also, make sure did git submodule update in the ESP IDF, after checking out the relevant commit that esp32/Makefile needs.

[pacmac]

I have not compared byte by byte, never had to do that before !

I am using the default values for flash size and spi speed (in the makefile)

Here's a list of the versions I have used:

esp-idf: 1e0710f1b24429a316c9c34732aa17bd3f189421 micropython-esp32: 62d40e8b9e5b62ed27445a8f50688537575d97d1 xtensa: xtensa-esp32-elf-osx-1.22.0-61-gab8375a-5.2.0

Yes, I did a git module update (I have bash scripts to reduce repetitive tasks and ensure repeatability)

function idf-update {
  cd "$ROOTDIR/build/esp-idf"

  if [ -n "$1" ]; then
    IDFBRANCH=$1
  fi

  echo; echo "@@@ Updating esp-idf ($IDFBRANCH) ..."
  git checkout $IDFBRANCH
  git submodule update --init

}

[dpgeorge]

I have not compared byte by byte, never had to do that before !

Just use "diff", it will tell you if they are the same or not.

I actually ran into this exact problem just now. The problem was (for my situation) that I had incorrectly set FLASH_SIZE = 2M and it seems that the ESP IDF now honors this setting and won't allow you to access flash above this value, hence all the EIO/EPERM errors. The solution (for me) was to do "make erase", fix the FLASH_SIZE to 4M, and rebuild. Then it worked and the old filesystem was still there.

[pacmac]

Thanks for following up.

What I omitted to mention (not intentionally) is that the folders are on a nfs drive, with the 8266 this was not a problem, but it may also be that the esp-32 has different requirements that cause the build to fail.

I decided to create a custom build environment with a fresh ARCH LINUX VM, as I have a suspicion that my problems maybe somehow related to OSX's ever increasing security / control over files.

After doing this and building the required folders and compiling, there is no such error anymore and the build is just fine.

Your explanation sounds like a very logical cause, but for sure mine is set to 4M, I may never find out the cause but the good news is everything is fine in my Arch builder.

[pacmac]

Hi Damien;

I have re-opened this just to clarify this:

"... the ESP IDF now honours this setting" - are you saying that these settings are actually used by the build process and get coded into the firmware file and not just for the flashing of the device (pytool.py) ?

If so, wouldn't that means that you will need to create a separate build for each device depending upon it's flash size, or if specifying some arbitrary low value, potentially cripple devices that have bigger flash sizes ?

[dpgeorge]

are you saying that these settings are actually used by the build process and get coded into the firmware file and not just for the flashing of the device

Yes, it seems that the flash mode, size and frequency are written to the 2nd and 3rd bytes of the bootloader.bin and application.bin files (and so form part of the final firmware). And I guess the IDF changed recently to actually read and use these values.

wouldn't that means that you will need to create a separate build for each device depending upon it's flash size

Yes. There will need to be separate configurations and builds for each board.

[pacmac]

Then it would be possible for the mode, size and frequency set in the .bin file to conflict with the flash update command line settings for the same variables ?

I know that you did not design this, but it seems a bit illogical to define these vars in 2 different places, does the flash settings override the firmware.bin settings or vice-versa or do the command line vars now have no effect ?

[dpgeorge]

does the flash settings override the firmware.bin settings or vice-versa or do the command line vars now have no effect

I don't know. You'll need to dig into the source code to find out.

[projectgus]

Hi folks,

Damien asked me about this today. Here's what IDF does, and how it effects Micropython:

• The bootloader .bin file and the app .bin file both have a header which contains some SPI metadata, including the size. Only the bootloader .bin file's header matters at all, the main app's header is ignored.

• The option in the config settings is used when generating these .bin files for what default settings should be placed there. The default is "detect" but you can set other values as well.

• It is also possible for esptool.py to override the values in the bootloader .bin file at the time the .bin file is flashed. The Micropython-ESP32 Makefile does this right now via --flash_size, etc. If you leave the --flash_size argument off, the default for this command line argument is "detect" which instructs esptool.py to try to detect the flash size and update the image accordingly. This works for every flash chip I've ever seen with an ESP8266/ESP32 attached, although it may not work for all flash chips on the market.

• So when using IDF the default is that the bootloader is updated at flashing time with the size of the particular flash it's being written to.

• However, in esptool.py, these metadata values are only applied when a binary file is being flashed at offset 0x1000 (ie the offset of the bootloader). Because Micropython combines all the .bin files into a single binary which is flashed at offset 0x0, esptool.py doesn't apply any of these changes. The bootloader is flashed with whatever header was written to the .bin file when it was built.

• This probably worked for a long time because IDF wasn't bounds-checking SPI flash operations. It's good that we do bounds check SPI flash operations now, because most SPI flash chips just wrap around once you reach the end of their address space!

It's possible to fix this now on the Micropython side by having make_img.py generate an image to be flashed at 0x1000 (the first 4KB at offset 0x0 is reserved for the secure boot digest), and changing the write_flash offset in the esptool.py command in the Makefile to match.

If you'd rather not do this, then I think it's possible for us to fix this on the ESP-IDF/esptool.py side by making esptool.py update the image metadata whenever it is flashing data that looks like a binary image to offset 0x1000, rather than only when that file is being explicitly flashed at offset 0x1000.

[dpgeorge]

Thanks @projectgus for the detailed explanation. Currently we write 0x1000 worth of 0xff values to location 0x0 in the flash, as padding before the bootloader. Is this the "wrong" thing to do? Is it safe and better to not touch the first 4k of flash?

[projectgus]

Is this the "wrong" thing to do? Is it safe and better to not touch the first 4k of flash?

If you're not enabling the ESP32 secure boot feature then the ESP32 ROM bootloader doesn't look at this range, so writing 0xFF here is not going to break anything.

[dpgeorge]

Ok, the idea above was implemented in 96de4d137fa9695df995fd5b237b832dedb5514b : firmware.bin should now be flashed at offset 0x1000 and the flash size can now be auto-detected by esptool.py.

[kaybee335]

Ok, the idea above was implemented in 96de4d1 : firmware.bin should now be flashed at offset 0x1000 and the flash size can now be auto-detected by esptool.py.

I was just setting up a new board (Feather Huzzah ESP32) and was wondering what was going wrong with the flash process until I realised from this post that the flash offset has changed to 0x1000. On a new board the boot after hard reset was falling back to the built-in BASIC interpreter with similar error messages to the above. Loading micropython at the correct offset solved the problem.

If you want to see something amusing, watch adafruit-ampy try to talk to the BASIC interpreter. The interpreter actually responds with phrases like "What?" and "How?". Likely something that only a SW geek would find amusing.... :-)

Thanks Damien!

[dpgeorge]

@kaybee335 good that you got it working! Hopefully the benefits of this change (auto flash size detection, and compatibility with secure boot) outweigh the need to specify 0x1000 as the place to flash the image.