Release 3.0.0 examples fail on Metro M4 Express, but 2.0.2 examples work just fine

[microscope]

As posted on https://forums.adafruit.com/viewtopic.php?f=63&t=154063:

With the latest version 3.0.0 of the Adafruit_SPIFlash library, I am having trouble getting the QSPI flash to work with Arduino on both my Metro M0 Express and Metro M4 Express boards:

• The flash_info and flash_erase_express examples from the Adafruit_SPIFlash library do work.

• The SdFat_format example fails like this: Adafruit SPI Flash FatFs Format Example Flash chip JEDEC ID: 0xC84015 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! This sketch will ERASE ALL DATA on the flash chip and format it with a new filesystem! Type OK (all caps) and press enter to continue. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Failed to wipe

• The flash_manipulator example fails: Adafruit Serial Flash Manipulator example JEDEC ID: C84015 Flash size: 2097152 Initializing SD card... Card failed, or not present

• The SdFat_full_usage example fails (of course, because the QSPI flash is not formatted): Adafruit SPI Flash FatFs Full Usage Example Flash chip JEDEC ID: 0xC84015 Error, failed to mount newly formatted filesystem! Was the flash chip formatted with the fatfs_format example?

It does not seem to be a hardware issue: I can restore the QSPI flash as CIRCUITPY drive for use with CircuitPython by double-clicking the reset button to enter the UF2 boot loader and dragging the appropriate file (today adafruit-circuitpython-metro_m4_express-en_US-4.0.2.uf) onto the METROBOOT drive.

Libraries are up-to-date:

• Arduino IDE 1.8.9 with standard libraries up-to-date (7/13/19)
• Adafruit SPIFlash version=3.0.0
• SdFat version=1.0.16 from the Adafruit fork at https://github.com/adafruit/SdFat

After some further digging ....

With version 2.0.2 from https://github.com/adafruit/Adafruit_SP ... /tag/2.0.2, all the important examples work with my Metro M4 Express board: fatfs_circuitpython, fatfs_datalogging, fatfs_format, fatfs_full_usage, fatfs_print_file, flash_erase, and flash_erase_express. (SPI_Flash_Manipulator fails looking for an SD card, which I don't have; I didn't try circuitpython_backupFiles).

Since 2.0.2 is deprecated, I'd rather get 3.0.0 to work (or help with getting it to work), but I am not sure where to start.

Thanks,

microscope

[ladyada]

yeah we did a bunch of refactoring (for good reason and also its way better now) but we are still fixing up stuff. @hathach please take a look :)

[hathach]

Just tested with Metro M0 and M4, it still works here. @microscope can you download the current master of https://github.com/adafruit/SdFat as zip and replace your local library. If the sketches still fails, try to enable SdFat DEBUG_MODE here to 1 and post your serial output here. https://github.com/adafruit/SdFat/blob/master/src/FatLib/FatVolume.h#L38

[microscope]

@hathach - In short, no joy.

• With both SdFat (https://github.com/adafruit/SdFat, version=1.1.0) and Adafruit_SPIFlash (https://github.com/adafruit/Adafruit_SPIFlash, version=3.0.1) freshly downloaded from Github on 7/14/19 at about 3:20 pm and manually installed in ~/Documents/Arduino/libraries/ folder

• Hardware setup:

  • bare Metro M4 Express (purple), no other components, BN: Adafruit Metro M4 (SAMD51) VID: 0x239A PID: 0x8020 SN: Upload any sketch to obtain it
  • Arduino IDE 1.8.9 with Boards Manager ... and Manage Libraries ... all up-to-date
  • MacBook Pro 15,1 with macOS High Sierra 10.13.6
  • Metro connected via USB cable to SATECHI Type-C Multi-Port Adapter 4K with Ethernet

• First, re-install CircuitPython (adafruit-circuitpython-metro_m4_express-en_US-4.0.2.uf2) with UF2 bootloader on METROBOOT because that's where I had started in the first place -- I had switched from programming the Metro M4 with CircuitPython to Arduino -- and that may affect the QSPI flash

• Testing the examples under Arduino:File > Examples > Adafruit SPI Flash > ...

  • flash_info (3.0.1) works: Adafruit Serial Flash Info example JEDEC ID: C84015 Flash size: 2097152

  • SDFat_full_usage (3.0.1) fails: Adafruit SPI Flash FatFs Full Usage Example Flash chip JEDEC ID: 0xC84015 Error, failed to mount newly formatted filesystem! Was the flash chip formatted with the fatfs_format example?

  • SdFat_format (3.0.1) fails: Adafruit SPI Flash FatFs Format Example Flash chip JEDEC ID: 0xC84015 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! This sketch will ERASE ALL DATA on the flash chip and format it with a new filesystem! Type OK (all caps) and press enter to continue. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Failed to wipe

  • flash_erase_express (3.0.1) works (green light about every second)

  • SdFat_format (3.0.1) fails again like above

• Following hathach's instructions, changed ~/Documents/Arduino/libraries/SdFat/src/FatLib/FatVolume.hm line 38 from '#define DEBUG_MODE 0' to '#define DEBUG_MODE 1'

  • flash_info (3.0.1) still works: Adafruit Serial Flash Info example JEDEC ID: C84015 Flash size: 2097152

  • SDFat_full_usage (3.0.1) fails again: Adafruit SPI Flash FatFs Full Usage Example Flash chip JEDEC ID: 0xC84015 /Users/kollmar/Documents/Arduino/libraries/SdFat/src/FatLib/FatVolume.cpp478 /Users/kollmar/Documents/Arduino/libraries/SdFat/src/FatLib/FatVolume.cpp493 Error, failed to mount newly formatted filesystem! Was the flash chip formatted with the fatfs_format example?

• At this point, I noted that the error message asks to run 'fatfs_format', but there is no such example for the Adafruit_SPIFlash 3.0.x library, only 'SdFat_format'.

• Tried running the fatfs_format code from 2.0.2 under the 3.0.x library, but that fails to compile because of the new dependency on the SdFat library.

• So I reverted to the 2.0.2 library to format the QSPI flash with fatfs_format and verified that the formating was successful:

  • fatfs_format (2.0.2) works: Adafruit SPI Flash FatFs Format Example Flash chip JEDEC ID: 0xC84015 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! This sketch will ERASE ALL DATA on the flash chip and format it with a new filesystem! Type OK (all caps) and press enter to continue. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Partitioning flash with 1 primary partition... Partitioned flash! Creating and formatting FAT filesystem (this takes ~60 seconds)... Formatted flash! Flash chip successfully formatted with new empty filesystem!

  • fats_full_usage (2.0.2) works: Adafruit SPI Flash FatFs Full Usage Example Flash chip JEDEC ID: 0xC84015 Mounted filesystem! Test directory not found, creating... Created test directory! Creating deep folder structure... Created /test/foo/bar and /test/foo/baz folders! Opened file /test/test.txt for writing/appending... Wrote to file /test/test.txt! First line of test.txt: Hello world! Total size of test.txt (bytes): 57 Current position in test.txt: 14 Available data to read in test.txt: 43 File name: test.txt Is file a directory? No Entire contents of test.txt: Hello world! Hello number: 123 Hello hex number: 0x7B Listing children of directory /test:

    • foo (directory)
    • test.txt Deleting /test/foo/test2.txt... Deleted file! Deleting /test directory and everything inside it... Test directory was deleted! Finished!

• And finally I updated again to 3.0.1 and tried its examples again with the freshly formated QSPI flash:

  • flash_info (3.0.1) still works: Adafruit Serial Flash Info example JEDEC ID: C84015 Flash size: 2097152

  • SdFat_full_usage (3.0.1) fails again: Adafruit SPI Flash FatFs Full Usage Example Flash chip JEDEC ID: 0xC84015 /Users/kollmar/Documents/Arduino/libraries/SdFat/src/FatLib/FatVolume.cpp493 /Users/kollmar/Documents/Arduino/libraries/SdFat/src/FatLib/FatVolume.cpp493 Error, failed to mount newly formatted filesystem! Was the flash chip formatted with the fatfs_format example?

  • SdFat_format (3.0.1) fails again: Adafruit SPI Flash FatFs Format Example Flash chip JEDEC ID: 0xC84015 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! This sketch will ERASE ALL DATA on the flash chip and format it with a new filesystem! Type OK (all caps) and press enter to continue. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! /Users/kollmar/Documents/Arduino/libraries/SdFat/src/FatLib/FatVolume.cpp493 /Users/kollmar/Documents/Arduino/libraries/SdFat/src/FatLib/FatVolume.cpp493 /Users/kollmar/Documents/Arduino/libraries/SdFat/src/FatLib/FatVolume.cpp555 Failed to wipe

So I am stumped.

[hathach]

@microscope thank you for your detail information, I am able to reproduce the issue after running the flash_erase_express example which erase the whole chip. Troubleshooting it now.

UPDATE: look like SdFat's wipe only erase existing system, and doesn't create a new one filesystem if not existed.

[ChristieLeeEMT]

yeah we did a bunch of refactoring (for good reason and also its way better now) but we are still fixing up stuff. @hathach please take a look :)

So, is this why I'm getting the below errors? Suddenly my Hallowing isn't running the full test code at all. I tried to run the SDFat sketches that @microscope did, and they all failed with the same error.

Arduino: 1.8.9 (Windows 10), Board: "Adafruit Hallowing M0, Arduino, Off"

In file included from C:\Users\Lisa\Documents\Hocus Pocus\Software\hallowing_full_test\full_test\full_test.ino:16:0:

C:\Users\Lisa\Documents\Arduino\libraries\Adafruit_SPIFlash\src/Adafruit_SPIFlash.h:39:4: error: #error ENABLE_EXTENDED_TRANSFER_CLASS must be set to 1 in SdFat SdFatConfig.h

error ENABLE_EXTENDED_TRANSFER_CLASS must be set to 1 in SdFat SdFatConfig.h

^

C:\Users\Lisa\Documents\Arduino\libraries\Adafruit_SPIFlash\src/Adafruit_SPIFlash.h:43:4: error: #error FAT12_SUPPORT must be set to 1 in SdFat SdFatConfig.h

error FAT12_SUPPORT must be set to 1 in SdFat SdFatConfig.h

^

exit status 1 Error compiling for board Adafruit Hallowing M0.

This report would have more information with "Show verbose output during compilation" option enabled in File -> Preferences.

[microscope]

@ChristieLeeEMT: The errors you describe are a different issue (failure to compile) and easy to fix - as the messages say, you need to edit two lines (change 0 to 1) in the SdFatConfig.h file of the SDFat library installed on your system. I also got these errors early on when I used the SdFat library from its original creator (https://github.com/greiman/SdFat). But the Adafruit fork of the SdFat library (https://github.com/adafruit/SdFat) has these changes already incorporated.

Making these changes or using the Adafruit fork of the SdFat library allowed me to successfully compile the examples from the SPIFlash 3.0.0/1 library. But the examples still failed to run, as describe above.

[S2Doc]

Same problem and same results as @microscope using an M4 Feather Express. Of the example code: Compiles and appears to work: flash_erase

Compiles but give errors (as reported by @microscope above): flash_manipulator SDFat_full_usage SDFat_ReadWrite SDFat_datalogging SDFat_print_file

[ChristieLeeEMT]

@microscope I'll have to try the Adafruit version. BUT, when I got the board 2 weeks ago, I didn't include any sdfat libraries, and the full test compiled and ran. The only thing I had a problem with was sound. I got a mini speaker (cuz I was told I needed one for sound), and when I tried it again, the compiler was asking for sdfat.

[ChristieLeeEMT]

@microscope I tried the Adafruit SDFat, and NG. I no longer have SDFat errors, but now it didn't recognize any of the calls to SPIFlash. Backing up to Adafruit SPIFlash v2.0.2 (even without SDFat), it's working again. TY for the help.

[hathach]

@microscope It should be fixed by PR #15 . It is kind of complicated and take more time than expected to troubleshoot. Short story: SdFat is not fully support FAT12. Its library only provide an formatter example for FAT16/FAT32 but FAT12. I have to use the good old f_mkfs() from Elm Chan's fatfs for SdFat_format example.

To make thing worst, SdFat insist number of FAT table must be 2, while by default f_mkfs() only create 1 FAT https://github.com/greiman/SdFat/blob/master/src/FatLib/FatVolume.cpp#L490

Anyway, please try to run SdFat_format example in the PR, once it is probably formatted, it should work well as before. When you could verify it works. We can merge and release it :)

[hathach]

Note: This issue only occurs if you erased the flash contents with flash_erase/flash_erase_express and haven't formatted it with appropriate disk management like windows/linux via USB MSC external flash example sketch.

Since Circuitpython also format using f_mkfs() with 1 FAT table, it also didn't help with SdFat insisting with 2 FAT

[microscope]

@hathach: It all works now!

• Dealing with pull requests is beyond my GitFu, but LadyAda just merged PR #15 into master seconds before I downloaded it.

• With the latest Adafruit_SPIFlash master from 7/15/19 on the Metro M4 Express:

  • flash_info still works

  • SdFat_format now works! Adafruit SPI Flash FatFs Format Example Flash chip JEDEC ID: 0xC84015 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! This sketch will ERASE ALL DATA on the flash chip and format it with a new filesystem! Type OK (all caps) and press enter to continue. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! /Users/kollmar/Documents/Arduino/libraries/SdFat/src/FatLib/FatVolume.cpp493 /Users/kollmar/Documents/Arduino/libraries/SdFat/src/FatLib/FatVolume.cpp493 Failed to init filesystem Creating and formatting FAT filesystem (this takes ~60 seconds)... Formatted flash! /Users/kollmar/Documents/Arduino/libraries/SdFat/src/FatLib/FatVolume.cpp478 Flash chip successfully formatted with new empty filesystem!

  • SdFat_full_usage now works!

  • SdFat_ReadWrite now works!

  • SdFat_datalogging now works!

  • SdFat_print_file now works!

• Testing after various erasures:

  • After flash_erase succeeds, SdFat_full_usage fails as it should. Then, after SdFat_format succeeds, SdFat_full_usage also succeeds as it should.

  • After flash_erase_express succeeds, SdFat_full_usage fails as it should. Then, after SdFat_format succeeds, SdFat_full_usage also succeeds as it should.

  • Reinstalling CircuitPython with UF2 bootleader succeeds. [Oddly enough, the drive shows up as "NO NAME" after teh bootloadin and is not recognized by the Mu editor. After renaming to "CIRCUITPY", as described in the guide (https://learn.adafruit.com/welcome-to-circuitpython?view=all), Mu recognizes the drive, and code.py saved to the drive runs fine.] After switching back to Arduino, SdFat_full_usage succeeds; this is unexpected (I thought the CircuitPython installation hoses the formating), but a good thing. Then both SdFat_format and SdFat_full_usage also succeed.

• For my edification: You talk about "appropriate disk management like windows/linux via USB MSC external flash example sketch." Is this another, non-Arduino-based tool that could be used instead of the now fixed SdFat_format example to format the QSPI flash? Where do I find that?

Thank you so much for fixing this!

[hathach]

For my edification: You talk about "appropriate disk management like windows/linux via USB MSC external flash example sketch." Is this another, non-Arduino-based tool that could be used instead of the now fixed SdFat_format example to format the QSPI flash? Where do I find that?

this sketch example will expose qspi flash as USB Mass Storage, you could then format it with windows/Linux by right click on the disk

[hathach]

UPDATE: we have changed the SdFat code to support bot 1 and 2 FAT tables. Once both of these PR is merged and release, things will work in harmony ( with circuitpython ) and existing board without running through SdFat_format. Thank you very much you all for reporting this issue.

https://github.com/adafruit/SdFat/pull/3 https://github.com/adafruit/Adafruit_SPIFlash/pull/16

[microscope]

@hathach @ladyada : Glad to be able to contribute a tiny bit. I greatly value Adafruit's tutorials and libraries--that's why I have been a customer for more than a decade. Cheers!