EEPROM.commit() hangs apparently due to rp2040.idleOtherCore() / rp2040.resumeOtherCore()

[ozbotics]

I am having a problem with EEPROM.commit() hanging randomly. Sometimes it works, mostly it doesn't. Be aware that I am also using FreeRTOS and using the PlatformIO environment. See example code below.

In the following code I use EEPROM.end(), but the failure is happening within EEPROM.commit() (called within end())

I have tried various things including running saveConfig() in a separate task running at highest priority, placing vTaskDelay() between calls to EEPROM methods and stepping through/over using the debugger (via picoprobe). Note: it runs fine if I step through it. So maybe there's a timing issue.

I finally tried editing EEPROMClass::commit() to; ... noInterrupts(); // rp2040.idleOtherCore(); flash_range_erase((intptr_t)_sector - (intptr_t)XIP_BASE, 4096); flash_range_program((intptr_t)_sector - (intptr_t)XIP_BASE, _data, _size); // rp2040.resumeOtherCore(); interrupts(); ...

Success! So it seems the problem is in idleOtherCore() / resumeOtherCore() Thankfully I have no tasks running on the other core - maybe I should, to keep this happy?

---

define EEPROM_SIZE 255

define EEPROM_ADDRESS 0

typedef long signed int MotorPosition_t;

struct Config { short configured; MotorPosition_t motorPosition; MotorPosition_t openedPosition; };

void saveConfig(MotorPosition_t motorPosition, MotorPosition_t openedPosition) { Config config; config.configured = ConfigMagicNumber; config.motorPosition = motorPosition; config.openedPosition = openedPosition;

EEPROM.begin(EEPROM_SIZE); EEPROM.put(EEPROM_ADDRESS , config); EEPROM.end(); }

[earlephilhower]

Can you post an MCVE (short complete sketch) that fails? Your snippet isn't enough to go on. Also, are you using Arduino or Platform.IO? And you're using the built-in LittleFS, yes?

One thing to note, it sounds like you're doing lots of EEPROM.commit(). That is very bad, because there is no EEPROM , only flash which is used to emulate it at a fixed address. Depending on the flash vendor you might get 500 to 1000 writes, or maybe 10k writes before it fails and no longer stores data reliably. You should consider using LittleFS which does much better wear leveling over 1000s of 4K pages.

The idle/resume is absolutely required if anything ever runs on the 2nd core, or you will get a bad crash on that core. When the flash is erasing or writing, it can't be read. So if the 2nd core tries to fetch an instruction while the erase/program is ongoing, you'll have a hardware fault.

[earlephilhower]

Also, one possibility is that in your code the 2nd core has already crashed when you try to idle it, so it never responds to the "pause" request. A GDB dump of both thread stacks when it locks up would let you determine this exactly.

[ozbotics]

Hi Earle,

Thanks for your response. I'm using PlatformIO and not using any FS. I am only accessing the flash via the EEPROM interface.

Yes I know that EEPROM emulation on flash has nothing like the repeatability of actual EEPROM and believe me, if there was actual EEPROM available (without resorting to an external chip) I'd use it.

I wasn't suggest that the idle/resume should be removed, just indicating that in my use case that part seemed to cause the failure.

I'll work on an MCVE, I'll have to do some serious surgery to get there. I'll get back to you when I have it ready (and have figured out how to do a GDB dump, this is my "first day" with hardware debugging).

[earlephilhower]

Great, thx. I meant to say "make sure you are using the included FreeRTOS" not LittleFS, sorry.

The pausing the other core bit is very simple and conceptually it could only be trouble if a) the other core has crashed or b) the other core is stuck in no-interrupts forever (which is the same as (a) really)...

Other possibility is the other core's FIFO lost the "go to sleep" message. That would be a logic bug somewhere in the sleeper code in the FreeRTOS port that I added to allow flash writes.

[ozbotics]

Hi Earle, I have prepared some demonstration code and sample output. Note the code doesn't ever reach "saveConfig - Ending EEPROM ok"

I commented out calls to rp2040.idleOtherCore() and rp2040.resumeOtherCore() and captured the second sample output.

main.cpp

#include <Arduino.h>

#include <FreeRTOS.h>
#include "queue.h"
#include "semphr.h"

#include <EEPROM.h>

#define DEBUG_LOAD_CONFIG
#define DEBUG_SAVE_CONFIG

#define DEBUG_SERIAL Serial1

#define EEPROM_ADRESS 0
#define EEPROM_SIZE 255

typedef long signed int MotorPosition_t;

const MotorPosition_t DefaultOpenedPosition = 100000;
const MotorPosition_t DefaultMotorPosition = 50000;

const unsigned int TaskUpdateStack = 100;
const unsigned short TaskUpdatePriority = 1;

const short ConfigMagicNumber = 1;

struct Config
{
  short configured;
  MotorPosition_t motorPosition;
  MotorPosition_t openedPosition;
};

short configured = 0;
MotorPosition_t motorPosition = 0;
MotorPosition_t openedPosition = 100000;

bool loadConfig();
void saveConfig();
void TaskUpdate(void *pvParameters);

void setup()
{
  DEBUG_SERIAL.begin(115200);
  delay(5000);

  DEBUG_SERIAL.print(F("Started\n"));

  if (!loadConfig())
  {
    saveConfig(); // save default values
  }

  xTaskCreate(
      TaskUpdate,
      "Update",
      100, NULL,
      4, NULL);
}

void loop()
{
}

void TaskUpdate(void *pvParameters)
{
  (void)pvParameters;

  for (;;)
  {
    vTaskDelay(pdMS_TO_TICKS(2000));

    motorPosition += 10;

    saveConfig();
  }
}

bool loadConfig()
{
  Config config;
  bool success = true;

#ifdef DEBUG_LOAD_CONFIG
  DEBUG_SERIAL.print(F("loadConfig - Begining EEPROM\n"));
#endif

  EEPROM.begin(EEPROM_SIZE);

#ifdef DEBUG_LOAD_CONFIG
  DEBUG_SERIAL.print(F("loadConfig - Begining EEPROM ok\n"));
  DEBUG_SERIAL.print(F("loadConfig - Getting EEPROM\n"));
#endif

  EEPROM.get(EEPROM_ADRESS, config);

#ifdef DEBUG_LOAD_CONFIG
  DEBUG_SERIAL.print(F("loadConfig - Getting EEPROM ok\n"));
  DEBUG_SERIAL.print(F("loadConfig - Ending EEPROM\n"));
#endif

  EEPROM.end();

#ifdef DEBUG_LOAD_CONFIG
  DEBUG_SERIAL.print(F("loadConfig - Ending EEPROM ok\n"));
#endif

  configured = config.configured;

  if (configured != ConfigMagicNumber)
  {
    openedPosition = DefaultOpenedPosition;
    motorPosition = DefaultMotorPosition;

#ifdef DEBUG_LOAD_CONFIG
    DEBUG_SERIAL.print(F("EEPROM has no config. Using default Config - "));
#endif
    configured = 0;
    success = false;
  }
  else
  {
#ifdef DEBUG_LOAD_CONFIG
    DEBUG_SERIAL.print(F("EEPROM has config.\nLoaded Config - "));
#endif

    openedPosition = config.openedPosition;
    motorPosition = config.motorPosition;
  }

#ifdef DEBUG_LOAD_CONFIG
  DEBUG_SERIAL.print(F("configured: "));
  DEBUG_SERIAL.print(configured);
  DEBUG_SERIAL.print(F(", motorPosition: "));
  DEBUG_SERIAL.print(motorPosition);
  DEBUG_SERIAL.print(F(", openedPosition: "));
  DEBUG_SERIAL.print(openedPosition);
  DEBUG_SERIAL.print(F("\n"));
#endif

  return success;
}

void saveConfig()
{
  Config config;
  config.configured = ConfigMagicNumber;
  config.motorPosition = motorPosition;
  config.openedPosition = openedPosition;

#ifdef DEBUG_SAVE_CONFIG
  DEBUG_SERIAL.print(F("Saving Config - "));
  DEBUG_SERIAL.print(F("configured: "));
  DEBUG_SERIAL.print(config.configured);
  DEBUG_SERIAL.print(F(", motorPosition: "));
  DEBUG_SERIAL.print(config.motorPosition);
  DEBUG_SERIAL.print(F(", openedPosition: "));
  DEBUG_SERIAL.println(config.openedPosition);
#endif

#ifdef DEBUG_SAVE_CONFIG
  DEBUG_SERIAL.print(F("saveConfig - Begining EEPROM\n"));
#endif

  EEPROM.begin(EEPROM_SIZE);

#ifdef DEBUG_LOAD_CONFIG
  DEBUG_SERIAL.print(F("saveConfig - Begining EEPROM ok\n"));
  DEBUG_SERIAL.print(F("saveConfig - Putting EEPROM\n"));
#endif

  EEPROM.put(EEPROM_ADRESS, config);

#ifdef DEBUG_LOAD_CONFIG
  DEBUG_SERIAL.print(F("saveConfig - Putting EEPROM ok\n"));
  DEBUG_SERIAL.print(F("saveConfig - Ending EEPROM\n"));
#endif

  EEPROM.end();

#ifdef DEBUG_SAVE_CONFIG
  DEBUG_SERIAL.print("saveConfig - Ending EEPROM ok\n");
#endif
}

platformio.ini

[env:pico]
platform = https://github.com/maxgerhardt/platform-raspberrypi.git
board = pico
framework = arduino
board_build.core = earlephilhower
debug_tool = picoprobe
upload_protocol = picoprobe
monitor_port = COM4
monitor_speed = 115200
; build_flags = 

Example Output

Started
loadConfig - Begining EEPROM
loadConfig - Begining EEPROM ok
loadConfig - Getting EEPROM
loadConfig - Getting EEPROM ok
loadConfig - Ending EEPROM
loadConfig - Ending EEPROM ok
EEPROM has no config. Using default Config - configured: 0, motorPosition: 50000, openedPosition: 100000
Saving Config - configured: 1, motorPosition: 50000, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM

Example Output when idleOtherCore / resumeOtherCore is disabled

Started
loadConfig - Begining EEPROM    
loadConfig - Begining EEPROM ok 
loadConfig - Getting EEPROM     
loadConfig - Getting EEPROM ok  
loadConfig - Ending EEPROM      
loadConfig - Ending EEPROM ok   
EEPROM has no config. Using default Config - configured: 0, motorPosition: 50000, openedPosition: 100000
Saving Config - configured: 1, motorPosition: 50000, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM    
saveConfig - Putting EEPROM ok 
saveConfig - Ending EEPROM     
saveConfig - Ending EEPROM ok
Saving Config - configured: 1, motorPosition: 50010, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok

[maxgerhardt]

Does this happen in the Arduino IDE too? I haven't had a look at FreeRTOS + PlatformIO yet here, and I think it may need lib_archive = no to even work correctly with all the weak function hooks, just like Adafruit TinyUSB. So please test with lib_archive = no in the platformio.ini too and check if it's any different.

[earlephilhower]

I had a quick look at it at way-too-early o'clock this morning and I think I see where things are busting. I don't think it's PIO or weak linking related.

What I'm seeing is that the USB task is being switched in and run, even though I've tried what I can to avoid any task switching in either the working or idle core. Need to dig some more, esp. the ARM SVC stuff vs IRQs and how they'd be blocked.

But in any case I am able to repro w/the IDE.

[earlephilhower]

719 in my testing fixes the core-caused crashes w/your MCVE. You should use GIT head and update the submodules (FreeRTOS needed fixes, too!) and try again.

HOWEVER, before you do that, the MCVE example you posted has too low a STACK size. The setting you use of 100 words (400 bytes) causes a stack overflow and system crash. Increase it and the MCVE + the patch work in my testing.

  xTaskCreate(
      TaskUpdate,
      "Update",
-      100, NULL,
+      400, NULL,
      4, NULL);

[ozbotics]

Hi Earle,

Thanks for this. Amazing turn around, much appreciated. Can you advise me how to "use GIT head and update the submodules" under PlatformIO? I have looked at the framework installation at (in my case) "C:\Users\jon.platformio\packages\framework-arduinopico" and I can't find any evidence of a .git directory.

I tried uninstalling the platform in PlatformIO | Platforms, then prompting the system to download it again by rebuilding my app, (I could see it re-downloading/installing various tools, but I get exactly the same failure in my test code.)

BTW: I gather from your previous comments that I am asking for trouble relying on the EEPROM emulation for many writes. I'm guessing that I should be using a file on a LittleFS partition (taking advantage of its wear-leveling design). I have read the docs about using Arduino IDE to specify a Flash partition scheme that leaves room for a FS. I have successfully formatted and mounted a LittleFS partition by specifying a "Flash Size: 2M (Sketch 1536KB, FS 512KB)" within Arduino IDE. My question is: Is it possible to specify this in PlatformIO and how? (sorry for asking about this in Issue thread)

[maxgerhardt]

In PlatformIO, you can use the git version of the Arduino-Pico framework as we documented at https://arduino-pico.readthedocs.io/en/latest/platformio.html#selecting-a-different-core-version

[ozbotics]

Thanks Max,

I have followed the directions at https://arduino-pico.readthedocs.io/en/latest/platformio.html#selecting-a-different-core-version

My platformio.ini file looks like

[env:pico]
platform = https://github.com/maxgerhardt/platform-raspberrypi.git
platform_packages =
   framework-arduinopico@https://github.com/earlephilhower/arduino-pico.git#master
board = pico
framework = arduino
board_build.core = earlephilhower
debug_tool = picoprobe
upload_protocol = picoprobe
monitor_port = COM4
monitor_speed = 115200
; build_flags = 

I recompiled (after the platform tasks completed). I still get the same failure.

Should I be pointing at #master, or is there a dev branch I should be pointing at. I assume the above process handles the "update the submodules" part.

What am I missing?

[maxgerhardt]

The platform_packages is correct. You can also do a pio pkg update -g -p https://github.com/maxgerhardt/platform-raspberrypi.git in the PlatformIO CLI (bottom taskbae icon) to make sure it's up-to-date.

You can also add lib_archive = no to test my theory.

[ozbotics]

Hmmm. I'm still not having any joy.

I ran pio pkg update -g -p https://github.com/maxgerhardt/platform-raspberrypi.git

PS C:\Users\jon\Documents\PlatformIO\Projects\xPicoEEPROM> pio pkg update -g -p https://github.com/maxgerhardt/platform-raspberrypi.git
Platform Manager: raspberrypi@1.7.0+sha.5677d9b is already up-to-date
Tool Manager: framework-arduinopico@1.20303.0+sha.005cba3 is already up-to-date
Tool Manager: tool-mklittlefs-rp2040-earlephilhower@5.100300.220714 is already up-to-date
Tool Manager: tool-openocd-rp2040-earlephilhower@5.100300.220714 is already up-to-date
Tool Manager: tool-rp2040tools@1.0.2 is already up-to-date
Tool Manager: toolchain-rp2040-earlephilhower@5.100300.220714 is already up-to-date

and added lib_archive = no to platformio.ini

[env:pico]
platform = https://github.com/maxgerhardt/platform-raspberrypi.git
platform_packages =
   framework-arduinopico@https://github.com/earlephilhower/arduino-pico.git#master
board = pico
framework = arduino
board_build.core = earlephilhower
debug_tool = picoprobe
upload_protocol = picoprobe
monitor_port = COM4
monitor_speed = 115200
; build_flags = 
lib_archive = no

I still get

Started
loadConfig - Begining EEPROM   
loadConfig - Begining EEPROM ok
loadConfig - Getting EEPROM    
loadConfig - Getting EEPROM ok
loadConfig - Ending EEPROM
loadConfig - Ending EEPROM ok
EEPROM has config.
Loaded Config - configured: 1, motorPosition: 58520, openedPosition: 100000
Saving Config - configured: 1, motorPosition: 58530, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM

[earlephilhower]

A simple check wound be to look At the map file. There should be a systick handler which souks be placed in ram. If it's still on flash, then the Freertos library did not get updated. I'm out of the house so can't find exact name.

In my testing it ran for over 60 cycles before I killed it. The problem was a pretty straightforward issue in creertos. SMP support is very new there and has lots of rough spots. And the official RP2040 port doesn't support any flash writes. It kind of makes sense because writing to flash kills any real-time capability. If never use it for critical systems since all tasks will freeze for tens of milliseconds on every write. An external eeprom is really needed if you care about timing.

[ozbotics]

A simple check wound be to look At the map file

I don't know what that is.

SMP support is very new there and has lots of rough spots

I'm not fussed about SMP, is there any way to configure it for Single Core use?

if you care about timing.

In my code, the timing critical parts are done just before the write.

I have started working on a version of the code that writes to a file hosted on a LittleFS partition, but I'm having trouble with it. Using Arduino IDE, I have managed to specify a "Flash Size: 2M (Sketch 1536KB, FS 512KB)" and LittleFS.format() successfully (although not yet read/write the file). When I try this under PlatformIO (with board_build.filesystem_size = 1.5m) it hangs trying to LittleFS.format() or LittleFS.begin(). It seems there are differences between ArduinoIDE and PlatformIO. Could this be related?

[ozbotics]

I have figured out how to generate a .map file. I have no idea how to interpret it.

Here it is output.map.txt

I found this reference to systick.

.data           0x00000000200000c0      0xe30 load address 0x000000001000f1b4
                0x00000000200000c0                __data_start__ = .
 *(vtable)
 *(.time_critical*)
 .time_critical.prvFIFOInterruptHandler
                0x00000000200000c0       0x54 .pio\build\pico\lib491\FreeRTOS\port.c.o
 .time_critical.isr_systick
                0x0000000020000114       0x38 .pio\build\pico\lib491\FreeRTOS\port.c.o
                0x0000000020000114                isr_systick
 .time_critical.SerialUART::_handleIRQ

[earlephilhower]

If you can't get LittleFS running in PIO but you can make it work in the Arduino IDE, then you've got a problem in your setup somewhere. I'd look that the right flash size and flash chip are being selected.

For sanity's sake, I pulled master, git submodule updated, and ran the MCVE (I changed ConfigMagicNumber so it would start from scratch) without incident:

Started
loadConfig - Begining EEPROM
loadConfig - Begining EEPROM ok
loadConfig - Getting EEPROM
loadConfig - Getting EEPROM ok
loadConfig - Ending EEPROM
loadConfig - Ending EEPROM ok
EEPROM has no config. Using default Config - configured: 0, motorPosition: 50000, openedPosition: 100000
Saving Config - configured: 1541, motorPosition: 50000, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50010, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50020, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50030, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50040, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50050, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50060, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50070, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50080, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50090, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50100, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50110, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50120, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok
Saving Config - configured: 1541, motorPosition: 50130, openedPosition: 100000
saveConfig - Begining EEPROM
saveConfig - Begining EEPROM ok
saveConfig - Putting EEPROM
saveConfig - Putting EEPROM ok
saveConfig - Ending EEPROM
saveConfig - Ending EEPROM ok

[ozbotics]

Did the map file tell you anything useful?

[maxgerhardt]

I'm using

[env]
platform = https://github.com/maxgerhardt/platform-raspberrypi.git
framework = arduino
board_build.core = earlephilhower
board_build.filesystem_size = 0.5m
platform_packages = 
   framework-arduinopico@https://github.com/earlephilhower/arduino-pico.git
lib_archive = no

[env:rpipico]
board = rpipico

with the exact sketch @ozbotics posted and I indeed still get a hangup. Also double checked that it's the 005cba3acd386fb3c2e605f50e4ae09887abda80 commit I have too, FreeRTOS is also at 0b55ee70ad08ab10af77f49e48a7381ac81a6827.

I'll have to double check with the Arduino IDE.

successfully (although not yet read/write the file). When I try this under PlatformIO (with board_build.filesystem_size = 1.5m) it hangs trying to LittleFS.format() or LittleFS.begin().

It's been some time since I re-tested LittleFS support in PlatformIO, I'll double check too.

[ozbotics]

Thanks Max, much appreciated

[earlephilhower]

elfsrc.zip

Here is the ELF, MAP and INO that I get out of the IDE. Ran for 10 loops w/no issues before I reset to UF2 upload to save my flash...

[earlephilhower]

This is on a stock Pico FWIW, if there is some flash boot2.S issue

[ozbotics]

Hi Max, Any progress on this?

I see that Earle has closed this issue. Should it be re-opened or another issue opened?

[maxgerhardt]

Sorry haven't looked at it. If I can reproduce the problem we can either reopen here or open at https://github.com/maxgerhardt/platform-raspberrypi/issues/, but likely after reproducing it I will just PR in a fix..

[ozbotics]

Great,

I have modified my EEPROM test code to (try to) use LittleFS. See attached 'xPicoEEPROM.zip' containing my PlatformIO project.

Note: there is a build_flag USE_LITTLEFS_CONFIG to select the use of LittleFS (instead of EEPROM) Also note: I have defined (in main.cpp) ALWAYS_FORMAT_LITTLEFS so that the code always tries (and sadly fails) to format the FS

Hope this helps

xPicoEEPROM.zip

[earlephilhower]

always tries (and sadly fails) to format the FS

Make sure you're actually allocating flash to the FS. Not sure of the PIO option to do this, but it's an IDE menu. Otherwise, well, there is no space to make a FS...

[maxgerhardt]

The project does have board_build.filesystem_size = 0.5m which is processed by the platform script the same way platform.txt and makeboards.py does and then the values are used in the builder script in the linkerscript generation. I tested LittleFS to be working in the initial version, but not recently. I'll have to look into it.

[maxgerhardt]

I cannot reproduce the hangup anymore in PlatformIO. Using the code here with Earle's suggestion of expanding the stack size to 400 elements instead of 100. Debug + release compiled mode work fine, using USB Serial or hardware Serial1 makes no difference too.

@ozbotics can you open a CLI and execute pio pkg update -g -p "https://github.com/maxgerhardt/platform-raspberrypi.git" and then check if the project eeprom_test.zip works for you too?

I will check LittleFS now.

[maxgerhardt]

And again I find that PlatformIO + LittleFS is working fine. As documented uploading a filesystem with a test.txt file (using the "Upload Filesystem" project task) and reading from it on the Pico works .

LittleFS.begin() SUCCESS
=== READ CONTENTS: ====
README lol
==== END OF FILE CONTENT ===

Note: I've tested this via regular UF2 (picotool) uploading and picoprobe, both worked.

Again, see for yourself using littlefs_test.zip

I will now have a look at your firmware..

[earlephilhower]

Thanks a bunch, @maxgerhardt , for the assistance!

[maxgerhardt]

Frst thing I see in your posted ZIP file is that again the stack size with 100 seems very small. I've corrected it to 400.

But wow, I could reproduce your error: If I upload the firmware and monitor it, I'm stuck at the

Failed to mount LittleFS - Formatting

line and it never gets past the formatting. However, if I restart the pico (in my case since I power it via the PicoProbe I just unplug the VBUS cable between them and plug it in again), it immediately goes past it.

Failed to mount LittleFS - Formatting
Mounted LittleFS
loadConfig - Opening config file
loadConfig - Failed to open config file
EEPROM has no config. Using default Config - configured: 0, motorPosition: 50000, openedPosition: 100000
Saving Config - configured: 1, motorPosition: 50000, openedPosition: 100000

I very remote guess is that I reboot the firmware in a wrong way and the second core is not started correctly and it ends up hanging for the second core to be halted.. Still have to figure it out how to attach in that exact configuration.

[maxgerhardt]

Yes, 100%. Attaching shows the last function is

(gdb) target remote :3333
Remote debugging using :3333
0x10002868 in _MFIFO::idleOtherCore() ()
(gdb) backtrace
#0  0x10002868 in _MFIFO::idleOtherCore() ()
#1  0x1000291a in littlefs_impl::LittleFSImpl::lfs_flash_erase(lfs_config const*, unsigned long) ()
#2  0x1000469c in lfs_bd_erase.isra ()
#3  0x10005418 in lfs_dir_compact ()
(gdb) info threads
  Id   Target Id                                           Frame
* 1    Thread 1 (Name: rp2040.core0, state: debug-request) 0x10002868 in _MFIFO::idleOtherCore() ()
  2    Thread 2 (Name: rp2040.core1, state: debug-request) 0x00000178 in ?? ()

Welp, I guess it's time to look into how to reset both cores properly with OpenOCD and double-check that picotool upload whether it bothers to reset both cores correctly..

[maxgerhardt]

Interesting. No matter what I do, I cannot get core1 out of the apparent bootloader code (0x174, really low address). OpenOCD has both targets selectable as targets.

> targets rp2040.core0
> reset halt
target halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ee msp: 0x20041f00
target halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ee msp: 0x20041f00
> targets rp2040.core1
> reset halt
target halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ee msp: 0x20041f00
target halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ee msp: 0x20041f00
> reset run

Won't get out of a 4-instruction loop.

target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0x00000178 msp: 0x20041f00
> step
target halted due to single-step, current mode: Thread
xPSR: 0x01000000 pc: 0x0000017a msp: 0x20041f00
> step
target halted due to single-step, current mode: Thread
xPSR: 0x01000000 pc: 0x00000174 msp: 0x20041f00
> step
target halted due to single-step, current mode: Thread
xPSR: 0x01000000 pc: 0x00000176 msp: 0x20041f00
> step
target halted due to single-step, current mode: Thread
xPSR: 0x01000000 pc: 0x00000178 msp: 0x20041f00

(gdb) x/10i $pc-6
   0x174:       wfe
   0x176:       ldr     r0, [r4, #80]   ; 0x50
   0x178:       lsrs    r0, r0, #1
=> 0x17a:       bcc.n   0x174
   0x17c:       ldr     r0, [r4, #88]   ; 0x58
(gdb) i r r4
r4             0xd0000000          -805306368
(gdb) i r r1
r1             0x2                 2

It seems to be waiting for the memory content 0xd0000050 to.. have the second bit set?

(gdb) x/10x 0xd0000050
0xd0000050:     0x0000000a      0x00000000      0x00000000      0x00008000
0xd0000060:     0xa5a5a5a5      0xa5a5a5a5      0xa5a5a5a5      0xa5a5a5a5
0xd0000070:     0xa5a5a5a5      0xa5a5a5a5

[maxgerhardt]

Per https://datasheets.raspberrypi.com/rp2040/rp2040-datasheet.pdf page 42, the 0xd0000050 belongs to the "SIO" (single I/O) address range, and the 0x50 offset specifically to the "FIFO_ST: Status register for inter-core FIFOs (mailboxes).", and bit "1" is "RDY: Value is 1 if this core’s TX FIFO is not full (i.e. if FIFO_WR is ready for more data)". Maybe the second core has to be sent a specific message to startup and somehow that isn't happening when I reset it like this?

Welp, that's all the time I have today. I did notice that regular, default picotool upload protocol (not via picoprobe, but UF2), the sketch seems to run. Maybe you can confirm that too, @ozbotics?

For now, no changing of core order seems to get this booting correctly, only a repowering..

C:\Users\Max\.platformio\packages\tool-openocd-rp2040-earlephilhower>bin\openocd.exe -s share\openocd\scripts -f interface/picoprobe.cfg -f target/rp2040.cfg -c "init; targets rp2040.core0; reset halt; targets rp2040.core1; reset halt; targets rp2040.core0; reset run; targets rp2040.core1; reset run; reset; shutdown"

[earlephilhower]

This was seen using OpenOCD to upload as well. What you are seeing is the 2nd core waiting in ROM for a "go" message to come thru the intercore FIFOs to start up. For OpenOCD, a simple delay(1) before firing off the 2nd core via the standard API calls from the SDK. No cause was found but it looks to be something HW related since the behavior doesn't happen on a cold boot.

One possibility would be to (try to) start the 2nd core over and over in a loop. If the multicore_launch_core1() called 2nd main doesn't set the global flag after a ms, then loop. Or we could try manually resetting the 2nd core before starting it. Neither of these should be needed, though, and no issues were ever seen like this w/UF2 upload.

[ozbotics]

I have increased the stack space to 400 and can confirm that the code runs as expected in EEPROM mode, but only after a reboot.

Similarly, the LittleFS version will format and mount after a reboot. However, it manages to make one save then hangs on the second write.

Started
Failed to mount LittleFS - Formatting
␀␀␀  
[Reboot]

Started
Failed to mount LittleFS - Formatting
Mounted LittleFS
loadConfig - Opening config file
loadConfig - Failed to open config file
EEPROM has no config. Using default Config - configured: 0, motorPosition: 50000, openedPosition: 100000
Saving Config - configured: 1, motorPosition: 50000, openedPosition: 100000
saveConfig - Opening config file
saveConfig - Opened configFile
saveConfig - Wrote configFile
saveConfig - Closed configFile
Saving Config - configured: 1, motorPosition: 50010, openedPosition: 100000
saveConfig - Opening config file