search

bdring / FluidNC

The next generation of motion control firmware
Other
1.59k stars 383 forks source link

Problem: Board reboots if $sd/list is issued after upload #932

Open Ams-Laser opened 1 year ago

Ams-Laser commented 1 year ago

Controller Board

MKS TinyBee V1.0

Machine Description

Co2 Laser, but problem was detected using plain board, no drivers no conections, only SD card inserted

Input Circuits

NA

Configuration file

board: AMS V1.0 XYYZA
name: AMS-Laser
planner_blocks: 64
kinematics:
  Cartesian:

i2so:
  bck_pin: gpio.25
  data_pin: gpio.27
  ws_pin: gpio.26

spi:
  miso_pin: gpio.19
  mosi_pin: gpio.23
  sck_pin: gpio.18

sdcard:
  cs_pin: gpio.5
  # uses TH2 IO34 active low - MAKE SURE jumper J2 is set to SDDET!!!
  card_detect_pin: gpio.34:low
  frequency_hz: 5000000

stepping:
  engine: I2S_STATIC
  idle_ms: 255
  pulse_us: 4
  dir_delay_us: 1
  disable_delay_us: 2

axes:
  x:
    steps_per_mm: 20
    max_rate_mm_per_min: 12000.000
    acceleration_mm_per_sec2: 1500.000
    max_travel_mm: 1230.000
    soft_limits: true
    homing:
      cycle: 2
      positive_direction: false
      mpos_mm: 0.000
      feed_mm_per_min: 300.000
      seek_mm_per_min: 1500.000
      settle_ms: 500
      seek_scaler: 1.100
      feed_scaler: 1.100

    motor0:
      limit_neg_pin: gpio.33:low:pu
      hard_limits: false
      pulloff_mm: 4.000
      stepstick:
        step_pin: I2SO.1
        direction_pin: I2SO.2
        disable_pin: I2SO.0

  y:
    steps_per_mm: 20
    max_rate_mm_per_min: 6000.000
    acceleration_mm_per_sec2: 500.000
    max_travel_mm: 610.000
    soft_limits: true
    homing:
      cycle: 3
      positive_direction: false
      mpos_mm: 0.000
      feed_mm_per_min: 300.000
      seek_mm_per_min: 2000.000
      settle_ms: 500
      seek_scaler: 1.100
      feed_scaler: 1.100

    motor0:
      limit_neg_pin: gpio.32:low:pu
      hard_limits: false
      pulloff_mm: 4.000
      stepstick:
        step_pin: I2SO.4
        direction_pin: I2SO.5
        disable_pin: I2SO.3

    # use E1 driver for 2nd Y axis motor
    motor1:
      limit_neg_pin: NO_PIN
      hard_limits: false
      pulloff_mm: 4.000
      stepstick:
        step_pin: I2SO.13
        direction_pin: I2SO.14
        disable_pin: I2SO.12

  z:
    steps_per_mm: 40.000
    max_rate_mm_per_min: 6000.000
    acceleration_mm_per_sec2: 500.000
    max_travel_mm: 2000.000
    soft_limits: false
    homing:
      cycle: 1
      positive_direction: true
      mpos_mm: 0.000
      feed_mm_per_min: 300.000
      seek_mm_per_min: 500.000
      settle_ms: 500
      seek_scaler: 1.100
      feed_scaler: 1.100

    motor0:
      limit_pos_pin: gpio.22:low:pu
      hard_limits: false
      pulloff_mm: 3.000
      stepstick:
        step_pin: I2SO.7
        direction_pin: I2SO.8
        disable_pin: I2SO.6

control:
  safety_door_pin: NO_PIN
  # on MT_DET connector
  reset_pin: gpio.35:low
  # on TH1 connector
  feed_hold_pin: gpio.36:low
  # on TB connector
  cycle_start_pin: gpio.39:low
  macro0_pin: NO_PIN
  macro1_pin: NO_PIN
  macro2_pin: NO_PIN
  macro3_pin: NO_PIN

coolant:
  # Heated Bed Terminal Block
  flood_pin: i2so.16
  # HE0 Terminal Block
  mist_pin: i2so.17
  delay_ms: 0

Laser:
  pwm_hz: 5000
  # on 3D Touch connector (has pdwn + PWM)
  output_pin: gpio.2:high:pd
  s0_with_disable: true
  tool_num: 0
  speed_map: 0=0.000% 1000=100.000%

start:
  must_home: false

Startup Messages

Startup after error and auto reboot:
Rebooting...
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x1b (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1184
load:0x40078000,len:13220
ho 0 tail 12 room 4
load:0x40080400,len:3028
entry 0x400805e4

[MSG:INFO: FluidNC v3.7.1]
[MSG:INFO: Compiled with ESP32 SDK:v4.4.4]
[MSG:INFO: Local filesystem type is spiffs]
[MSG:ERR: Skipping configuration file due to panic]
[MSG:INFO: Using default configuration]
[MSG:INFO: Axes: using defaults]
[MSG:INFO: Machine Default (Test Drive)]
[MSG:INFO: Board None]
[MSG:INFO: Stepping:RMT Pulse:4us Dsbl Delay:0us Dir Delay:0us Idle Delay:255ms]
[MSG:INFO: Axis count 3]
[MSG:INFO: Axis X (-1000.000,0.000)]
[MSG:INFO:   Motor0]
[MSG:INFO: Axis Y (-1000.000,0.000)]
[MSG:INFO:   Motor0]
[MSG:INFO: Axis Z (-1000.000,0.000)]
[MSG:INFO:   Motor0]
[MSG:INFO: Kinematic system: Cartesian]
[MSG:INFO: Using spindle NoSpindle]
[MSG:INFO: BT Started with FluidNC]

Startup after sending $bye:
rst:0xc (SW_CPU_RESET),boot:0x1b (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1184
load:0x40078000,len:13220
ho 0 tail 12 room 4
load:0x40080400,len:3028
entry 0x400805e4

[MSG:INFO: FluidNC v3.7.1]
[MSG:INFO: Compiled with ESP32 SDK:v4.4.4]
[MSG:INFO: Local filesystem type is spiffs]
[MSG:INFO: Configuration file:ams12060.yaml]
[MSG:INFO: Machine AMS-Laser]
[MSG:INFO: Board AMS V1.0 XYYZA]
[MSG:INFO: I2SO BCK:gpio.25 WS:gpio.26 DATA:gpio.27]
[MSG:INFO: SPI SCK:gpio.18 MOSI:gpio.23 MISO:gpio.19]
[MSG:INFO: SD Card cs_pin:gpio.5 detect:gpio.34:low freq:5000000]
[MSG:INFO: Stepping:I2S_static Pulse:4us Dsbl Delay:2us Dir Delay:1us Idle Delay:255ms]
[MSG:INFO: Axis count 3]
[MSG:INFO: Axis X (0.000,1230.000)]
[MSG:INFO:   Motor0]
[MSG:INFO:     stepstick Step:I2SO.1 Dir:I2SO.2 Disable:I2SO.0]
[MSG:INFO:  X Neg Limit gpio.33:low:pu]
[MSG:INFO: Axis Y (0.000,610.000)]
[MSG:INFO:   Motor0]
[MSG:INFO:     stepstick Step:I2SO.4 Dir:I2SO.5 Disable:I2SO.3]
[MSG:INFO:  Y Neg Limit gpio.32:low:pu]
[MSG:INFO:   Motor1]
[MSG:INFO:     stepstick Step:I2SO.13 Dir:I2SO.14 Disable:I2SO.12]
[MSG:INFO: Axis Z (-2000.000,0.000)]
[MSG:INFO:   Motor0]
[MSG:INFO:     stepstick Step:I2SO.7 Dir:I2SO.8 Disable:I2SO.6]
[MSG:INFO:  Z Pos Limit gpio.22:low:pu]
[MSG:INFO: reset_pin gpio.35:low]
[MSG:INFO: feed_hold_pin gpio.36:low]
[MSG:INFO: cycle_start_pin gpio.39:low]
[MSG:INFO: Kinematic system: Cartesian]
[MSG:INFO: Laser Ena:NO_PIN Out:gpio.2:pd Freq:5000Hz Period:8191]
[MSG:INFO: Using spindle Laser]
[MSG:INFO: Flood coolant I2SO.16]
[MSG:INFO: Mist coolant I2SO.17]
[MSG:INFO: BT Started with FluidNC]

Grbl 3.7 [FluidNC v3.7.1 (bt) '$' for help]
[MSG:WARN: Low memory: 14484 bytes]
[MSG:WARN: Low memory: 14476 bytes]
[MSG:WARN: Low memory: 14256 bytes]
[MSG:WARN: Low memory: 14240 bytes]
[MSG:WARN: Low memory: 14112 bytes]
[MSG:WARN: Low memory: 13992 bytes]
[MSG:WARN: Low memory: 13864 bytes]
[MSG:WARN: Low memory: 13736 bytes]
[MSG:WARN: Low memory: 13576 bytes]
[MSG:WARN: Low memory: 13424 bytes]
[MSG:WARN: Low memory: 13296 bytes]

User Interface Software

Fluid Term

What happened?

uploaded a file to SD using fluid term, then issued $sd/list (to verify upload) board re booted

Other Information

Backtrace: 0x40085539:0x3ffd5550 0x40096729:0x3ffd5570 0x4009d091:0x3ffd5590 0x401c1aa7:0x3ffd5610 0x401db490:0x3ffd5630 0x401c177a:0x3ffd5650 0x401dd471:0x3ffd56c0 0x401dd755:0x3ffd5700 0x400d98d3:0x3ffd57a0 0x400feaf3:0x3ffd57c0 0x400ff209:0x3ffd5870 0x400ff289:0x3ffd5900 0x400fc00f:0x3ffd5950 0x400fc370:0x3ffd5a50 0x401f9345:0x3ffd5a70 0x400f028a:0x3ffd5aa0 0x400f04cb:0x3ffd5b20 0x400f0527:0x3ffd5b50 0x400f1606:0x3ffd5b80 0x400e45f1:0x3ffd5bb0 0x40104495:0x3ffd5c00

ELF file SHA256: 007f6a0ee47fc85b

dymk commented 1 year ago

Looking into this, want to drop the decoded backtrace here - 

$> ~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-addr2line -e bt-firmware.elf -fCp < backtrace.txt

panic_abort at (snip)/esp-idf-public/components/esp_system/panic.c:408
esp_system_abort at (snip)/esp-idf-public/components/esp_system/esp_system.c:137
abort at (snip)/esp-idf-public/components/newlib/abort.c:46
__cxxabiv1::__terminate(void (*)()) at (snip)/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:47
__cxa_call_terminate at (snip)/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_call.cc:54
__gxx_personality_v0 at (snip)/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_personality.cc:676
_Unwind_RaiseException_Phase2 at (snip)/xtensa-esp32-elf/src/gcc/libgcc/unwind.inc:64
_Unwind_Resume at (snip)/xtensa-esp32-elf/src/gcc/libgcc/unwind.inc:241
__gnu_cxx::new_allocator<char>::~new_allocator() at (snip)/xtensa-esp32-elf/include/c++/8.4.0/ext/new_allocator.h:86 (discriminator 1)
std::filesystem::__cxx11::directory_entry::directory_entry(std::filesystem::__cxx11::path const&, std::filesystem::file_type) at (snip)/xtensa-esp32-elf/include/c++/8.4.0/bits/fs_dir.h:304
std::filesystem::__cxx11::recursive_directory_iterator::increment(std::error_code&) at (snip)/FluidNC/stdfs/std-dir.cpp:177
std::filesystem::__cxx11::recursive_directory_iterator::operator++() at (snip)/FluidNC/stdfs/std-dir.cpp:155
_ZN5WebUIL14listFilesystemEPKcS1_NS_19AuthenticationLevelER7Channel$isra$76 at (snip)/FluidNC/src/WebUI/WebSettings.cpp:407
WebUI::listSDFiles(char*, WebUI::AuthenticationLevel, Channel&) at (snip)/FluidNC/src/WebUI/WebSettings.cpp:433
WebCommand::action(char*, WebUI::AuthenticationLevel, Channel&) at (snip)/FluidNC/src/WebUI/WebSettings.cpp:107
do_command_or_setting(char const*, char*, WebUI::AuthenticationLevel, Channel&) at (snip)/FluidNC/src/ProcessSettings.cpp:927
settings_execute_line(char*, Channel&, WebUI::AuthenticationLevel) at (snip)/FluidNC/src/ProcessSettings.cpp:989
execute_line(char*, Channel&, WebUI::AuthenticationLevel) at (snip)/FluidNC/src/ProcessSettings.cpp:1017
protocol_main_loop() at (snip)/FluidNC/src/Protocol.cpp:282
loop() at (snip)/FluidNC/src/Main.cpp:191
loopTask(void*) at (snip)/framework-arduinoespressif32/cores/esp32/main.cpp:50

So we're seeing a failure that involves the copy/pasted std::filesystem contents in std-dir.cpp, in particular, something in the line while (!_M_dirs->top().advance(skip_permission_denied, ec) && !ec)

Continuing to investigate...

For those interested in generating their own backtrace, the file backtrace.txt piped into the platformio bundled addr2line is a newline separated addresses from the backtrace.

MitchBradley commented 1 year ago

I wonder if it is just out of memory at that point. "new_allocator"

dymk commented 1 year ago

Could be, but it's a little odd that it's crashing in ~new_allocator(), the dtor. Here's lines 303 to 305 of fs_dir.h (fs_dir.h:304 is in the backtrace) - 

    directory_entry(const filesystem::path& __p, file_type __t)
    : _M_path(__p), _M_type(__t)
    { }

so perhaps it runs out of memory when trying to make a copy of __p (_M_path is a full filesystem::path object, not taking a reference).

MitchBradley commented 1 year ago

We have reason to believe that memory could be low, based on the WARN messages. I never use the BT version so I do not have a good sense of its memory usage. I wonder if it can be configured similarly to WiFi.

dymk commented 1 year ago

@Ams-Laser are you able to reliably reproduce this issue when issuing $sd/list just after startup? Could you try with the wifi or noradio builds and see if you can repro the reboot?

Ams-Laser commented 1 year ago

@Ams-Laser are you able to reliably reproduce this issue when issuing $sd/list just after startup? Could you try with the wifi or noradio builds and see if you can repro the reboot?

It does not happen after reboot, only after upload, Will try on WiFi and let you know

MitchBradley commented 1 year ago

The Bluetooth build is using a lot of RAM memory. The Arduino Espressif32 Framework compiles the Bluetooth library with a lot of unnecessary options enabled - BLE, audio, secure pairing, and various other things that chew up memory.

I made some improvements to the WebSocketsNoStall branch to get back some of the memory, but there is still a potential problem. Reducing the framework library's memory footprint can be done, but it requires some tricks that create a maintenance problem.

MitchBradley commented 1 year ago

@Ams-Laser Does this prerelease help? https://github.com/bdring/FluidNC/releases/tag/v3.7.2-pre3 It reduces the memory usage of the Bluetooth stack, in addition to other memory-related improvements.

Ams-Laser commented 1 year ago

Will try and get back

Ams-Laser commented 1 year ago

just flashed and connected does not look good image it works in vector mode but if i send a raster job it just hangs image