[Bug]: CAN communication failure with toolboard when doing multi tool print

AndyEveritt commented 4 months ago

Duet Forum Discussion Thread

NA

Which Duet products are you using?

[ ] Duet2-Wifi
[ ] Duet2-Ethernet
[ ] Duet Expansion Breakout Board
[ ] Duex2
[ ] Duex5
[ ] Duet2-Maestro
[ ] Maestro Dual Driver Expansion
[X] Duet3-6HC
[ ] Duet3-3HC
[ ] Duet3-1XD
[X] Duet3-1LC
[ ] Duet3-Tool Distribution Board
[ ] Duet3-Mini5+
[ ] Duet3-Mini2+
[ ] Raspberry Pi or other SBC
[ ] SmartEffector
[ ] Magnetic Filament Sensor
[ ] Laser Filament Sensor
[ ] PT100 Daughterboard
[ ] Thermocouple Daughterboard
[ ] PanelDue
[ ] Other
[ ] None

Firmware Version

3.5.0-rc3+8

Duet Web Control Version

3.5.0-rc3+8

Are you using a Single Board Computer (RaspberryPi) with your Duet?

[X] Yes I use a SBC.
[X] No I do not use a SBC.

Please upload the results of sending M122 in the gcode console.

M122 Report

Please upload the content of your config.g file.

Config.g

Please upload the content of any other releveant macro files.

No response

Details specific to your printer.

E3D Toolchanger

Links to additional info.

No response

What happened?

Start a print that uses multiple tools with 1LC toolboards
CAN comms to a tools 1LC stop working ~10-30 seconds after that tool has been dropped off while printing
Pausing the print will recover from the issue
Pausing the print before the issue occurs will prevent it from occurring
It happens on both the 1LC tools on my ToolChanger (TC). Comms stop to a specific 1LC only after that tool has be unselected and a certain amount of time has passed.
It is 100% repeatable / consistent on my TC, Tony's TC (Duet 3 w/ SBC) and David's TC (Duet 3 standalone)

Setting the temperature is an example of a command to a TB that fails due to this issue.

3C_Lizard_2 (MMU v2) (1).zip

gloomyandy commented 4 months ago

I've been trying to reproduce this on my toolchanger running with 2 1LC toolboards but using an stm32H743 mainboard. All boards running the latest 3.5 build as of yesterday (27/3/2024). I ran the test file (without filament) up to about the 20% point which involved 10 tool changes. During that process I did not see any CAN-FD problems. I had to delete the startup code as it did not match my setup. I had the active temperature of both tools set to 220 and the standby temperature set to 210 for the first 4 tool changes, so there was a short delay each time while the tool came up to temperature. After that I modified the standby temperature to be 220 so there was no delay for the final 6 tool changes. I did not run a G29 probe sequence but loaded an existing mesh. I've now tried this a couple of times with the same results.

Not sure why I did not see the problem, happy to try other things if anyone has any suggestions.

For completeness my toolchanger has 4 tools tool 0 is connected directly to the mainboard, tool 1 uses a Fly RRF36 toolboard, tools 2 and 2 use !LC toolboards. I also have a prototype Fly SHT32 toolboard as part of the toolhead that is used to control the tool locking mechanism and a SZP type probe. So four CAN-FD toolboards in total.

A couple of notes the test file temperature settings look a little odd as they set active and standby temps for tools 0 and 1, but the print uses tools 2 and 3.

Edited to add: I checked the M122 output of all three boards after running the above test, no CAN errors reported by any of the boards.

AndyEveritt commented 4 months ago

M568 P0 S200 R180 A1
M568 P1 S200 R180 A1

var delay = 5

while iterations < 10
    T0
    G4 S{var.delay}
    G1 X105 Y105 E5 F2000
    G4 S{var.delay}
    G1 X100 Y100 E5 F2000
    G4 S{var.delay}
    G1 X105 Y105 E5 F2000

    T1
    G4 S{var.delay}
    G1 X105 Y105 E5 F2000
    G4 S{var.delay}
    G1 X100 Y100 E5 F2000
    G4 S{var.delay}
    G1 X105 Y105 E5 F2000

echo "Test finished"

This test script causes the issue.

However removing the XY movements stops the issue from occurring:

M568 P0 S200 R180 A1
M568 P1 S200 R180 A1

var delay = 5

while iterations < 10
    T0
    G4 S{var.delay}
    G1 E5 F1414
    G4 S{var.delay}
    G1 E5 F1414
    G4 S{var.delay}
    G1 E5 F1414

    T1
    G4 S{var.delay}
    G1 E5 F1414
    G4 S{var.delay}
    G1 E5 F1414
    G4 S{var.delay}
    G1 E5 F1414

echo "Test finished"

AndyEveritt commented 4 months ago

Issue does NOT occur if the extrude moves are removed from the print file (still extrudes in the tpost/tfree scripts) and the XY moves are added back:

M568 P0 S200 R180 A1
M568 P1 S200 R180 A1

var delay = 5

while iterations < 10
    T0
    G4 S{var.delay}
    G1 X105 Y105 F2000
    G4 S{var.delay}
    G1 X100 Y100 F2000
    G4 S{var.delay}
    G1 X105 Y105 F2000

    T1
    G4 S{var.delay}
    G1 X105 Y105 F2000
    G4 S{var.delay}
    G1 X100 Y100 F2000
    G4 S{var.delay}
    G1 X105 Y105 F2000

echo "Test finished"

AndyEveritt commented 4 months ago

Issue does not occur in 3.4.6-rc1 but does occur in 3.5.0 (tested rc2, rc3, rc4)

AndyEveritt commented 4 months ago

Issue occurs when removing the delays

M568 P0 S200 R180 A1
M568 P1 S200 R180 A1

while iterations < 10
    T0
    G1 X105 Y105 E5 F2000
    G1 X100 Y100 E5 F2000
    G1 X105 Y105 E5 F2000

    T1
    G1 X105 Y105 E5 F2000
    G1 X100 Y100 E5 F2000
    G1 X105 Y105 E5 F2000

echo "Test finished"

dc42 commented 4 months ago

Fixed in 3.5.0-rc.4. Problem was that when filament monitors were returning live data they were sending it too frequently.

Duet3D / RepRapFirmware