search

Klipper3d / klipper

Klipper is a 3d-printer firmware
GNU General Public License v3.0
9.39k stars 5.29k forks source link

handling of multiple MCUs #55

Closed hg42 closed 6 years ago

hg42 commented 6 years ago

I am testing with two boards, a Mega2560+RAMPS and a TronXY board that is a Melzi derivate.

The avr currently flashes with -cwiring, but the TronXY needs -carduino. I don't know if this applies to all Melzi boards.

I also wonder how we should handle .config for multiple MCUs?

I would like to use different output directories OUT=... and different .config files for different MCUs. Currently, I configure each MCU and copy the .config to a named file outside the git-repo. I use a script to compile the firmware for both MCUs:

#!/bin/zsh

sudo service klipper stop

cp config-RAMPS klipper/.config
pushd klipper
make flash FLASH_DEVICE=/dev/ttyACM0 PROGRAMMER=wiring OUT=out-RAMPS/
popd

cp config-tronxy klipper/.config
pushd klipper
make flash FLASH_DEVICE=/dev/ttyUSB0 PROGRAMMER=arduino OUT=out-tronxy/
popd

sudo service klipper start

To make this work, I had to use another variable PROGRAMMER=....

I changed the Makefile for avr like this:

PROGRAMMER=wiring
...
flash: $(OUT)klipper.elf.hex
    @echo "  Flashing $(FLASH_DEVICE) via avrdude"
    $(Q)if [ -z $(FLASH_DEVICE) ]; then echo "Please specify FLASH_DEVICE"; exit 1; fi
    $(Q)avrdude -p$(CONFIG_MCU) -c$(PROGRAMMER) -P"$(FLASH_DEVICE)" -D -U"flash:w:$(OUT)klipper.elf.hex:i"

I also thought, it would be more logical to add those variables FLASH_DEVICE and PROGRAMMER (if not configured via menuconfig) to printer.cfg and automatically compile all MCUs that change in printer.cfg when FIRMWARE_RESTART is invoked.

Configuration via make menuconfig could then be invoked by something like: make MCU=RAMPS1 menuconfig generating a .config-$(MCU). Output directory woud be out-$(MCU).

KevinOConnor commented 6 years ago

On Mon, Dec 11, 2017 at 01:30:13PM +0000, Harald wrote:

I am testing with two boards, a Mega2560+RAMPS and a TronXY board that is a Melzi derivate.

Are you testing with two MCUs on a single printer?

The Mega2560 flashes with -cwiring and the TronXY flashes with -carduino. I don't know if this applies to all Melzi boards.

I also wonder how we should handle .config for multiple MCUs?

I would like to use different output directories OUT=... and different .config files for different MCUs. Currently, I configure each MCU and copy the .config to a named file outside the git-repo.

One can use the KCONFIG_CONFIG make variable to handle this. For example:

make KCONFIG_CONFIG=${PWD}/.config-xxx OUT=out-xxx/ menuconfig all

As you see I had to use another variable PROGRAMMER=...

I changed the Makefile for avr like this:

It's possible to do this, but I think it's simpler to invoke avrdude directly in these less common setups. By the time someone finds out they need an uncommon flash command they will likely have figured out what the direct flash command is.

So, for example:

avrdude -p atmega1284p -c arduino -P /dev/ttyUSB0 -U out/klipper.elf.hex

I can update the config/generic-melzi.cfg file to indicate sometimes "-c arduino" is needed instead of "-c avrisp" - the current description is based on the feedback from issue #35 .

-Kevin

hg42 commented 6 years ago

Are you testing with two MCUs on a single printer?

yes

KCONFIG_CONFIG=${PWD}/.config-xxx

thanks for the tip. I'm not very familiar with KCONFIG, but I admit I had thought something like this would exist. I should have looked for it myself.

it's simpler to invoke avrdude directly in these less common setups

ok, I did this, too. But I wanted it more clean at that level. When invoking make, I do not (want to) know the file names etc. They might change on updates.

But does it hurt to add the PROGRAMMER variable? you also give it the FLASH_DEVICE. Default is wiring like before. And if someone wants to override this, it's possible. It's also easy to find, just search for FLASH_DEVICE. Also it could be added as a comment for AVR MCUs somewhere.

Another reason for this variable is, that I can also use different programmers. There are more possibilities to load the firmware onto the device, e.g. it can have a bootloader or it is connected via usbasp or via avrisp or via arduinoisp etc. So this may be different for each user. So it's similar to the serial device.

I can create a PR if you want. Also the name could be different AVR_PROG or something (though it is called PROGRAMMER in the IDE).

KevinOConnor commented 6 years ago

On Mon, Dec 11, 2017 at 07:26:34AM -0800, Harald wrote:

Are you testing with two MCUs on a single printer?

yes

Okay. I'm interested in your results (success or failure).

it's simpler to invoke avrdude directly in these less common setups

ok, I did this, too. But I wanted it more clean at that level. When invoking make, I do not (want to) know the file names etc. They might change on updates.

But does it hurt to add the PROGRAMMER variable? you also give it the FLASH_DEVICE. Default is wiring like before. And if someone wants to override this, it's possible.

I'm more concerned about the documentation than the implementation. If a user settable variable is added then we need documentation describing what to set it to. That's a huge challenge as there's no good way to know what it should be set to. For example, one of my AVR devices can't be flashed at all with avrdude - it needs a device specific flashing utility to program it.

In my opinion the code and build should try to be simple for the common case and it should hint users where to research for the less common cases. (So, for example, users get a hint to do "man avrdude" or a web search for "avrdude" when trouble occurs.) I don't think there is a good solution here and I fear additional complexity could make a difficult situation worse.

I'll see if I can add a FAQ entry for what to do if "make flash" fails.

-Kevin

hg42 commented 6 years ago

Okay. I'm interested in your results (success or failure).

I started yesterday afternoon with a little "offline" test. I first used an old Mega+RAMPS with two motors. Then I added the TronXY Melzi board with a third motor. To test offline, there are way too much security checks, it's difficult to spin the motors if you need an endstop. Inverting an endstop also fails, because you detect, if it is still on after retraction. So I finally gave up for that evening.

Today I decide to be brave and removed my old Smoothieboard v1 from my TronXY X5S (CoreXY) and installed the two boards instead, together with a Pi3 that I installed with OctoPi as you described.

The documentation is awesome, I could do everything with some quick searches, looking on some examples and without open questions. Really nice... Well, I am not a beginner and I am also a software developer. But for such a new project, it's really well done.

That said, I am here for testing :-)

To be honest, I first have to become more familiar with this. It's still not completely configured, because I am testing several things before going forward. I have a lot of buttons in octoprint I also want to use for testing. But several cannot be used with Klipper. There are several gcodes I am missing, or that are different. But they may not be strictly necessary. E.g. I liked to have 0, 0 in the center of the print platform of my CoreXY, because I find this more convenient and well known from my Delta printer. I have to find out how I can do these kinds of things now. No problem, but all this needs some time.

The axes move, the extruder runs and I am now calibrating. Most things are the same values, but I also added another geared extruder etc.

Btw. I am not really happy with those small floating point numbers for step_distance. I find it easier to have steps per mm. E.g. I now have a value of 0.00022376543 mm/step for my extruder. I cannot intuitively say if that's reasonable. A value of 4469 steps/mm is much easier to handle. Perhaps you could change that? or allow an alternative value steps_per_mm?

I also have to decide which drivers to use now. First I used A4983 for testing. Last action today was to install a LV8729 with 128 microsteps on the extruder. I also got a few TMC2130 some days ago (I didn't know Klipper at that time). I want to use them in this machine finally, but for now I want to keep it simpler (no SPI etc.). Going forward step by step.

hg42 commented 6 years ago

According documentation of FLASH_PROGRAMMER (my preferred name now): You could eventually give a hint for experts, where they should look (Makefile). I guess, everyone who is able to use this will also find it. Another Variable would be FLASH_BAUDRATE. I would not try to document everything for everyone, because they will fail with that anyways. Such topics could also be added to an "advanced" document.

hg42 commented 6 years ago

just a quick note before bedtime.

I bravely installed four(!) LV8729 with 128 usteps on my Mega2560+RAMPS for limit testing. As expected, this doesn't work well. The moves sounded a bit raw for 128 usteps (so there might be jitter) and I often got shutdowns. I would really like to distribute these axes on several boards. Unfortunately my only board with sockets is this single RAMPS v1.3. So this has to wait...

Good night...

hg42 commented 6 years ago

should I close this issue?

KevinOConnor commented 6 years ago

On Mon, Dec 11, 2017 at 01:36:55PM -0800, Harald wrote:

Okay. I'm interested in your results (success or failure).

I started yesterday afternoon with a little "offline" test. I first used an old Mega+RAMPS with two motors. Then I added the TronXY Melzi board with a third motor. To test offline, there are way too much security checks, it's difficult to spin the motors if you need an endstop. Inverting an endstop also fails, because you detect, if it is still on after retraction. So I finally gave up for that evening.

In my testing, I'll set "self.verify_retract = False" in homing.py:Homing:init() to disable the retract test.

Today I decide to be brave and removed my old Smoothieboard v1 from my TronXY X5S (CoreXY) and installed the two boards instead, together with a Pi3 that I installed with OctoPi as you described.

The documentation is awesome, I could do everything with some quick searches, looking on some examples and without open questions. Really nice... Well, I am not a beginner and I am also a software developer. But for such a new project, it's really well done.

That said, I am here for testing :-)

To be honest, I first have to become more familiar with this. It's still not completely configured, because I am testing several things before going forward. I have a lot of buttons in octoprint I also want to use for testing. But several cannot be used with Klipper. There are several gcodes I am missing, or that are different. But they may not be strictly necessary. E.g. I liked to have 0, 0 in the center of the print platform of my CoreXY, because I find this more convenient and well known from my Delta printer. I have to find out how I can do these kinds of things now. No problem, but all this needs some time.

You should be able to do something like stepper_x.position_min=-100, stepper_x.position_endstop=-100 to change the location of 0,0. Another option is to use the M206 command.

The axes move, the extruder runs and I am now calibrating. Most things are the same values, but I also added another geared extruder etc.

Btw. I am not really happy with those small floating point numbers for step_distance. I find it easier to have steps per mm. E.g. I now have a value of 0.00022376543 mm/step for my extruder. I cannot intuitively say if that's reasonable. A value of 4469 steps/mm is much easier to handle. Perhaps you could change that? or allow an alternative value steps_per_mm?

You should round that value to 6 decimal places (anything more precise is literally slicing atoms). It's done this way to keep the units consistent in the config - mm for distance, seconds for time, celcius for temperature, etc.. I think specifying this one parameter in units of inverse millimeters is confusing.

I also have to decide which drivers to use now. First I used A4983 for testing. Last action today was to install a LV8729 with 128 microsteps on the extruder. I also got a few TMC2130 some days ago (I didn't know Klipper at that time). I want to use them in this machine finally, but for now I want to keep it simpler (no SPI etc.). Going forward step by step.

On Mon, Dec 11, 2017 at 10:35:33PM +0000, Harald wrote:

just a quick note before bedtime.

I bravely installed four(!) LV8729 with 128 usteps on my Mega2560+RAMPS for limit testing. As expected, this doesn't work well. The moves sounded a bit raw for 128 usteps (so there might be jitter) and I often got shutdowns.

What shutdowns did you get?

I would really like to distribute these axes on several boards. Unfortunately my only board with sockets is this single RAMPS v1.3.

If you're looking to run multiple MCUs to get a higher overall cpu capability, then I'm not sure you'll have great luck. The X and Y need to be on the same MCU with corexy (so that the endstops work) and that MCU will have the majority of steps. Also, there will be slight timing differences between steps executed on multiple MCUs - it's an open question how noticeable those timing differences will be.

On Wed, Dec 13, 2017 at 03:31:33PM +0000, Harald wrote:

should I close this issue?

Up to you - I usually close issues after a week or so of no activity. I'm still interested in your results with a printer running on multiple MCUs if you pursue that.

-Kevin

hg42 commented 6 years ago

self.verify_retract = False ...

thanks for these infos

stepper_x.position_min=-100, stepper_x.position_endstop=-100

I think I tried this...and something went wrong, but I check again at some time, when I managed to get a first print.

M206 command

this is what I do currently using octoprint

...to 6 decimal places (anything more precise is literally slicing atoms)

yes, but it' easier to just copy it from my calculator (one click)

It's done this way to keep the units consistent in the config - mm for distance, seconds for time, celcius for temperature, etc.. I think specifying this one parameter in units of inverse millimeters is confusing.

but it's not a distance, it's a relation mm/usteps. You don't use sec/mm only because sec is a unit. And it's usually usteps/mm because you can easily determine this mental calculation, e.g. 200 steps * 16 usteps/step * .... The count of digits is not my problem, but the count of zeros. I like to be consistent, e.g. it's inconvenient to have mm/sec and mm/min in the same system (gcode). You always have to recalculate and sometimes you get it wrong. It's probably done because early machines probably had integers for these numbers. But it's also a standard, which users are used to.

What shutdowns did you get?

MCU shutdowns with different messages like "MCU 'mcu' shutdown: Rescheduled timer in the past".

128 usteps is very fast for one 8bit MCU running all four steppers. I usually have X80 Y80 Z400 E100 usteps/mm with 16 usteps, so with 128 it's X640 Y640 Z3200 E800.

My Smoothieboard with LPC1768 at 100MHz has a heavy time to manage that with higher speeds, you can go up to about 150mm/s for XY. At some point it totally freezes (no reaction).

With Klipper the MCUs were shutdown, which is nice, because you can restart them without power down.

The sound wasn't smooth like it should be with 128 usteps, so I think the MCU was lagging, being at it's limits.

Currently, I still have all steppers on one MCU, but XYZ on 16 usteps and the extruder on 128 usteps. The MCU shuts down when doing G1 E... F2000 and it works at G1 E... F1000, I did not try values in between yet, because I want to get a normal print running first.

hg42 commented 6 years ago

According to the multiple MCUs I got some problems.

Perhaps you can tell, how the heater_fan usually works. I have:

[heater_fan hotend_fan]
pin: ramps:ar10
heater: extruder
heater_temp: 50.0
fan_speed: 0.5
max_power: 1.0

max_power is set to 1.0 because it doesn't work with soft pwm otherwise (it tells me). When I enable the heater, it starts with fan_speed I think. Shouldn't it wait until heater_temp is reached? This doesn't matter much, I only want to know if my fix is working correctly.

hg42 commented 6 years ago

first print worked good so far.

I see I have to calibrate my printer a bit different, may be because Klipper doesn't use something like jerk or junction deviation?

Btw. I still haven't finished reading the forum thread "New experimental firmware: all kinematics in host" http://forums.reprap.org/read.php?147,667655,page=5 Perhaps I am asking things that are answered there already. But I have been too excited to wait :-)

Some thoughts:

I reimplemented mcu.py / add_printer_objects like this:

def add_printer_objects(printer, config):
    mainsync = clocksync.ClockSync(printer.reactor)
    first = True
    for s in sorted(config.get_prefix_sections('mcu'), key=lambda s: s.section):
        if s.section == 'mcu' or s.section.startswith('mcu '):
            printer.add_object(s.section, MCU(
                printer, s,
                mainsync if first else clocksync.SecondarySync(printer.reactor, mainsync)))
            first = False

'mcu' is not created by default and therefore I use the first mcu with mainsync. Then I found that the sequence for connecting is sorted and thus I have to use the first mcu alphabetically instead. Currently I think MCU synchronization is symmetrical. Which means you cannot say this one or that one is more suitable to be mainsync. At least if you put each axis on it's own MCU, which one should be mainsync? Does it matter?

Do you have any numbers on how exact the synchronization works?

Being mad, I could even put the STEP pin on one MCU and the DIR pin on another...would that work? I guess not...

I think starting at Friday I'll try to put X and Y on different MCUs and see, how that works.

KevinOConnor commented 6 years ago

On Wed, Dec 13, 2017 at 08:28:48PM +0000, Harald wrote:

According to the multiple MCUs I got some problems.

  • MCUs cannot have UPPERcase in names, because you do lower() on the section name

I'm not sure why that is. Maybe I ran into some case sensitivity issue with the python ConfigParser.

  • when only using named MCUs, you get several problems.

I'm not sure what you are trying to accomplish here. One mcu has to be the "main" mcu - all the other mcu's will try to sync their clocks to the main one. The main mcu is not impacted by clock synchronization issues - the idea is to put the X and Y axes on the main mcu, and use the other MCUs for things with slightly less stringent timing.

There's a little bit of info on this in: docs/Code_Overview.md

Perhaps you can tell, how the heater_fan usually works. I have:

[heater_fan hotend_fan]
pin: ramps:ar10
heater: extruder
heater_temp: 50.0
fan_speed: 0.5
max_power: 1.0

max_power is set to 1.0 because it doesn't work with soft pwm otherwise (it tells me). When I enable the heater, it starts with fan_speed I think. Shouldn't it wait until heater_temp is reached? This doesn't matter much, I only want to know if my fix is working correctly.

The intended behaviour is to turn on the fan as soon as a non-zero target temp is requested. It's done that way because I didn't want to worry about the heater PID being disturbed by the fan coming on mid-way through heating.

On Thu, Dec 14, 2017 at 12:06:03AM +0000, Harald wrote:

first print worked good so far.

I see I have to calibrate my printer a bit different, may be because Klipper doesn't use something like jerk or junction deviation?

Klipper uses junction deviation - same as in grbl and smoothieware. See the bottom of config/example.cfg for details on setting it.

Some thoughts:

  • I have this script on a button to set the speeds and the sequence matters. Most important, if I move Z to the end, XY moves are with tha speed. I think each axis must have it's own speed setting. Also Marlin and Smoothieware are different according to separation of speed variables for G0 vs. G1. I think gcode standard is to have them different, but Marlin does not separate them.

I don't understand the above. In Klipper, G0 and G1 are aliases for the same command - see klippy/gcode.py:cmd_G1().

  • PrinterHeaterFan uses 
    self.mcu = printer.objects['mcu']
...
print_time = self.mcu.estimated_print_time(eventtime) + FAN_MIN_TIME
    self.fan.set_speed(print_time, power)

    Because I have no 'mcu' without name, I tried to use another mcu that is available at that place, which seems to work: self.mcu = self.fan.mcu_fan._mcu

That's fine. The code probably should just read:

print_time = self.fan.mcu_fan.get_mcu().estimated_print_time(eventtime)

I reimplemented mcu.py / add_printer_objects like this:

def add_printer_objects(printer, config):
    mainsync = clocksync.ClockSync(printer.reactor)
    first = True
    for s in sorted(config.get_prefix_sections('mcu'), key=lambda s: s.section):
        if s.section == 'mcu' or s.section.startswith('mcu '):
            printer.add_object(s.section, MCU(
                printer, s,
                mainsync if first else clocksync.SecondarySync(printer.reactor, mainsync)))
            first = False

'mcu' is not created by default and therefore I use the first mcu with mainsync. Then I found that the sequence for connecting is sorted and thus I have to use the first mcu alphabetically instead. Currently I think MCU synchronization is symmetrical. Which means you cannot say this one or that one is more suitable to be mainsync. At least if you put each axis on it's own MCU, which one should be mainsync? Does it matter?

See above - the code currently synchronizes secondary clocks to the main mcu clock.

It's possible to change that (eg, use the rpi as the clock master), but it would require a bit of code development.

Do you have any numbers on how exact the synchronization works?

I'm not sure what you are asking.

-Kevin

hg42 commented 6 years ago

Maybe I ran into some case sensitivity issue with the python ConfigParser

yes, you lower the names and ConfigParser doesn't find it. Eventually you could lower only the first word. Or the Parser has some option to ignore it. I'm not very used to python.

One mcu has to be the "main" mcu

Even if it has to be, I would still like to give it a name. It's clearer and it's used for other purposes as well and not having a name also makes it more difficult to communicate, e.g error messages etc.

Originally (in my own plans), I had a goal to give each axis (and each extruder, heater, etc.) it's own controller, say a Nano and a driver and an endstop. You get a Nano for 2-5 EUR. This way the system would be easily expandable. Just add as many axes as you need and as many extruders etc. Which would mean, all MCUs are made equal. They could be called X, Y, Z, E, hotend, bed, etc. Which one should be without name? and why?

I also wanted to sync them by hardware, one would use a timer and interrupt and ideally output the timer directly on a pin and all others would use an interrupt on an input pin connected to that output. Alternatively the Pi could output the clock. If all units are reset together and then given the clock, all should count the same amount of clock totally in sync.

I now realized, that there are dependencies like needing both endstops for each axis for corexy etc. In case of corexy you could simply connect the endstops which is easy because they are pull downs. I may test this...

I agree that there must be some master to generate the clock (if it's not the host). If there are dependencies like "MCU for steppers X and Y should be mainsync" then it should be easy to find out which one that is. But there is no need to omit the name. When collecting the configs the main mcu could be stored in a variable. And I think, it should be possible to override this by config, e.g. an option "mainsync: 1".

I can code this and you may see how it looks. My current version simply takes the first in sorted sequence. It only needs small and simple changes. However, this doesn't fit to what you wanted. So I would improve this.

There's a little bit of info on this in: docs/Code_Overview.md

I apologize, I didn't read that much and started coding and talking early. I probably should read more and talk less. But it's a fact that I am excited to test all this... but I promise to read more before disturbing you with my uncompleted thoughts. Now I have done some prints and I calm down, so I am back to a more normal level now... :-)

G0 and G1 are aliases

yes, that's like Marlin does it. Smoothieware developers claim that it's not obeying the "standard".

EDIT: because it's the speed along the direction of the combined movement of all axes, it's obviously not stored for each axis separately.

hg42 commented 6 years ago

I really want to find out how well X and Y can run in sync, so I tried with my corexy printer.

As stated above there is a dependency, that prevents separation of X and Y on different MCUs.

Basically there are only theses lines:

            self.steppers[0].mcu_endstop.add_stepper(self.steppers[1].mcu_stepper)
            self.steppers[1].mcu_endstop.add_stepper(self.steppers[0].mcu_stepper)

that add the Y stepper to the X endstop and the X stepper to the Y endstop. I understand, that it's necessary, because on hitting an endstop both steppers have to be stopped (because they act together in X direction and Y direction).

So, as a solution I made a shortcut between both endstop pins (which is ok, because both switches only connect the input to ground) and disabled both code lines. Now, if one endstop is pressed, both inputs are connected to ground.

How I understand the code, each stepper is still attached to it's own endstop (X stepper to X endstop etc.). So I think, hitting any endstop should now stop both steppers.

But it does not work. The other axis continues to run some distance before stopping and bumps into the switch.

Do you have a hint, what causes this behavior? Why isn't it sufficient to pull both endstop inputs down at the same time?

KevinOConnor commented 6 years ago

On Mon, Dec 18, 2017 at 12:23:55AM +0000, Harald wrote:

I really want to find out how well X and Y can run in sync, so I tried with my corexy printer.

As stated above there is a dependency, that prevents separation of X and Y on different MCUs.

Basically there are only theses lines:

            self.steppers[0].mcu_endstop.add_stepper(self.steppers[1].mcu_stepper)
            self.steppers[1].mcu_endstop.add_stepper(self.steppers[0].mcu_stepper)

that add the Y stepper to the X endstop and the X stepper to the Y endstop. I understand, that it's necessary, because on hitting an endstop both steppers have to be stopped (because they act together in X direction and Y direction).

So, as a solution I made a shortcut between both endstop pins (which is ok, because both switches only connect the input to ground) and disabled both code lines. Now, if one endstop is pressed, both inputs are connected to ground.

How I understand the code, each stepper is still attached to it's own endstop (X stepper to X endstop etc.). So I think, hitting any endstop should now stop both steppers.

But it does not work. The other axis continues to run some distance before stopping and bumps into the switch.

Do you have a hint, what causes this behavior? Why isn't it sufficient to pull both endstop inputs down at the same time?

Not only would you need to break the dependency between the x stepper and the y endstop, you'd need to create an additional y endstop object on the x mcu and an additional x endstop on the y mcu. So, it would require 4 endstop mcu objects and the corresponding steppers would need to be registered between them. Also, the home() method would need to pass the list of endstops (eg, when homing x, both mcu_x's and mcu_y's x endstop) to the homing_state.home() call.

I'm certainly interested in seeing your results with X and Y on different MCUs!

-Kevin

hg42 commented 6 years ago

Not only would you need to break the dependency between the x stepper and the y endstop, you'd need to create an additional y endstop object on the x mcu and an additional x endstop on the y mcu.

why?

It is clear that each endstop must stop both axes.

My understanding is, that for corexy two steppers are running and each is attached to it's "own" endstop. So it should stop, if this input is pulled down.

This is why I connected the endstops on the hardware side. Both endstops are connected to both endstop inputs. Which means, if one endstop is pressed, it pulls down both endstop inputs of X and Y.

To be sure, I tested this configuration by using cartesian kinematics. Then my axes run diagonal. I can stop both steppers with each of the endstops. So this part works. But in corexy kinematics one axis stops and the other keeps on running (for a short time of about a second). I can trigger this with both endstops, so the hardware part still works.

I tested this manually triggering the endstop in the middle of the bed. Then I see the motor named like the currently homing axis is stopped and then I get a diagonal move (= corexy with one motor) of about a cm until the other motor stops.

I also get these error messages:

Recv: !! Internal error on command:"G28"
Recv: ok
Recv: // Internal error on command:"G28"
Recv: // Once the underlying issue is corrected, use the
Recv: // "FIRMWARE_RESTART" command to reset the firmware, reload the
Recv: // config, and restart the host software.
Recv: !! Printer is shutdown
Recv: ok
Recv: // Internal error on command:"G28"
Recv: // Once the underlying issue is corrected, use the
Recv: // "FIRMWARE_RESTART" command to reset the firmware, reload the
Recv: // config, and restart the host software.
Recv: !! Printer is shutdown
Recv: ok
Recv: // Internal error on command:"G28"
Recv: // Once the underlying issue is corrected, use the
Recv: // "FIRMWARE_RESTART" command to reset the firmware, reload the
Recv: // config, and restart the host software.
Recv: !! Printer is shutdown
Recv: ok

Cartesian homing is different from corexy because only one motor moves. So the other doesn't need to be stopped. This works and does not produce error messages.

here is a log starting yesterday: klippy.log.gz

I have some additional logging outputs and some changes in the code but that shouldn't matter.

I could try to print with XY on separate MCUs using cartesian kinematics, but homing is difficult. And I would really like to get this running.

Also, the home() method would need to pass the list of endstops (eg, when homing x, both mcu_x's and mcu_y's x endstop) to the homing_state.home() call.

now, this sentence makes me think about it again...I seem to have assumptions that are wrong.

I already noticed, that endstops do not stop movements while printing (which I first assumed and would prefer). So, when homing, the home() method only gets those endstops, that should stop the movement and others are ignored?

Could I give it both endstops and it would work? Or is there a way to simply enable endstops for all movements (and ignore them when retracting or moving out of them, which could be difficult to define for other kinematics)?

hg42 commented 6 years ago

now...this works...

I gave home() both endstops in case of X or Y and now both axes stop.

So I think, I can test a print with distributed XY this evening...

Btw. to make all this work, I already had to move out of the endstop after the second homing move, which I prefer anyways, because of several reasons.

Also, I have several things in my queue:

I am basically looking around everywhere, partly understanding what I read and partly overwhelmed.

hg42 commented 6 years ago

Well...generally it worked much better as expected.

The first about 20 layers were printed flawlessly and I couldn't see any difference to former results of the same object. The printing speed was 100mm/s with 120mm/s travel. I used acceleration of 1500mm/s^2 (which was 1000mm/s^2 before!).

Then I got some shifts, but I am not sure about the reasons.

I had shifts with this model on other occasions (I mostly test with a 80x10x10mm bar with 8+8+1 4mm holes in each direction every 10mm). The model might be very sensitive to acceleration changes or similar. So the increase in acceleration should be blamed first.

Theoretically, some jitter or missing synchronization could have been generated by klipper resulting in too high acceleration at some points. But there were very fast moves that did work flawlessly. So I still think klipper worked perfectly.

There are also several other possible reasons. E.g. the TronXY board has DRV8825 drivers while the other board currently uses TMC2130. So (corexy-)X was on TMC2130 and (corexy-)Y was on DRV8825. Also, the TMC is new and I am still not experienced with it. It printed well before, but not at 1500mm/s^2, so this might have caused the shifts.

As a conclusion, I have to redo the test. Eventually I can find an alternative test object, which allows to see synchronization problems. But I currently cannot think of something that could fit this expectation better.

KevinOConnor commented 6 years ago

On Mon, Dec 18, 2017 at 06:14:21PM +0000, Harald wrote:

Not only would you need to break the dependency between the x stepper and the y endstop, you'd need to create an additional y endstop object on the x mcu and an additional x endstop on the y mcu.

why?

It is clear that each endstop must stop both axes.

My understanding is, that for corexy two steppers are running and each is attached to it's "own" endstop. So it should stop, if this input is pulled down.

This is why I connected the endstops on the hardware side. Both endstops are connected to both endstop inputs. Which means, if one endstop is pressed, it pulls down both endstop inputs of X and Y.

The way I would implement it would be to run the X endstop to a pin on both MCUs, and run the Y endstop to a different pin on both MCUs. (Thus, both MCUs can get the state of both endstops). I wouldn't recommend running both endstops to a single pin for the same reason I wouldn't recommend that on a single MCU - in that case it's not always safe to home. In particular, if the software starts up and the pin shows the endstops triggered it's possible the head could be an min x/max y or it could be at max x/min y - no way to know which position the carriage is at and any move could potentially force the head into the frame.

[...]

Also, the home() method would need to pass the list of endstops (eg, when homing x, both mcu_x's and mcu_y's x endstop) to the homing_state.home() call.

now, this sentence makes me think about it again...I seem to have assumptions that are wrong.

I already noticed, that endstops do not stop movements while printing (which I first assumed and would prefer).

Runtime endstop detection is on the todo list. Several sources recommend against doing runtime endstop detection (for fear of noise on the endstop pins), so I haven't bothered with it. It's not a high priority for me.

So, when homing, the home() method only gets those endstops, that should stop the movement and others are ignored?

Yes.

On Mon, Dec 18, 2017 at 01:46:11PM -0800, Harald wrote:

Well...generally it worked much better as expected.

In the first about 20 layers were printed flawless and I couldn't see any difference from before. The printing speed was 100mm/s with 120mm/s travel. I used acceleration of 1500mm/s^2 (which was 1000mm/s^2 before!).

Then I got some shifts, but I am not sure about the reasons.

I had shifts with this model on other occasions (I mostly test with a 80x10x10mm bar with 8+8+1 4mm holes in each direction every 10mm). The model might be very sensitive to acceleration changes or similar. So the increase in acceleration should be blamed first.

Theoretically, some jitter or missing synchronization could have been generated by klipper resulting in too high acceleration at some points. But there were very fast moves that did work flawlessly. So I still think klipper worked perfectly.

I think it unlikely that time synchronization issues would lead to lost steps. The timing of steps within an MCU should still be smooth. I'd more expect to see things like print blemishes at corners because one stepper changes velocity slightly out of sync with another stepper.

That said - testing will be the ultimate judge.

-Kevin

hg42 commented 6 years ago

the direction of the shifts indicates it's a problem with the X motor (diagonal shifts). I think I overstrained the TMC2130 on that motor with accel=1500. I now print the same test with accel=1000.

The corexy endstops do work this way, if I ensure, they are only touched for short times and always freed after the touch. It's one small thing to know, just like you never move sideways with a delta when in top (home) position. It's also safe if you reserve a small border of some mm at the max positions for homing (my printer has some room there). This is kind of a design decision, so we may want both options.

Anyways, the usual state of the machine should be with free endstops (which I implemented here, it's a simple second retract, parameters could be easier, though). It would be faster and more secure to retract with monitoring the endstops until they are free instead of length based retract. At some point I will probably rethink homing on corexy. For instance it could probably run diagonal. But finally, I want to home with force detection (e.g. using TMC2130).

I generally think, you are a bit too restrictive. As an engineer I understand why, but every restriction comes with a tradeoff. E.g. not being able to move the axes when you still didn't home, makes it impossible to solve some situations, when you cannot home, but have to move before (someone already rised an issue because of that).

The safer way is certainly a good strategy for a production machine (then I totally agree). I also think, it's a good thing to program for safety from the beginning instead of repairing it afterwards and I appreciate that.

However, to be honest, we are all pioneers, especially people using klipper (at least for now). People like me usually do not want to be restricted (and what about you?). I need full freedom for my experimental projects, mainly because I have to solve unusual situations.

As a solution to that dilemma, there could be a kind of expert mode. This could be (security-)feature-wise (e.g. I currently implemented a variable move_without_homing) or it may be easier to define some kind of user levels (like noob, average, expert, professional, wizard, adventurer, GOD=you). It's also possible to do both. Then the level could be a preset of several features which is modified by the config file. So, you would set the defaults (in the software itself), then a user mode file like expert.cfg, then printer.cfg.

Btw. is it ok to discuss here? or should I open an issue "ideas" as a place to discuss my ideas? Or should they all be separate issues? I don't want to flood the issue queue (which also looks bad for outsiders), so I tend to have a single thread for quick discussion and then create an issue or PR after some settling (or may not public? or at a forum? but then it's dividing information).

hg42 commented 6 years ago

I think it unlikely that time synchronization issues would lead to lost steps. The timing of steps within an MCU should still be smooth.

I didn't look at your synchronization algorithm (or tbh I looked shortly but couldn't completely understand it in that time). The MCUs could loose steps if the time jumps too much. If you use some kind of control loop the time difference will always run smooth and stay in bounds. I assume you do it similar to ntp?

I'd more expect to see things like print blemishes at corners because one stepper changes velocity slightly out of sync with another stepper. That said - testing will be the ultimate judge.

the test print is finished now and it went very well. So it is clear that the shifts came from the acceleration, which was too high for the TMC2130.

The corners are a bit edgy, but this is identical to the last print with XY on the same MCU (I intentionally didn't enable pressure_advance, to not overcomplicate the situation). The (very small) outstanding curve is totally symmetric to the corner. If it would come from a failure in synchronization, I would expect it to be asymmetric instead.

However, the test is certainly not at speed limits.

To test the synchronization separately under extreme conditions, we could detune the crystal oscillator of one MCU (or detune it by software if possible, which would allow more reproducible conditions) and drive two detached (or deconnected) steppers at the same speed and look at the difference signal (oscilloscope). It would be interesting how it follows clock frequency changes, how much it can compensate and if it stays stable and smooth.

And what do you think about hardware synchronization as an additional option? I think it should be easy to output the clock and use it as clock input for other MCUs?

I have to admit, that my experience with current MCUs is limited. My last dives into deeper regions of MCU software was a good couple of years ago. They are a lot more capable today and I am not familiar with all these short abbreviations. Though, I generally understand what's behind. Example:

void
serial_init(void)
{
    gpio_peripheral('A', PIO_PA8A_URXD, 'A', 1);
    gpio_peripheral('A', PIO_PA9A_UTXD, 'A', 0);

    // Reset uart
    PMC->PMC_PCER0 = 1 << ID_UART;
    UART->UART_PTCR = UART_PTCR_RXTDIS | UART_PTCR_TXTDIS;
    UART->UART_CR = UART_CR_RSTRX | UART_CR_RSTTX | UART_CR_RXDIS | UART_CR_TXDIS;
    UART->UART_IDR = 0xFFFFFFFF;

    // Enable uart
    UART->UART_MR = (US_MR_CHRL_8_BIT | US_MR_NBSTOP_1_BIT | UART_MR_PAR_NO
                     | UART_MR_CHMODE_NORMAL);
    UART->UART_BRGR = SystemCoreClock / (16 * CONFIG_SERIAL_BAUD);
    UART->UART_IER = UART_IER_RXRDY;
    NVIC_EnableIRQ(UART_IRQn);
    NVIC_SetPriority(UART_IRQn, 0);
    UART->UART_CR = UART_CR_RXEN | UART_CR_TXEN;
}

Some things are obvious, like MR = (M)ode (R)egister? IER = (I)nterrupt (E)nable (R)egister... I wonder why every library uses different names? Everything should be defined by the datasheet of the CPU.

KevinOConnor commented 6 years ago

On Tue, Dec 19, 2017 at 05:18:09AM -0800, Harald wrote:

I generally think, you are a bit too restrictive. As an engineer I understand why, but every restriction comes with a tradeoff.

The goal isn't to be restrictive - the goal is to have clear instructions and documentation so that new users with common hardware can quickly get high quality prints. My fear with lots of config file options and/or g-code commands is that it quickly becomes overwhelming. Ultimately, I think it may be easier to modify the python code then to try and cover every possible scenario in the config or g-code.

Flexibility is the primary reason for implementing the firmware in python.

On Tue, Dec 19, 2017 at 06:23:11AM -0800, Harald wrote:

I think it unlikely that time synchronization issues would lead to lost steps. The timing of steps within an MCU should still be smooth.

I didn't look at your synchronization algorithm (or tbh I looked shortly but couldn't completely understand it in that time). The MCUs could loose steps if the time jumps too much. If you use some kind of control loop the time difference will always run smooth and stay in bounds. I assume you do it similar to ntp?

The current synchronization algorithm is pretty simple. The MCU clock is queried every second and a weighted linear regression is used to build a prediction between the transmission time of the clock request and the actual received clock (see ClockSync class). Thus, the code builds a correlation between the host system time and the MCU clock. When multiple MCUs are used, once a second the secondary MCUs will estimate what the print_time will be on the primary MCU in four seconds, and then they virtually change their own clock frequency so that they would make the same print_time estimate (see SecondarySync class). There are some additional complexities, but that's the core algorithm.

I'd more expect to see things like print blemishes at corners because one stepper changes velocity slightly out of sync with another stepper. That said - testing will be the ultimate judge.

the test print is finished now and it went very well. So it is clear that the shifts came from the acceleration, which was too high for the TMC2130.

Can you provide the klippy.log file from the test? I'd like to see the stats - in my unit tests, the estimated frequencies were stable and the adjustments were all minor - hopefully the stats from your tests would show the same.

And what do you think about hardware synchronization as an additional option? I think it should be easy to output the clock and use it as clock input for other MCUs?

I'm hopeful that hardware clock synchronization isn't necessary. There are additional software tricks that could be done. In particular, MCUs with native USB have access to a high accuracy hardware timer on the USB bus.

I have to admit, that my experience with current MCUs is limited. My last dives into deeper regions of MCU software was a good couple of years ago. They are a lot more capable today and I am not familiar with all these short abbreviations. Though, I generally understand what's behind. Example:

I'm not sure if you're asking a question here. The code you site is all part of the low-level chip interface on the SAM3X8E processor - you'd need to pull the datasheet for that MCU to get the description of the fields.

Btw. is it ok to discuss here? or should I open an issue "ideas" as a place to discuss my ideas? Or should they all be separate issues?

Here is okay. There is also the mailing list and IRC (as described in docs/Contact.md). I have a small preference for the mailing list, but it's not a big deal to hold a discussion here.

-Kevin

hg42 commented 6 years ago

The goal isn't to be restrictive

ok, that's a statement :-)

the goal is to have clear instructions and documentation so that new users with common hardware can quickly get high quality prints. My fear with lots of config file options and/or g-code commands is that it quickly becomes overwhelming.

I agree that the defaults have to be simple and clear. A power user could have access to every variable. I would place several constants in a config file (may be a simple python file, but could also be something more like a in-memory database, simple key-value storage).

Flexibility is the primary reason for implementing the firmware in python

not every power user is able to program or can even understand the code. And it's not very convenient to find a parameter in the code.

That's one reason why I would like to have some kind of global config object, where you can find everything and also allow to change some values online. A gcode command could set these values. I will probably try some things in this direction anyways.

I already implemented a gcode to be able to execute python code (for myself). But I wasn't able to do everything I wanted, because many variables are read from the config and then become inaccessible, because they are hidden behind object oriented mechanisms. I am not sure if I missed something. I have access to "printer" and I thought "printer.objects" would give access to everything. I certainly have to revisit this topic.

The current synchronization algorithm is pretty simple

KISS principle :-) and it obviously works. It's kind of a PLL algorithm, which is proved.

and then they virtually change their own clock frequency so that they would make the same print_time estimate

the key is, how do they change their virtual clock frequency? After thinking a bit more about it I see that you can simply change the frequency. It's not like a jump in time but the time change is distributed over many clock ticks. Though, this could become oscillating faster-slower-faster-slower-...

hg42 commented 6 years ago

Can you provide the klippy.log file from the test? I'd like to see the stats - in my unit tests, the estimated frequencies were stable and the adjustments were all minor - hopefully the stats from your tests would show the same.

of course...I'm sorry I didn't append it already...here it is:

klippy.log.2017-12-19-hg42-XY-on-separate-MCUs.gz

the distributed print test starts at: Start printer at Tue Dec 19 11:38:55 2017 (1513683535.3 4727.0)

Before the test I did some things according to bed pid parameters and M220 implementation, which should work now (I used it after the print). After the test, I also ran other tests aiming at higher speeds (but without filament), e.g. I added:

max_extrude_only_velocity: 500
max_extrude_only_accel: 3000

and maxvelocity was at 500mm/s (but with accel=1000 maximum speed is reached seldom). I ran tests without filament and heaters removed from gcode. The last test(s) should be with extruder and heaters removed from gcode at M220 S500 (probably maximum speed limited by max... settings).

hg42 commented 6 years ago

I'm not sure if you're asking a question here

just a little bit moaning :-)

The code you site is all part of the low-level chip interface on the SAM3X8E processor - you'd need to pull the datasheet for that MCU to get the description of the fields.

yes, I know... I tried to rewrite that for the LPC1768 and it was late in the night...

hg42 commented 6 years ago

Here is okay. There is also the mailing list and IRC (as described in docs/Contact.md). I have a small preference for the mailing list, but it's not a big deal to hold a discussion here.

these are all with different features, e.g. IRC has to be in sync (both must be available), github is more public.

KevinOConnor commented 6 years ago

The stats in that log look excellent! They're even better than the results I got on some of my earlier unit tests. I graphed the frequencies reported in the stats (see below). One can see from the graph that the two micro-controllers have slightly different frequencies (neither is exactly at 16Mhz) and one can see that there is a small amount of drift during the 70 minute print (neither line is straight). However, after the first few seconds the measured frequencies are very stable and it seems klippy gets an excellent (~microsecond) lock on the micro-controller clocks.

There's only a couple small blips in the secondary mcu adjusted time (the green line) - that's likely due to new minimum rtt being found - an area that the current clocksync code could be improved. Even there, though, the adjustments made are minuscule.

Of course, this is all based on Klippy's stats - which could be off. There's also the possibility of systemic drift (one mcu having a large constant time difference), but the results look very promising.

frequency