Klipper flash script is broken

[cab404]

Steps To Reproduce

Steps to reproduce the behavior:

1. Build klipper-flash with conf for some AVR printer
2. Launch it (klipper-flash-?)

sh: line 1: /nix/store/6qc8zvk3y12dcisg9f7f9yr1i221cjzz-klipper-firmware-printer-unstable-2023-11-16/board-link: Read-only file system
Makefile:103: /nix/store/6qc8zvk3y12dcisg9f7f9yr1i221cjzz-klipper-firmware-printer-unstable-2023-11-16/board-link: No such file or directory
make: *** [Makefile:101: /nix/store/6qc8zvk3y12dcisg9f7f9yr1i221cjzz-klipper-firmware-printer-unstable-2023-11-16/board-link] Error 1
make: Leaving directory '/nix/store/8wzv1hfgxxpc9lnw2k3dry9mnknmwfga-source'

Expected behavior

avrdude launching and flashing fw

Additional context

It appears to be broken across all boards, not just AVR

Notify maintainers

@cab404 :)

Add a :+1: reaction to issues you find important.

[Silverdev2482]

I am able to reproduce this, at least on nixos-unstable with an avr file for an ender 3. I suspect this commit has broke this so I am trying with the one right before it and seeing if it fixes it. I am back a few hours later after writing this comment, this does not fix it.

[Silverdev2482]

I have had no luck, I tried taking the script and the directoryies it use out of the nix store but that doesn't do anything, I also had a bad config but that is irrelvant as I fixed it. the firmware part of it should need to be packaged anyways so I will manually compile it.

[cab404]

I guess the fix is to copy or mount an overlayfs on top of the build folder, so flash script can modify it

[Silverdev2482]

Did you do that? did that work? if so how did you do it? I have tried manually compiling but have had no luck so I would like it if things started working. log.txt

[Silverdev2482]

This appeared to work for an overlay fs: however I still get errors.

[root@Raspi:~]# cp /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06 klipper-firmware-mcu -r

[root@Raspi:~]# mount --bind /root/klipper-firmware-mcu /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06

[root@Raspi:~]# klipper-flash-mcu 
make: Entering directory '/nix/store/22cs893bpkk4i5abmc476fqgsj3j66k6-source'
  Creating symbolic link /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06/board
  Building /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06/autoconf.h
  Compiling /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06/src/sched.o
make: arm-none-eabi-gcc: No such file or directory
make: *** [Makefile:64: /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06/src/sched.o] Error 127
make: Leaving directory '/nix/store/22cs893bpkk4i5abmc476fqgsj3j66k6-source'

I now have more hope, I'll look into those errors.

[cab404]

This appeared to work for an overlay fs: however I still get errors.

[root@Raspi:~]# cp /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06 klipper-firmware-mcu -r

[root@Raspi:~]# mount --bind /root/klipper-firmware-mcu /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06

[root@Raspi:~]# klipper-flash-mcu 
make: Entering directory '/nix/store/22cs893bpkk4i5abmc476fqgsj3j66k6-source'
  Creating symbolic link /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06/board
  Building /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06/autoconf.h
  Compiling /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06/src/sched.o
make: arm-none-eabi-gcc: No such file or directory
make: *** [Makefile:64: /nix/store/7zmdy34zb0scifjj0yaqgq3l417z8b0a-klipper-firmware-mcu-unstable-2024-01-06/src/sched.o] Error 127
make: Leaving directory '/nix/store/22cs893bpkk4i5abmc476fqgsj3j66k6-source'

I now have more hope, I'll look into those errors.

Do you want to take this issue over, or maybe have a mob session over it together?

[Silverdev2482]

I would like to work on it together, although I have personally forked nixpkgs but I'm not that comfortable with git and making PRs. I tried adding gcc-arm-embedded to the klipper-firmware package. That however had no result, I suspect I needed to add gcc-arm-embedded to buildInputs. The nixpkgs fork is on my home network, and I don't have a VPN to my home network and it is powered off so I won't be able to work on this for the next few hours. I also am not sure how to properly fix the package as it wants to overwrite itself. I also got stumped at this error manually compiling so I'm hoping that this same thing won't happen when I try fixing the package.

[Silverdev2482]

Ok, what I had to do was add gcc12 and gcc-arm-embedded to fix those errors but now I have gotten back to thoses errors, and I'm not too hopeful for fixing them but I am going to try some more.

[cab404]

Ok, what I had to do was add gcc12 and gcc-arm-embedded to fix those errors but now I have gotten back to thoses errors, and I'm not too hopeful for fixing them but I am going to try some more.

I am quite surprised you have to encounter them. Firmware part is building, the only thing missing is the flash script, right?

If not, please share your buildconfig.

[Silverdev2482]

I should say this is my first time installing klipper, but this is my first time installing klipper, I haven't read much of the docs but there don't appear to be many for installing klipper, so I assume I am doing everything correctly. I'll attach all files that may be notable. possibly-notable-files.tar.zstd It says .gz but it is zstd and for some reason github doesn't like other extensions.

[Silverdev2482]

I think the klipper.bin file is the firmware, if so I wonder if I can manually flash it? do you know if I can and how I could? using gcc13 and gcc-arm-embedded13 did not help.

[Silverdev2482]

wait, I can just flash it using a sd card. What am I thinking?

[Silverdev2482]

I have given up on using klipper on nixos, I wish you luck in fixing this issue and hope you succeed.

[cab404]

I have given up on using klipper on nixos, I wish you luck in fixing this issue and hope you succeed.

I can still help you with that. I am using Klipper on NixOS successfully for quite a while now. You can book a 15m slot in my schedule, when you will be near your printer

[Silverdev2482]

I might return to this another day, I am simply mentally tired, I might feel like doing this in a few days to a few months, for now. If you would like to talk about tech with me feel free to call me on discord as Silverdev2482 ( I am US Central Time) I would like to do that. I kinda burned myself out on this after spending way to many hours in one place.

[bamhm182]

I've been digging into flashing the MCU on my 3D printer all day, and while it's kind of trash, I got it to build and flash... The thing I don't understand is that the path that services.klipper.firmwares.${mcu} takes puts the location that klipper-firmware outputs (in /nix/store), and then klipper-flash attempts to use that as the OUT argument for make... make wants to write to the OUT directory, so you're going to have issues... I see that up above, it was fixed by bind mounting a rw directory over the path, but that's a little convoluted for "I just need my files in a rw location".

It also seems pretty redundant that we are building the firmware a second time, when klipper-firmware has already done that for us... I'm not familiar enough with building and flashing Klipper to see a different way to do it, though...

At any rate, I reworked the script a little bit and got it to work for my 3D printer with a BTT Manta M8P motherboard and I feel like it ought to work on others...

{ lib
, writeShellApplication
, gnumake
, pkgsCross
, klipper
, klipper-firmware
, python3
, avrdude
, stm32flash
, coreutils-full
, gcc-arm-embedded
, gcc
, libusb1
, dfu-util
, pkg-config
, flashMethodOverride ? null
, mcu ? "mcu"
, flashDevice ? "/dev/null"
, firmwareConfig ? ./simulator.cfg
}:
let
  supportedArches = [ "avr" "stm32" "lpc176x" ];
  matchBoard = with builtins; match ''^.*CONFIG_BOARD_DIRECTORY="([a-zA-Z0-9_]+)".*$'' (readFile firmwareConfig);
  boardArch = if matchBoard == null then null else builtins.head matchBoard;
  workingDir = "/tmp/flash.${mcu}/";
  flashMethod = (if flashMethodOverride == null then
    (if (boardArch == "stm32") then "serialflash" else "flash")
  else flashMethodOverride);
in
writeShellApplication {
  name = "klipper-flash-${mcu}";
  runtimeInputs = [
    python3
    pkgsCross.avr.stdenv.cc
    gnumake
    coreutils-full
    gcc-arm-embedded
    gcc
    libusb1
    dfu-util
    pkg-config
  ] ++ lib.optionals (boardArch == "avr") [ avrdude ] ++ lib.optionals (boardArch == "stm32") [ stm32flash ];
  text = if (builtins.elem boardArch supportedArches) then ''
    mkdir -p "${workingDir}"
    cp -r "${klipper-firmware}"/* "${workingDir}"
    make -C ${klipper.src} FLASH_DEVICE="${toString flashDevice}" OUT="${workingDir}" KCONFIG_CONFIG="${workingDir}config" ${flashMethod}
    rm -rf ${workingDir}
  '' else ''
    printf "Flashing Klipper firmware to your board is not supported yet.\n"
    printf "Please use the compiled firmware at ${klipper-firmware} and flash it using the tools provided for your microcontroller."
    exit 1
  '';
}

I want to see if I can figure out how to make things a little bit better before I go through trying to submit a PR, but I just wanted to post what I got working in the meantime...

Here's the file I'm using it in:

{ config, lib, pkgs, ... }:
let
  firmware = (pkgs.klipper-firmware.override {
    mcu = "mantam8pv2";
    firmwareConfig = ./mantam8pv2.cfg;
    gcc-arm-embedded = pkgs.gcc-arm-embedded-10;
  });
  flasher = (pkgs.callPackage ./klipper-flash.nix {
    flashMethodOverride = "flash";
    mcu = "mantam8pv2";
    klipper-firmware = firmware;
    firmwareConfig = ./mantam8pv2.cfg;
    flashDevice = "/dev/serial/by-id/usb-Klipper_stm32h723xx_0B0026001751313434373135-if00";
    gcc-arm-embedded = pkgs.gcc-arm-embedded-10;
  });
in
{
  config = lib.mkIf config.main.services.klipper.enable {
    environment.systemPackages = [
      firmware
      flasher
    ];
  };
}

[wdcocq]

This seems to be broken by the following change: https://github.com/NixOS/nixpkgs/commit/48384b974484ef14ee65c202423810b871cacd42

The klipper-flash package (which is used to create the flash scripts) calls make flash on the klipper source instead of using the flashing utilities directly (stm32flash, etc). And the missing files trigger a rebuild. This obviously fails because it's read-only.

Reverting the change with an overlay makes it work again:

(final: prev: 
  klipper-firmware = prev.klipper-firmware.overrideAttrs (old: {
    installPhase = ''
      mkdir -p $out
      cp ./.config $out/config
      cp -r out/* $out
    '';
  });
)

That said, I'm not exactly sure whether we should fix klipper-firmware or klipper-flash in this case.

Anyway the earlier solutions of overlay-fs, etc. don't make a lot of sense as the klipper firmware is build upon nixos rebuild.

[cab404]

Anyway the earlier solutions of overlay-fs, etc. don't make a lot of sense as the klipper firmware is build upon nixos rebuild.

This issue is mostly about flash script itself, not firmware building process.

[wdcocq]

This issue is mostly about flash script itself, not firmware building process.

Yes, and they are broken because klipper-firmware now only provides the final binary, where klipper-flash expects all project files.

Besides that there are still issues with the flash scripts, but those are outside of the scope of klipper-flash (it just exposes the flash options klipper does).

But those have nothing to do with your initial reported error:

line 1: /nix/store/6qc8zvk3y12dcisg9f7f9yr1i221cjzz-klipper-firmware-printer-unstable-2023-11-16/board-link: Read-only file system
Makefile:103: /nix/store/6qc8zvk3y12dcisg9f7f9yr1i221cjzz-klipper-firmware-printer-unstable-2023-11-16/board-link: No such file or directory
make: *** [Makefile:101: /nix/store/6qc8zvk3y12dcisg9f7f9yr1i221cjzz-klipper-firmware-printer-unstable-2023-11-16/board-link] Error 1
make: Leaving directory '/nix/store/8wzv1hfgxxpc9lnw2k3dry9mnknmwfga-source'

It tries to rebuild the binary while it's already built because of how the Makefile is set up.