workaround esp-idf version detection

[madmo]

If version.txt is missing some part of the cmake toolchain tries to read refs/heads/fetchgit in the idf repo which failed.

This is not a perfect solution, but works nicely when using cargo, and fixes some of the issues in #5

[madmo]

Ok, back to the drawing board. The issue is that fetchgit with leaveDotGit is not always reproducible: https://github.com/NixOS/nixpkgs/issues/8567.

[jonahbron]

Seeing other comments around this affecting non-ESP related stuff, might have some useful resources

https://github.com/NixOS/nixpkgs/issues/169193

[madmo]

For me it is working now. Had a issue with the installPhase script writing to the source dir, which in turn caused some hash issue.

[jonahbron]

I can now get into a shell with nix develop with your fork. However, I'm still having trouble pulling it into another flake for building.

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    nixpkgs-esp-dev.url = "github:madmo/nixpkgs-esp-dev";
  };

  outputs = { self, nixpkgs, nixpkgs-esp-dev }: let
    forAllSystems = function:
      nixpkgs.lib.genAttrs [
        "x86_64-linux"
      ] (system: function (import nixpkgs {inherit system;}));
  in {
    packages = forAllSystems (pkgs: let
      berkeley-db-1_xx = pkgs.fetchFromGitHub {
        owner = "pfalcon";
        repo = "berkeley-db-1.xx";
        rev = "35aaec4418ad78628a3b935885dd189d41ce779b";
        sha256 = "sha256-XItxmpXXPgv11LcnL7dty6uq1JctGokHCU8UGG9ic04=";
      };
      micropython-src = pkgs.fetchFromGitHub {
        owner = "micropython";
        repo = "micropython";
        rev = "v1.22.2";
        sha256 = "sha256-67CAR34VrMOzvNkukDeGRnUfoOLO66R37wsrRHjpp5E=";
      };
    in {
      esp32genericc3 = pkgs.stdenv.mkDerivation {
        name = "esp32genericc3";
        nativeBuildInputs = [
          pkgs.git
          nixpkgs-esp-dev.packages.${pkgs.system}.esp-idf-esp32c3
        ];

        src = micropython-src;
        phases = ["unpackPhase" "patchPhase" "buildPhase"];

        patchPhase = ''
          rmdir lib/berkeley-db-1.xx
          ln -s ${berkeley-db-1_xx} lib/berkeley-db-1.xx
        '';

        buildPhase = ''
          make V=1 -C mpy-cross
          make V=1 -C ports/esp32 BOARD=ESP32_GENERIC_C3
        '';
      };
    });
  };
}

This is incomplete but I can't even get past the buildPhase before it hits the Git error

...
make: Leaving directory '/build/source/mpy-cross'
make: Entering directory '/build/source/ports/esp32'
idf.py -D MICROPY_BOARD=ESP32_GENERIC_C3 -D MICROPY_BOARD_DIR=/build/source/ports/esp32/boards/ESP32_GENERIC_C3  -B build-ESP32_GENERIC_C3 build || (echo -e "See \033[1;31mhttps://github.com/micropython/micropython/wiki/Build-Troubleshooting\033[0>
Executing action: all (aliases: build)
Running cmake in directory /build/source/ports/esp32/build-ESP32_GENERIC_C3
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DPYTHON=/nix/store/41253d4jjd23jjdaiidgnb8q2qambh1a-python3-3.11.7-env/bin/python3.11 -DESP_PLATFORM=1 -DMICROPY_BOARD=ESP32_GENERIC_C3 -DMICROPY_BOARD_DIR=/build/source/ports/esp32/boards/ESP32_G>
-- Found Git: /nix/store/bd0dcc9qzpfz5xfz7fm6cwf7q64cyfhi-git-2.44.0/bin/git (found version "2.44.0")
-- git rev-parse returned 'fatal: detected dubious ownership in repository at '/nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2'

Any idea why the issue is persisting when run in a derivation?

[madmo]

@jonahbron This is related to the safe.directory setting of git. I completely disabled this security feature on my machine.

[jonahbron]

After hours of fiddling I got it to almost work. But now, it looks like the compiler is trying to download files over the network!

Dependencies lock doesn't exist, solving dependencies.
CMake Error at /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/build.cmake:540 (message):
  ERROR: Solver failed processing dependency "espressif/mdns" from the
  manifest file "/build/source/ports/esp32/main_esp32c3/idf_component.yml".

  HTTPSConnectionPool(host='components-file.espressif.com', port=443): Max
  retries exceeded with url: /components/espressif/mdns.json (Caused by
  NameResolutionError("<urllib3.connection.HTTPSConnection object at
  0x7ffff5c2b190>: Failed to resolve 'components-file.espressif.com' ([Errno
  -3] Temporary failure in name resolution)"))

Call Stack (most recent call first):
  /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/project.cmake:547 (idf_build_process)
  CMakeLists.txt:53 (project)

-- Configuring incomplete, errors occurred!
cmake failed with exit code 1, output of the command is in the /build/source/ports/esp32/build-ESP32_GENERIC_C3/log/idf_py_stderr_output_1601 and /build/source/ports/esp32/build-ESP32_GENERIC_C3/log/idf_py>
See https://github.com/micropython/micropython/wiki/Build-Troubleshooting
make: *** [Makefile:66: all] Error 1
make: Leaving directory '/build/source/ports/esp32'
CMake Error at /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/build.cmake:540 (message):
  ERROR: Solver failed processing dependency "espressif/mdns" from the
  manifest file "/build/source/ports/esp32/main_esp32c3/idf_component.yml".

  HTTPSConnectionPool(host='components-file.espressif.com', port=443): Max
  retries exceeded with url: /components/espressif/mdns.json (Caused by
  NameResolutionError("<urllib3.connection.HTTPSConnection object at
  0x7ffff5c2b190>: Failed to resolve 'components-file.espressif.com' ([Errno
  -3] Temporary failure in name resolution)"))

Call Stack (most recent call first):
  /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/project.cmake:547 (idf_build_process)
  CMakeLists.txt:53 (project)

This might be make building purely in a flake a no-go 😢

[jonahbron]

It's not my favorite, but the solution (plus a bunch of other little fixes) was to make it into a known-output derivation; adding a outputHash prop etc. so that the derivation could make the network calls to Espressif for pulling in dependency components.

Here is my flake for anyone interested in how it works.

https://git.sr.ht/~jonahbron/doorbell/tree/963d4cc33cbba10d555237a4b4e2ef8df6f24776/item/mpy-flake/flake.nix

Take specific note of the workaround I applied to the Git problem, specifically marking only the correct directory safe, and using the GIT_CONFIG_GLOBAL env var to apply that config.

I'd love to see this PR submitted, it's helping me immensely!

[jonahbron]

Settled onto what I think is quite a nice Nix expression for this purpose.

https://git.sr.ht/~jonahbron/doorbell/tree/83e24a1477a65b0d3902d2492baab623cfc7baed/item/firmware.nix

Hopefully this can be a useful reference so nobody has to burn as much time figuring this out as I have 😄

[jonahbron]

Actually, because I necessarily have to have the Git fix in my project too, this PR has no efffect. Removing it my build still works.

https://git.sr.ht/~jonahbron/doorbell/tree/701ca6b16861b7183e72719be843a34308d2cc5b/item/flake.nix

@madmo At this point the issue might only be coming down to the "dubious ownership" issue, which I've solved with this code:

https://git.sr.ht/~jonahbron/doorbell/tree/701ca6b16861b7183e72719be843a34308d2cc5b/item/firmware.nix#L133