`mix firmware` failure during squashfs

[jjcarstens]

With squashfs 4.6+ on the host machine, some mix firmware builds may fail after building the rootfs and then squashing it

|nerves_bootstrap| Building OTP Release...

* skipping runtime configuration (config/runtime.exs not found)
* creating _build/rpi0_dev/rel/my_fw/releases/0.1.0/vm.args
Updating base firmware image with Erlang release...
Copying rootfs_overlay: .../_build/rpi0_dev/nerves/rootfs_overlay
Copying rootfs_overlay: .../my_fw/rootfs_overlay
** (Mix) Nerves encountered an error. %IO.Stream{device: :standard_io, line_or_bytes: :line, raw: true}

The workaround is to downgrade squashfs to 4.5 and things should work again.

We should investigate the breakage with squashfs 4.6 and how to mitigate

[LostKobrakai]

The bit of debugging I did seemed to fail in a unsquashfs call of ~/.nerves/artifacts/our-fw/scripts/merge-squashfs

[fhunleth]

I could use some help reproducing this. I have had Squashfs 4.6.1 installed on two MBP's for quite a while now and haven't seen this at all. Does this affect all projects? Aside from work projects, I build Elixir Circuits Quickstart and Nerves Livebook frequently. Does it reproduce for either of you with those?

[jjcarstens]

It is a bit stalled until we have some reproduction. A few others are looking into it and we'll report once we have more

[jjcarstens]

I was able to reproduce this with a private repo. Once digging through the scripts and commands being run during squashing, this is what I ran into

FATAL ERROR: write_file: file squashfs-root/usr/lib/xtables/libip6t_hl.so already exists

Frank gave me the idea to check for files without case-sensitivity and sure enough, there are 2

> my_fw √ % grep -i libip6t_hl /var/folders/l9/61kntssd7g19yw5h4l9qxsr00000gn/T/tmp.P40vXDGq/pseudofile
/usr/lib/xtables/libip6t_HL.so m 755 root wheel
/usr/lib/xtables/libip6t_hl.so m 755 root wheel

MacOS is case-insensitive by default. This actually hits at a lots of random times. For cases during squashfs, it's probably running into files like this that are provided by the underlying Nerves system and is going to vary widely.

I need to think about how to handle this. I'm assuming the recent squashfs update has some sort of change to ignore case and just need to see how to work around it

[schrockwell]

For folks who need to downgrade squashfs on macOS using Homebrew:

brew tap-new zappy/edith
brew extract --version="4.5.1" squashfs zappy/edith
brew install squashfs@4.5.1
brew link --overwrite squashfs@4.5.1
unsquashfs -version # => should report 4.5.1

[SteffenDE]

Just a warning that building a firmware on macOS actually leads to a somewhat corrupted system.

I recently spent a couple of hours trying to find out why a specific iptables command failed on a nerves firmware built on macOS using the downgraded version of squashfs that has been suggested. I later noticed that a system built on linux works just fine.

If you're seeing logs like these it could happen that some things will not work as expected:

Pseudo modify file "lib/modules/5.15.138/kernel/net/netfilter/xt_dscp.ko" does not exist in source filesystem.  Ignoring.
Pseudo modify file "lib/modules/5.15.138/kernel/net/netfilter/xt_hl.ko" does not exist in source filesystem.  Ignoring.
Pseudo modify file "lib/modules/5.15.138/kernel/net/netfilter/xt_rateest.ko" does not exist in source filesystem.  Ignoring.
Pseudo modify file "lib/modules/5.15.138/kernel/net/netfilter/xt_tcpmss.ko" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libip6t_hl.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libipt_ttl.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_connmark.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_dscp.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_mark.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_rateest.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_set.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_tcpmss.so" does not exist in source filesystem.  Ignoring.
Pseudo modify file "usr/lib/xtables/libxt_tos.so" does not exist in source filesystem.  Ignoring.

[salseeg]

I was able to reproduce this with a private repo. Once digging through the scripts and commands being run during squashing, this is what I ran into

FATAL ERROR: write_file: file squashfs-root/usr/lib/xtables/libip6t_hl.so already exists

Frank gave me the idea to check for files without case-sensitivity and sure enough, there are 2

> my_fw √ % grep -i libip6t_hl /var/folders/l9/61kntssd7g19yw5h4l9qxsr00000gn/T/tmp.P40vXDGq/pseudofile
/usr/lib/xtables/libip6t_HL.so m 755 root wheel
/usr/lib/xtables/libip6t_hl.so m 755 root wheel

MacOS is case-insensitive by default. This actually hits at a lots of random times. For cases during squashfs, it's probably running into files like this that are provided by the underlying Nerves system and is going to vary widely.

I need to think about how to handle this. I'm assuming the recent squashfs update has some sort of change to ignore case and just need to see how to work around it

@jjcarstens Have just added issue about this - #937 I worked around this by replacing mktemp and moving project to case sensitive volume

[schrockwell]

I went down this road again and came to the same conclusion as @salseeg. Would it make sense to add a $NERVES_TMP_DIR environment variable that can be passed to mktemp?

I also found that mktemp -d -p '' will honor $TMPDIR which should already exist, so there would be no need to define a special environment variable.

[joserpintuitivo]

For folks who need to downgrade squashfs on macOS using Homebrew:

brew tap-new zappy/edith
brew extract --version="4.5.1" squashfs zappy/edith
brew install squashfs@4.5.1
brew link --overwrite squashfs@4.5.1
unsquashfs -version # => should report 4.5.1

This is not working for me, I changed my Mac and installed version 4.6.1 on homebrew, I am not being able to downgrade to 4.5.1