search

t2linux / wiki

Repository for the t2linux.org wiki
https://wiki.t2linux.org
Creative Commons Attribution Share Alike 4.0 International
166 stars 60 forks source link

Firmware: Add support for retrieving firmware from macOS Recovery Image #540

Closed AdityaGarg8 closed 1 month ago

AdityaGarg8 commented 1 month ago

@soopyc , you may want to consider this PR when updating nixos guide

AdityaGarg8 commented 1 month ago

@soopyc , can you test this on nixos?

AdityaGarg8 commented 1 month ago

@sharpenedblade , can you test on fedora?

AdityaGarg8 commented 1 month ago

@NoaHimesaka1873 @Redecorating , can you check the code for Arch. I couldn't find a better way to get dmg2img from aur.

NoaHimesaka1873 commented 1 month ago

That seems like the most sane way to install AUR package without helper.

AdityaGarg8 commented 1 month ago

That seems like the most sane way to install AUR package without helper.

Nice!

AdityaGarg8 commented 1 month ago

The macOS recovery image method occasionally fails with python showing error with wifi. I've noticed that symlinks are being affected in such cases, most probably due to bad support for hfsplus on Linux. The script shows a message to try again with some other macOS version in such cases.

AdityaGarg8 commented 1 month ago

In case someone tests and it fails at 1st attempt, but succeeds on automatic retry, do share the no of retries.

soopyc commented 1 month ago

@soopyc , can you test this on nixos?

The script does not work OOTB (expected), but fortunately it's possible to temporarily make packages available to use with one of the following commands.

While possible, the notion of installing packages with commands is discouraged on NixOS so I personally think the script should halt when it detects both the distro and missing packages.

Nevertheless, with this method the same thing about manually creating /lib/firmware/brcm applies. Apart from these 2 issues, the script works on both the installation environment and installed NixOS. (N.B.: The script took 5 automatic attempts and a re-run to succeed on the Live environment.)

Instructions could be added to the package manager checks, and one can detect NixOS by checking if /etc/NIXOS exists.

Minor nitpick - when creating temporary directories, prefixing them with something like firmware-{work,mount} makes things easier to manage while testing.

AdityaGarg8 commented 1 month ago

@soopyc , can you test this on nixos?

The script does not work OOTB (expected), but fortunately it's possible to temporarily make packages available to use with one of the following commands.

  • nix shell nixpkgs#dmg2img -c bash ./firmware.sh
  • nix-shell -p dmg2img --run bash ./firmware.sh

While possible, the notion of installing packages with commands is discouraged on NixOS so I personally think the script should halt when it detects both the distro and missing packages.

Nevertheless, with this method the same thing about manually creating /lib/firmware/brcm applies. Apart from these 2 issues, the script works on both the installation environment and installed NixOS. (N.B.: The script took 5 automatic attempts and a re-run to succeed on the Live environment.)

I'll have to fix this thing first. Thanks for testing!

Instructions could be added to the package manager checks, and one can detect NixOS by checking if /etc/NIXOS exists.

I'm not very sure what to put here.

Minor nitpick - when creating temporary directories, prefixing them with something like firmware-{work,mount} makes things easier to manage while testing.

AdityaGarg8 commented 1 month ago

@soopyc , can you test this now. Also share how many retries it took.

soopyc commented 1 month ago

@soopyc , can you test this now. Also share how many retries it took.

I will agree that this issue is very annoying to debug as the errors happen inconsistently. I've tried the [latest revision of the] script on various occasions, and the success rate is 80%, where the script works first try without any re-attempts. The other 20% fails completely and no automatic retries work, just like before. A re-run of the script after a while usually works. I'm starting to wonder if this is an image problem.

AdityaGarg8 commented 1 month ago

@soopyc , can you test this now. Also share how many retries it took.

I will agree that this issue is very annoying to debug as the errors happen inconsistently. I've tried the [latest revision of the] script on various occasions, and the success rate is 80%, where the script works first try without any re-attempts. The other 20% fails completely and no automatic retries work, just like before. A re-run of the script after a while usually works. I'm starting to wonder if this is an image problem.

What I have done till now:

  1. When the script fails, I've backed up the converted .img file which was causing the failure. I mounted that again manually and it magically worked. Interestingly, that was the principle of the previous retry bit of the previous revision you tested.

  2. Now I use cp to first copy the firmware from the .img and run python rename firmware bit over there. In such cases when it fails, again I had a backup of the copied script. Here things get stranger. If I run python3 firmware.sh ${workdir}/firmware fw.tar, I get the same error. But, if I copy the SAME firmware folder to another place, say $HOME, and run python3 firmware.sh $HOME/firmware fw.tar, it succeeds! I made this as the principle of the latest revision retries, but they don't seem to be working.

AdityaGarg8 commented 1 month ago

@soopyc , I've added some debug info to the script. Can you share the error generated by python. (Un)fortunately the script is succeeding every time for me.

AdityaGarg8 commented 1 month ago

@soopyc , I've added some debug info to the script. Can you share the error generated by python. (Un)fortunately the script is succeeding every time for me.

Finally got the error. Apparently it's not able to get the symlinks properly sometimes.

AdityaGarg8 commented 1 month ago

@soopyc , whenever possible, can you test the latest revision of the script in this pr. I've attempted to get firmware using tar by fixing path since lstrip was sometimes breaking folder names, print a detailed error, and removed the retries, since they anyways were useless.

Also, in case you get error/warning, please share that as well.

AdityaGarg8 commented 1 month ago

I believe the bug has finally been fixed. Apparently if a temp directory had a "C" in its name, the lstrip command broke the code by removing "C" from the folder names in the wifi folder of source firmware. My previous assumption of broken symlinks was wrong.

Still I need tests and feedbacks from more people. @soopyc , it would be nice if you could test, since you were able to reproduce the bug before.

soopyc commented 1 month ago

Thanks for figuring out the bug! I wouldn't have been able to figure that out myself without a few more days :P

Anyways, I've tried running the script normally and forcing a C into the tmpdir. Both runs correctly, so unless I've messed up the position of the C, this looks good from my side.

AdityaGarg8 commented 1 month ago

Thanks for figuring out the bug! I wouldn't have been able to figure that out myself without a few more days :P

Anyways, I've tried running the script normally and forcing a C into the tmpdir. Both runs correctly, so unless I've messed up the position of the C, this looks good from my side.

Can you try running the way you did before, maybe there could be some other hidden bug.

AdityaGarg8 commented 1 month ago

@soopyc , can you test this now. Also share how many retries it took.

I will agree that this issue is very annoying to debug as the errors happen inconsistently. I've tried the [latest revision of the] script on various occasions, and the success rate is 80%, where the script works first try without any re-attempts. The other 20% fails completely and no automatic retries work, just like before. A re-run of the script after a while usually works. I'm starting to wonder if this is an image problem.

This way

AdityaGarg8 commented 1 month ago

Thanks for figuring out the bug! I wouldn't have been able to figure that out myself without a few more days :P Anyways, I've tried running the script normally and forcing a C into the tmpdir. Both runs correctly, so unless I've messed up the position of the C, this looks good from my side.

Can you try running the way you did before, maybe there could be some other hidden bug.

My bad, I just read your tried to run normally.

AdityaGarg8 commented 1 month ago

Thanks for testing @soopyc !

AdityaGarg8 commented 1 month ago

Is anything nixos specific required over here?

soopyc commented 1 month ago

The only other NixOS specific thing is to check for /lib/firmware/brcm and not just /lib/firmware with this method, as tar errors and quits if that doesn't exist.

AdityaGarg8 commented 1 month ago

The only other NixOS specific thing is to check for /lib/firmware/brcm and not just /lib/firmware with this method, as tar errors and quits if that doesn't exist.

You probably can then update your other pr to add instructions to run this way then.

AdityaGarg8 commented 1 month ago

Also, https://github.com/t2linux/wiki/commit/8f1e8bc9bd9e4ada51f108366786a33896d0128e

AdityaGarg8 commented 1 month ago

Updated the wiki. Preview on https://adityagarg8.github.io/wiki/guides/wifi-bluetooth/

A review is required.

AdityaGarg8 commented 1 month ago

Just mention that you need a wired internet connection for method 4

https://github.com/t2linux/wiki/pull/540/files#diff-b12df5aa5ceba67e40ea53c7f633b0d49c17a5ee909608930f740b3cbd77fd23R188