Using the raspberrypi foundation repository directly / git sparse checkout

[dragetd]

According to the readme on https://github.com/Hexxeh/rpi-firmware, that repository is a mirror of https://github.com/raspberrypi/firmware

But it only includes the necessary sub-folders for rpi-update. According to [1] it is possible since git-1.7 to clone only parts of a repository. The setup is a bit hacky but this might solve the issue.

The repository would not have to be maintained anymore. Reducing the workload to mirror it and enabling users to directly load the latest files.

[1] http://stackoverflow.com/questions/600079/is-there-any-way-to-clone-a-git-repositorys-sub-directory-only/13738951#13738951

I have not tested the sparse checkout yet, but thought I'd share.

[Ruffio]

@dragetd what is the issue here? If there is no issue then please close this issue. Thanks

[dragetd]

I tried to explain above. The copy of the original repository here is not needed if git sparse checkout feature was use and rpi-update would pull the sources directly from https://github.com/raspberrypi/firmware.

I thought about trying to do the changes myself, but then I realized that not git is used but instead curl to download an archive of the repository.

So git sparse checkout will not help here. In https://github.com/Hexxeh/rpi-update/blob/master/README.md the suggestion is that the upstream-repository might split boot off into a separate repository. That is something that might be a good idea, but will not really help since raspi-update is also using the modules and vc sub-folder. So these would have to be split into other repositories as well. Which would not be a very sane approach.

Looking at the downloads, we are talking about 55mb (hexxeh) vs 116mb (the upstream archive), i.e. an extra 61mb temporary storage to update the kernel. I think this is a small drawback compared to having one less repository that needs to be cared of and one less link in the trust-chain.

But after all, it might not be a bad idea to swtich to git instead of curl. It would not only allow the sparse checkout, but also reduce some of the 'what is the latest version' logic etc. and eventually simplify the script even.

Any thoughts?

[Ruffio]

@dragetd I'm in no way an advanced user nor have I any influence in how any of these RPi Gits are operated. From what I read from your post, changing to your suggestion will increase/double the amount of data to download when performing an update? If that is the case, then I don't think that is a good idea, as the units are often operated in low bandwidth environments with small sd-cards, so the less data, the better.

Regarding the logic of determine newest version, I have no knowledge of how that is working, but I haven't seen any issues so I guess there are none.

As I wrote, I'm in no way an advanced user, just happy that upgrading the firmware is so easy as it is. What is the key benefit of changing and who will benefit from it? In my experience changing a working setup takes a lot of convincing/work and must have some/a big selling point before the change is performed. I'm not saying that your suggestion isn't good, but with my lack of knowledge/usages I can't see it.

What is your use case/needs, since you suggest this change?

[dragetd]

So far I was under the impression that repository Hexxeh/rpi-firmware was a private repository of the user Hexxeh who initially developed rpi-update. This utility rpi-update pulls the kernel updates from the same repository instead of the repository from the RPi-Foundation.

Purely theoretical case here, bear with me: Why would I trust Hexxeh? It he was to add a backdoor in the binaries he would have near-instant root to thousands or millions of RPi devices. Ofcourse the RPi-Foundation could be infiltrated/hacked too, just like my own PC. And the rpi-update script could be malicious. But auditing rpi-update is a lot easier than some binary blobs. The trust-chain is currently RPi-Foundation -> Hexxeh -> Me. I would prefer RPi-Foundation -> Me. This would make the trust-chain more robust.

I do see the point that we want to keep the size as small as possible (bandwith, sd-cards…). If rpi-update was changed to using git instead of curl, it could selectively checkout only the needed files (we would have the small size) but could do it from the RPi-Foundation repository (we would have the short trust-chain).

Ofcourse, this is a non-trivial change.

[Ruffio]

@dragetd I do see your points and you are correct, Hexxah/rpi-firmware is a private repository, so one could be harsh and say, that if you don't trust him, then don't use his tools. I also agree that the foundation should have their own/official firmware updater as new firmwares are generated frequently, just as every 'normal pc' manufacture has their own firmware updater and doesn't rely on third party. One could ague that RPi is based on volunteers (like me) so this is just another example of volunteer projects around the RPi, but one could also ague that the firmware is such a vital and essential part of the product, so they should have their 'own' official updater especially since this: https://www.theguardian.com/technology/2015/feb/18/raspberry-pi-becomes-best-selling-british-computer

To be to be honest I don't think that this is going to be changed anytime soon (but I don't know) as it has worked perfectly (IMHO) until now and developers time are better used fixing bugs. Some say, why fix it when it ain't broken?

My suggestion is, that this issue is closed as this Gist is a private one not under the foundation and that you raise your concerns/feature request directly to the foundation. That is just my opinion.

[dragetd]

If someone would provide a PR that would implement updating via git (and via sparse checkout from the foundation repository), would such a patch be considered or rejected right away? Just asking. :)

[lurch]

Have a look at who commits updates to raspberrypi/firmware ( https://github.com/raspberrypi/firmware/commits/master ), and then compare that to who commits updates to Hexxeh/rpi-firmware ( https://github.com/Hexxeh/rpi-firmware/commits/master ) and who commits (the majority of) updates to Hexxeh/rpi-update ( https://github.com/Hexxeh/rpi-update/commits/master ) ;-D You may find it interesting comparing the commit-messages in raspberrypi/firmware to the commit-messages in Hexxeh/rpi-firmware too. I don't think you have anything to worry about.

I might be wrong, but IIRC rpi-update used to work by doing a shallow git clone of Hexxeh/rpi-firmware, but that became too slow and problematic, so it was changed to downloading tarballs from github using wget instead, which was later changed to use curl when there were issues with wget validating github's SSL certificates.

[dragetd]

I see, thanks for the info! I still believe this could be done more elegantly, but eh… :P

Closing this then.