search

hydrabus / hydrafw

HydraBus HydraFW official firmware for open source multi-tool for anyone interested in learning/developping/debugging/hacking/Penetration Testing for basic or advanced embedded hardware
https://hydrabus.com/hydrabus-1-0-specifications
Apache License 2.0
362 stars 92 forks source link

CI/CD basic implementation based on GitHub Actions #167

Closed ia closed 5 months ago

ia commented 5 months ago

TL;DR: utilize GitHub capabilities to make builds on every commit and on every pull-request.

Hello. I'm a bit surprised that this haven't been done yet. So I decided to take some liberty to help with that. There may be a lot of room for improvements in the future but this is work-able proof-of-concept which can be merged & adopted as is currently in my humble opinion. However, to fully integrate push.yml into hydrafw github project, admins may need to set some additional settings somewhere in Settings of the project using GitHub web interface manually.

The benefits are:

Partially this overlaps with #132 and I can't promise anything but every time I have some free time I try to keep contributing around these CI/CD improvements as well.

Now, to technical details so everyone who is interested & curious could be on the same page. Basically, push.yml here is the most minimal declarative file which tells GitHub to start Ubuntu 22.04 VM, then to start Debian oldstable container inside, install toolchain packages, build hydrafw.dfu and upload it as hydrafw.dfu artifact. The live "demo" can be seen in my forked repo here:

Oh, and that tiny update for python script comes from specific behavior of GitHub actions: on every checkout before build, GitHub infrastructure doesn't make full valid git checkout but downloads source tree in detached state so getting meta information about repo and its status (tag, revision, sha, etc.) may be tricky because I saw that with other GitHub projects so for a while we set revision as 0.0 for GitHub builds (which is default value) but it's even good in its own way because by that it will be the indicator that it is the latest upstream mainline build but not a stable release.

ia commented 5 months ago

Aaaand off we go! So it seems it's picked automagically even with this pull-request right away for the main hydrabus/hydrafw project:

bvernoux commented 5 months ago

Thanks for the contribution but could you use the official tested & validated GCC ARM 4.9 2015q3(GNU_ARM_4_9_2015q3) used and defined in https://github.com/hydrabus/hydrafw/wiki/how-to-build-flash-and-use-hydrafw-on-linux ?

ia commented 5 months ago

could you use the official tested & validated GCC ARM 4.9 2015q3(GNU_ARM_4_9_2015q3) used and defined in https://github.com/hydrabus/hydrafw/wiki/how-to-build-flash-and-use-hydrafw-on-linux ?

Sure! Thanks for pointing this out. I perfectly understand that for specific hardware projects it may be very essential to have reference devel environment to avoid a chance of wasting time to deal with toolchain problem instead of improving the project itself. I just forgot about that GCC version requirement for hydrafw since I didn't have any issues myself with host arm-none-eabi toolchain on 18.04 [yet].

Basically, I just need to think how better incorporate downloading, unpacking & setting that toolchain. Will try to be back with results today or tomorrow.

ia commented 5 months ago

After very brief look, it looks good to me. And these two log lines give a hint that now the reference toolchain is used. And here is the link to the latest successful build, in case if you (as far as I understand) is interested in some kind of reproducible build analysis of the resulting dfu file (direct link to zip with hydrafw.dfu after setting toolchain).

P.S. For the record: I'm not a big fan of modern Dev[Put Something Else Fancy Today Here]Ops over-hyped buzzwords but even that may be useful with reasonable approach. And what I really like in other Free & Open source projects is that when you can just go to download/build section/tab and download the latest build/app without a need to spend time to set up devel environment yourself, especially if you just want to try/test something "right here and right now" and you're in a rush. So since hydrafw is located on GitHub and since GitHub provides such infrastructure for free (as in beer), why wouldn't utilize it for the benefits of the project, right? ;)

bvernoux commented 5 months ago

I have checked your latest build https://github.com/hydrabus/hydrafw/actions/runs/8449573440/artifacts/1362399528 and it seems there is an issue in the Firmware version embedded in the DFU HydraFW (HydraBus) 1738895 2024-03-27 instead of expected HydraFW (HydraBus) v0.11-22-g34e3d63 2024-02-23

ia commented 5 months ago

I have checked your latest build https://github.com/hydrabus/hydrafw/actions/runs/8449573440/artifacts/1362399528 and it seems there is an issue in the Firmware version embedded in the DFU HydraFW (HydraBus) 1738895 2024-03-27 instead of expected HydraFW (HydraBus) v0.11-22-g34e3d63 2024-02-23

Aha. Well, I may be wrong but probably I was not so clear presenting this feature. So what you're looking for in hydrafw.dfu is a version string. But this string can be seen only in binary which is built from git release tag with v0.11 value and with related to that tag git sha1 commit id which is g34e3d63. But builds on GitHub actions are "baked" by checking out the most recent commit from main upstream devel branch (which is master) so version string should be different depends on commit.

However, the mystery is why 1738895? It seems I need more time to figure out what's what and how exactly this string is generated diving a bit deeper into related python scripts. But I appreciate the interest to this work. Now we all need to agree on some kind of version string format which will be indicative to determine tag/branch/commit_id but at the same time won't be pain in the neck to get this meta information during build, whether local or github or from tarball (where git meta info is not available at all). My suggestion is something like: HydraFW (HydraBus) GIT_TAG_if_available GIT_SHA_ID_if_availablee BUILD_DATE. But this is just for reference. So just tell me how it should be and I will try to figure something out.

bvernoux commented 5 months ago

I have checked your latest build https://github.com/hydrabus/hydrafw/actions/runs/8449573440/artifacts/1362399528 and it seems there is an issue in the Firmware version embedded in the DFU HydraFW (HydraBus) 1738895 2024-03-27 instead of expected HydraFW (HydraBus) v0.11-22-g34e3d63 2024-02-23

Aha. Well, I may be wrong but probably I was not so clear presenting this feature. So what you're looking for in hydrafw.dfu is a version string. But this string can be seen only in binary which is built from git release tag with v0.11 value and with related to that tag git sha1 commit id which is g34e3d63. But builds on GitHub actions are "baked" by checking out the most recent commit from main upstream devel branch (which is master) so version string should be different depends on commit.

However, the mystery is why 1738895? It seems I need more time to figure out what's what and how exactly this string is generated diving a bit deeper into related python scripts. But I appreciate the interest to this work. Now we all need to agree on some kind of version string format which will be indicative to determine tag/branch/commit_id but at the same time won't be pain in the neck to get this meta information during build, whether local or github or from tarball (where git meta info is not available at all). My suggestion is something like: HydraFW (HydraBus) GIT_TAG_if_available GIT_SHA_ID_if_availablee BUILD_DATE. But this is just for reference. So just tell me how it should be and I will try to figure something out.

I really prefer to keep it like it is today with Release using latest release version "v0.11" (or newer) then use number of new commits since that official release see https://github.com/hydrabus/hydrafw/releases/tag/v0.11 today we have 22 commits after v0.11 so it shall be formatted v0.11-22 then it shall use latest commit of the master shortened which is https://github.com/hydrabus/hydrafw/commit/34e3d63189762a1138da4e1fffc7b0922ae9f186 so g34e3d63 and the Date of that latest commit so you have at end HydraFW (HydraBus) v0.11-22-g34e3d63 2024-02-23 You can check https://github.com/hydrabus/hydrafw/blob/master/src/build-scripts/hydrafw-version.py it format it like that and it is embedded in the Makefile see https://github.com/hydrabus/hydrafw/blob/34e3d63189762a1138da4e1fffc7b0922ae9f186/src/Makefile#L344

ia commented 5 months ago

I have checked your latest build https://github.com/hydrabus/hydrafw/actions/runs/8449573440/artifacts/1362399528 and it seems there is an issue in the Firmware version embedded in the DFU HydraFW (HydraBus) 1738895 2024-03-27 instead of expected HydraFW (HydraBus) v0.11-22-g34e3d63 2024-02-23

Aha. Well, I may be wrong but probably I was not so clear presenting this feature. So what you're looking for in hydrafw.dfu is a version string. But this string can be seen only in binary which is built from git release tag with v0.11 value and with related to that tag git sha1 commit id which is g34e3d63. But builds on GitHub actions are "baked" by checking out the most recent commit from main upstream devel branch (which is master) so version string should be different depends on commit. However, the mystery is why 1738895? It seems I need more time to figure out what's what and how exactly this string is generated diving a bit deeper into related python scripts. But I appreciate the interest to this work. Now we all need to agree on some kind of version string format which will be indicative to determine tag/branch/commit_id but at the same time won't be pain in the neck to get this meta information during build, whether local or github or from tarball (where git meta info is not available at all). My suggestion is something like: HydraFW (HydraBus) GIT_TAG_if_available GIT_SHA_ID_if_availablee BUILD_DATE. But this is just for reference. So just tell me how it should be and I will try to figure something out.

I really prefer to keep it like it is today with Release using latest release version "v0.11" (or newer) then use number of new commits since that official release

No problem. Maybe I will figure out eventually how to do that, maybe not because this is very tricky version string policy. I mean, I've just never heard about counting commits.

But now I'm just really very curious to understand... because the more it will be clear to me, the more chances that I will do it as demanded without making some mistakes because something unclear to me. So...

see https://github.com/hydrabus/hydrafw/releases/tag/v0.11 today we have 22 commits after v0.11 so it shall be formatted v0.11-22

... correct me if I'm wrong but as far as I understand, v0.11 is just a git tag, and git tag is just a human readable alias with the pointer to the specific commit in the past. Once any commit is committed on top of the tagged commit, tagged commit is losing it's "recent" status and become out-of-dated by definition. But here it should be not only kept but on top of that there is counting of number of commits between release tag and the most recent commit on top of upstream devel branch... why so complicated? Git and its commit id system allows to establish exact source tree state from which build has been produced just by sha id. Tags can be embedded if it's build from commit with the exact tagged sha id. I'm just definitely missing something here which I really try to understand, sorry to bother with these questions.

UPD. Okay, I see that this is some sort of utilization of git describe --tags... but why they would add commit counter in the first place... interesting. Anyway, you see, another one tricky part: if build is produced from pull-request [feature] branch like here from my ia:master branch where I already currently have ~50 commits, then what version string should be for the builds from ia:master here:

a) v0.11-73-g595a6e1 (which is technically valid but not correct from the look of hydrafw:master especially when after merge 73 turns into 23);

b) v0.11-22-g595a6e1 (which is correct from the look of hydrafw:master but tag 595a6e1 originally corresponds to commit inside feature branch and not inside hydrafw:master which may be a bit confusing in cases of searching for regression or triaging an issue);

c) v0.11-22-g34e3d63 (which makes the source tree state untraceable at all because the meta information doesn't correspond to the actual source tree state at the time of build because this version string won't be pointing out to the source tree state at the time of build which is the whole point of version string as far as I understand);

?

P.S. Regardless, today I learned so much already. Much appreciate the feedback. :pray:

ia commented 5 months ago

Ok, it seems that after all with the power of bash & git commands we can get all needed meta information at the time of building on github to embed version string.

The question remains what string should be put into hydrafw.dfu, so considering my comment above just let me know and I will do some further modifications. I hope my comments are not looking rude in any way because I was just trying to figure things out. Will return to this in a day or too.

bvernoux commented 5 months ago

Ok, it seems that after all with the power of bash & git commands we can get all needed meta information at the time of building on github to embed version string.

The question remains what string should be put into hydrafw.dfu, so considering my comment above just let me know and I will do some further modifications. I hope my comments are not looking rude in any way because I was just trying to figure things out. Will return to this in a day or too.

We must just build it like it was done on computer it create src/common/hydrafw_version.hdr which is then used during build and those informations will be present in the final dfu (like it is in the bin ...) so there is nothing more to do Example content of src/common/hydrafw_version.hdr created during build with latest commit:

#define HYDRAFW_GIT_TAG "v0.11-22-g34e3d63"
#define HYDRAFW_CHECKIN_DATE "2024-02-23"

In my case I'm using https://github.com/hydrabus/hydrafw of course in your case with your fork you will have different commit ... When I check your fork https://github.com/ia/hydrafw you have +60 commits more (but your issue is that you do not have any tags in your fork see https://gist.github.com/kayagultekin/557975964ae26b99b46be3082ee828cf#what-about-syncing-tags) vs https://github.com/hydrabus/hydrafw and your latest commit https://github.com/ia/hydrafw/commit/1e882a150de410e4049a4159ad6c955c168304e2 done on 2024-02-27, so we should have src/common/hydrafw_version.hdr:

#define HYDRAFW_GIT_TAG "v0.11-82-g1e882a1"
#define HYDRAFW_CHECKIN_DATE "2024-02-27"

At the end we shall find in the bin & dfu HydraFW (HydraBus) v0.11-82-g1e882a1 2024-03-27

ia commented 5 months ago

Ookay, some status update. Here is the content of hydrafw_version.hdr from local build on my local machine from my git feature branch:

#define HYDRAFW_GIT_TAG "v0.11-85-gb0f78b2"
#define HYDRAFW_CHECKIN_DATE "2024-03-29"

And here is the content of hydrafw_version.hdr from GitHub build here:

$ strings hydrafw.dfu  | grep v0.11
HydraFW (HydraBus) v0.11-86-gb0f78b2 2024-03-29

Are these appropriate results? Just asking for confirmation before cleaning up all those printf debugging stuff from scripts before final merge.

ia commented 5 months ago

Hello. Sorry to bother, but I'm just giving a friendly reminder of the most recent update since I'm very curious what you think.

So, could you, please, just give me the feedback about the current patch-set for this pull-request. Although I understand that we all may be very busy sometimes so take your time, of course, so I'm not rushing, but just giving an update. Thank you.

bvernoux commented 5 months ago

All seems good for me Baldanos what do you think ?

Baldanos commented 5 months ago

Looks good to me.

@ia Thank you for the contribution !

bvernoux commented 5 months ago

Thanks for your contribution

bvernoux commented 5 months ago

I have created a new issue https://github.com/hydrabus/hydrafw/issues/168 as the build has failed (after merge)