confused with the latest changes - one branch?

[perexg]

It seems that this repo moved again from the branch system (one branch = one version) to all-in-one branch. It's a bit unclear to me, how I can fetch one version for packaging. And it's also unclear, how you'll tag the releases.

[lgirdwood]

@perexg sorry again, we've had a lot of discussion on how to improve this with the SOF developers, and @marc-hb has been improving how we move forward here. We have a pending Readme patch to be merged before we tag today, i.e. the binaries are all correct but we will tag once the readme update is merged today.

[perexg]

Ok, but how I can fetch only v1.8 with the current scheme? If you tag the main branch, I'll fetch both 1.7 and 1.8. And if there are 10 releases, I'll end to fetch all of them.

The solution may be to filter the files in the release assets, but they have wrong names (v1.7.tar.bz instead sof-bin-1.7.tar.gz) or return back to the release branches.

[lgirdwood]

@marc-hb please also tag the release. You have maintainer access so should be able to do this via GH or git. The known issues are here https://github.com/thesofproject/sof/milestone/15 for linking to in release.

[lgirdwood]

Ok, but how I can fetch only v1.8 with the current scheme? If you tag the main branch, I'll fetch both 1.7 and 1.8. And if there are 10 releases, I'll end to fetch all of them.

The solution may be to filter the files in the release assets, but they have wrong names (v1.7.tar.bz instead sof-bin-1.7.tar.gz).

I think @marc-hb has some flows for installation. Lets wait until he's online later today.

[marc-hb]

It seems that this repo moved again from the branch system (one branch = one version) to all-in-one branch.

That's correct.

It's a bit unclear to me, how I can fetch one version for packaging.

You can't do that anymore. The longer term fix for that is to move away from using git to release binaries. For now the entire .git/ folder is 10M.

And it's also unclear, how you'll tag the releases.

As you noted, tagging the releases in sof-bin does not really make sense anymore.

The 1.7 version in the README.md is obviously an example.

[lgirdwood]

@marc-hb thanks for your inputs here, are there any docs in sof-docs or GH feature request that describe the release process roadmap and your current plans. I assume the end goal is to have a script that can create a tar.bz2 of FW, ldcs and topolgies for someone to attach to GH releases ?

[perexg]

You can't do that anymore. The longer term fix for that is to move away from using git to release binaries. For now the entire .git/ folder is 10M.

Just a note: From the storage perspective, it does not matter, if you store different source binary files to one branch or multiple branches. Perhaps, git merging may help, too.

I'm using the auto-generated tar balls by github now (using the commit-id not your tag which was floating previously). In this case, only the necessary files are downloaded. Your move to artifacts means that you plan to use own scripts for the release, which is also fine, but the storage can be used efficiently with the branches, too like:

• main branch (without binary files, only license files and scripts)
• version branch (with binary files for the specific version + merged main branch)

In this case, you can just tag the version branch and you'll get the automatic tar ball from github.

Example of commit-id URL to get the tar ball from github for v1.7: https://github.com/thesofproject/sof-bin/archive/6bd5501e6ac3f29dde42a28ffe03a2942aa60640/sof-bin-6bd5501.tar.gz .

[perexg]

Or you may create meta branch with scripts and doc files which will be merged to main (all branches) and to separate release branches. There are multiple possibilities.

[marc-hb]

I assume the end goal is to have a script that can create a tar.bz2 of FW, ldcs and topolgies ...

Most of this script already exists and I use it many times everyday: https://github.com/thesofproject/sof/tree/main/installer

It's still missing a few features: final tarball, checksums, compilation flags,... I was hoping to get some user feedback to help prioritize further development but I'm afraid most people still use xtensa-build-all.sh + build-tools.sh + their own, private scripts on top. Cause why change? This is also the case of our internal Jenkins-based build and validation; this mismatch makes each release time-consuming and error-prone.

It's been unsurprisingly hard to get people excited about packaging and releases so the only way forward has been slow: try something (not too) different and wait a bit to see if anyone is confused or complains. There :-)

... for someone to attach to GH releases ?

Yes, my idea was to eventually just upload the output of the installer directly to https://docs.github.com/en/github/administering-a-repository/releasing-projects-on-github/managing-releases-in-a-repository and deprecate the entire sof-bin repo: everything required to perform releases would be self-contained in the main sof repo, no external dependencies and no other branches or tag besides the "real" sof ones. Hopefully simple enough not to require any Powerpoint: configure and run the installer/, upload to github releases, done. No secret sauce hidden in Jenkins or wherever.

Example: https://github.com/zephyrproject-rtos/sdk-ng/releases

I'm using the auto-generated tar balls by github now (using the commit-id not your tag which was floating previously). In this case, only the necessary files are downloaded.

First, thanks for this clarification and your expert feedback in general: even more useful considering how rare you are.

I didn't even know github had such a convenient git archive implementation, sorry I "broke" that. I understand the convenience of tagging sof-bin now, I will do today.

Once I tag sof-bin this git archive feature should not really be "broken" any more, just less efficient. Do you mind ignoring the older versions until we move to github releases? Just a few spurious megabytes every few months.

For the record I "announced" the move to a single branch in January 2021 (among many other things, some abandoned). I understand that didn't make any headline :-) https://github.com/thesofproject/sof/pull/3775#issuecomment-766929974

Cc: @aiChaoSONG , @keyonjie , @fredoh9 , @plbossart , @mrhpearson @cujomalainey , please forward to any other (rare!) person who might have an interest for this topic.

[plbossart]

FWIW we added recently a pointer to sof-bin in the drivers:

sound/soc/sof/loader.c:                 "you may need to download the firmware from https://github.com/thesofproject/sof-bin/\n");
sound/soc/sof/topology.c:                       "you may need to download the firmware from https://github.com/thesofproject/sof-bin/\n");

you may change directions on how you want to deliver the binaries but that site has to stay.

[marc-hb]

Good point thanks! We can have a simple "redirection" in the top-level sof-bin/README.md

As you made no other comment, I assume you agree with all the rest? :-)

[plbossart]

As long as @perexg is happy I am happy.

[perexg]

Once I tag sof-bin this git archive feature should not really be "broken" any more, just less efficient. Do you mind ignoring the older versions until we move to github releases? Just a few spurious megabytes every few months.

I think that you can upload the manually generated assets to the release tag until you have the final solution. From my packager perspective, I would like just to change the source package URL in my rpm spec file. For 1.8, the assets should be the archive of the main branch with removed v1.7.x tree and 1.8*.rc trees / files. Only the final 1.8 files should be there.

Do not forget that we store a copy of the source packages in our build system forever, too.

BTW: It seems that this URL also works: https://github.com/thesofproject/sof-bin/archive/v1.7/sof-bin-1.7.tar.gz

For the record I "announced" the move to a single branch in January 2021 (among many other things, some abandoned). I understand that didn't make any headline :-) thesofproject/sof#3775 (comment)

I probably missed that for other things...

[marc-hb]

I think that you can upload the manually generated assets to the release tag until you have the final solution.

Good idea and close enough to the final solution anyway; just in a different repo for now. Will do after README.md #37 is merged and after I create the v1.8-rc2 tag in sof-bin.

From my packager perspective, I would like just to change the source package URL in my rpm spec file

Thanks, I think I understand better now. In fact I just found https://koji.fedoraproject.org/koji/packageinfo?packageID=30869. I ran dnf download --source alsa-sof-firmware on some Fedora box and had a look at the .spec and other files. I should have done this earlier.

Do not forget that we store a copy of the source packages in our build system forever, too.

Are you expressing a space concerning here and/or other(s)?

Just to be extra sure: in this particular case, "source package" means "sof-bin sources" which means firmware binaries, correct?

[perexg]

I think that you can upload the manually generated assets to the release tag until you have the final solution.

Good idea and close enough to the final solution anyway; just in a different repo for now. Will do after README.md #37 is merged and after I create the v1.8-rc2 tag in sof-bin.

From my packager perspective, I would like just to change the source package URL in my rpm spec file

Thanks, I think I understand better now. In fact I just found https://koji.fedoraproject.org/koji/packageinfo?packageID=30869. I ran dnf download --source alsa-sof-firmware on some Fedora box and had a look at the .spec and other files. I should have done this earlier.

The Fedora package sources are available also via web portal: https://src.fedoraproject.org/rpms/alsa-sof-firmware

Here is a straight change from the commit-id to the version style for 1.7 (sitting in my private repo until we know the final source URL for 1.8):

--- a/alsa-sof-firmware.spec
+++ b/alsa-sof-firmware.spec
@@ -4,8 +4,6 @@
 %global _firmwarepath  /usr/lib/firmware

 %global sof_version 1.7
-%global sof_commit 6bd5501e6ac3f29dde42a28ffe03a2942aa60640
-%global sof_shortcommit %(c=%{sof_commit}; echo ${c:0:7})

 %global tplg_version 1.2.4

@@ -16,7 +14,7 @@ Release:        1%{?dist}
 # See later in the spec for a breakdown of licensing
 License:        BSD
 URL:            https://github.com/thesofproject/sof-bin
-Source:         https://github.com/thesofproject/sof-bin/archive/%{sof_commit}/sof-bin-%{sof_shortcommit}.tar.gz
+Source:         https://github.com/thesofproject/sof-bin/archive/v%{sof_version}/sof-bin-%{sof_version}.tar.gz
 BuildRequires:  alsa-topology >= %{tplg_version}
 BuildRequires:  alsa-topology-utils >= %{tplg_version}
 Conflicts:      alsa-firmware <= 1.2.1-6
@@ -36,7 +34,7 @@ License:        BSD
 This package contains the debug files for the Sound Open Firmware project.

 %prep
-%autosetup -n sof-bin-%{sof_commit}
+%autosetup -n sof-bin-%{sof_version}

 mkdir -p firmware/intel/sof

Do not forget that we store a copy of the source packages in our build system forever, too.

Are you expressing a space concerning here and/or other(s)?

I only mean that the source package contents propagate to all distributions which need to deal with it somehow. So removing duplicates is really welcome.

Just to be extra sure: in this particular case, "source package" means "sof-bin sources" which means firmware binaries, correct?

Yes, I mean the source for the packaging here (thus the sof-bin tar or zip archive) - Source: line in the rpm spec file.

Thank you for your attention and your work on this.

[lgirdwood]

@mwasko who from your team can work with @marc-hb on aligning the signing process with the release process to make it less manual and error prone. I would imagine that some helper scripts could be used to take the signed output from the signing system and upload/copy to the release destination. This can be done for v1.9 so we have 3 months.

[marc-hb]

@perexg can you try this when you get a chance? https://github.com/thesofproject/sof-bin/releases/tag/v1.8-rc2

[perexg]

diff --git a/alsa-sof-firmware.spec b/alsa-sof-firmware.spec
index e61e51b..cc65f3b 100644
--- a/alsa-sof-firmware.spec
+++ b/alsa-sof-firmware.spec
@@ -3,7 +3,8 @@
 %define _binaries_in_noarch_packages_terminate_build   0
 %global _firmwarepath  /usr/lib/firmware

-%global sof_version 1.7
+%global sof_version 1.8
+%global sof_version_extra -rc2

 %global tplg_version 1.2.4

@@ -14,7 +15,7 @@ Release:        1%{?dist}
 # See later in the spec for a breakdown of licensing
 License:        BSD
 URL:            https://github.com/thesofproject/sof-bin
-Source:         https://github.com/thesofproject/sof-bin/archive/v%{sof_version}/sof-bin-%{sof_version}.tar.gz
+Source:         https://github.com/thesofproject/sof-bin/releases/download/v%{sof_version}%{sof_version_extra}/sof-bin-v%{sof_version}%{sof_version_extra}.tar.gz
 BuildRequires:  alsa-topology >= %{tplg_version}
 BuildRequires:  alsa-topology-utils >= %{tplg_version}
 Conflicts:      alsa-firmware <= 1.2.1-6
@@ -34,15 +35,15 @@ License:        BSD
 This package contains the debug files for the Sound Open Firmware project.

 %prep
-%autosetup -n sof-bin-%{sof_version}
+%autosetup -n sof-bin-v%{sof_version}%{sof_version_extra}

 mkdir -p firmware/intel/sof

 # we have the version in the package name
-mv sof-v%{sof_version}/* firmware/intel/sof
+mv v%{sof_version}.x/sof-v%{sof_version}%{sof_version_extra}/* firmware/intel/sof

 # move topology files
-mv sof-tplg-v%{sof_version} firmware/intel/sof-tplg
+mv v%{sof_version}.x/sof-tplg-v%{sof_version}%{sof_version_extra} firmware/intel/sof-tplg

 # remove NXP firmware files
 rm LICENCE.NXP

The paths are so complex, but it does work. Also, see the difference between v1.7 (easy) and v1.8-rc2 (complex).

[marc-hb]

I just verified that I can change the "v1.8-rc2 only" tarball like this with changing neither the .install.sh nor the README.md:

mv v1.8.x/* .
rmdir v1.8.x

EDIT: automated and submitted in #48

So this should look exactly like v1.7 and should let you get back to a single %{sof_version} variable, please confirm.

I just updated the sof-bin-v1.8-rc2.tar.gz accordingly. Note: I normally never change anything released and I'm surprised github let me do that without even a warning. However this is a release candidate and it's a new approach that I uploaded less than one hour ago. So I'm pretty confident you've been the only user so far :-)

[perexg]

Much better now (it's diff from 1.7 release):

diff --git a/alsa-sof-firmware.spec b/alsa-sof-firmware.spec
index e61e51b..5ae4f3b 100644
--- a/alsa-sof-firmware.spec
+++ b/alsa-sof-firmware.spec
@@ -3,18 +3,21 @@
 %define _binaries_in_noarch_packages_terminate_build   0
 %global _firmwarepath  /usr/lib/firmware

-%global sof_version 1.7
+%global sof_ver 1.8
+%global sof_ver_pre rc2
+%global sof_ver_rel %{?sof_ver_pre:.%{sof_ver_pre}}
+%global sof_ver_pkg v%{sof_ver}%{?sof_ver_pre:-%{sof_ver_pre}}

 %global tplg_version 1.2.4

 Summary:        Firmware and topology files for Sound Open Firmware project
 Name:           alsa-sof-firmware
-Version:        %{sof_version}
-Release:        1%{?dist}
+Version:        %{sof_ver}
+Release:        1%{?sof_ver_rel}%{?dist}
 # See later in the spec for a breakdown of licensing
 License:        BSD
 URL:            https://github.com/thesofproject/sof-bin
-Source:         https://github.com/thesofproject/sof-bin/archive/v%{sof_version}/sof-bin-%{sof_version}.tar.gz
+Source:         https://github.com/thesofproject/sof-bin/releases/download/%{sof_ver_pkg}/sof-bin-%{sof_ver_pkg}.tar.gz
 BuildRequires:  alsa-topology >= %{tplg_version}
 BuildRequires:  alsa-topology-utils >= %{tplg_version}
 Conflicts:      alsa-firmware <= 1.2.1-6
@@ -34,15 +37,15 @@ License:        BSD
 This package contains the debug files for the Sound Open Firmware project.

 %prep
-%autosetup -n sof-bin-%{sof_version}
+%autosetup -n sof-bin-%{sof_ver_pkg}

 mkdir -p firmware/intel/sof

 # we have the version in the package name
-mv sof-v%{sof_version}/* firmware/intel/sof
+mv sof-%{sof_ver_pkg}/* firmware/intel/sof

 # move topology files
-mv sof-tplg-v%{sof_version} firmware/intel/sof-tplg
+mv sof-tplg-%{sof_ver_pkg} firmware/intel/sof-tplg

 # remove NXP firmware files
 rm LICENCE.NXP

[mwasko]

@mwasko who from your team can work with @marc-hb on aligning the signing process with the release process to make it less manual and error prone. I would imagine that some helper scripts could be used to take the signed output from the signing system and upload/copy to the release destination. This can be done for v1.9 so we have 3 months.

@slawblauciak and me are coordinating sof releases. I would recommend to prepare/extend script to validate release content and report if anything is missing. We can also work on some helper scripts to upload signed binaries, but there will be still some manual work dependency.

[marc-hb]

1.8 released at https://github.com/thesofproject/sof-bin/releases

@perexg is everything good or good enough now?

[lgirdwood]

Thanks @marc-hb - Adding release notes shortly - give me a few hours.

[perexg]

It seems fine. Fedora 34 package update: https://bodhi.fedoraproject.org/updates/FEDORA-2021-c838e6f856

I'm closing this now. It seems that all was settled.