Automatically update yum repo

[dewyatt]

• Automatically update riboseinc/yum
• Add packager tag to packages
• Add SRPMs

One thing I don't do here is update the commits in the yum repo, as my initial impression is that it wouldn't be needed with this workflow.

[ronaldtse]

Thanks @dewyatt ! What exactly did you mean by:

One thing I don't do here is update the commits in the yum repo, as my initial impression is that it wouldn't be needed with this workflow.

?

[dewyatt]

@ronaldtse I'm referring to this: https://github.com/riboseinc/packaging/blob/f08025a96b58fc996dd299b3930500e7e79b807d/scripts/_common.sh#L106

It looks like it stores the rpm-spec-* commit hashes in riboseinc/yum/commits to determine whether a package update is needed.

The workflow here is different, so my initial thought is that it wouldn't make sense to do now.

[ronaldtse]

@dewyatt the intention of that commit compare system was to not re-build/re-sign packages if they were already built, since the process of signing is time-sensitive.

Is this not necessary anymore?

(ping @ribose-jeffreylau )

[dewyatt]

@dewyatt the intention of that commit compare system was to not re-build/re-sign packages if they were already built, since the process of signing is time-sensitive.

Is this not necessary anymore?

Ok that's pretty much what I was thinking, I just don't have much insight into how the jenkins setup worked, so definitely appreciate a second set of eyes.

So the way this PR is currently, the only conditions in which it would waste time re-building and re-signing a package (IIRC), would be: say you made a commit to rpm-spec-parallel that did not alter the Version or Release tag of the spec (maybe a README change w/out skip ci), it would waste time doing a re-build and re-sign (it would not update the github release nor the yum repo).

So I guess the amount of time wasted would depend on how frequently you expect the rpm-spec-* repos to have non-version-changing commits.

Should I try to improve this?

[ronaldtse]

So I guess the amount of time wasted would depend on how frequently you expect the rpm-spec-* repos to have non-version-changing commits.

Our concern is not much on the time-wasted, but more on the sizes of git commits for publishing one package. If publishing one package triggers the re-signing of other packages, the size of the git commits will be much larger. And the resulting repo will always show stray updates.

Could you help update the current scheme to take advantage of the original system? @ribose-jeffreylau wrote that and he would be happy to help (on Monday...)

[dewyatt]

Our concern is not much on the time-wasted, but more on the sizes of git commits for publishing one package.

Ok that makes sense.

If publishing one package triggers the re-signing of other packages, the size of the git commits will be much larger. And the resulting repo will always show stray updates.

I'm not sure I understand fully, but maybe things will become clearer on Monday when @ribose-jeffreylau has time.

Just to clarify though, an example of how this system would work is:

1. Someone makes a commit to rpm-spec-parallel and bumps the Version tag in the spec to something newer (20190222). Or they fix some issue in the packaging and bump the Release tag, etc.
2. GitLab CI runs a pipeline for rpm-spec-parallel, which includes these jobs:
   • build (build the parallel SRPM+RPM)
   • sign (sign the parallel SRPM+RPM)
   • github-release (publish parallel github release)
   • update-repo (createrepo, sign repodata, etc).

An example of the yum repo update commit can be seen here in my fork: https://github.com/dewyatt/yum/commit/a0104adc2808fdc523026774bb2c6704f457c4d6

[dewyatt]

@ribose-jeffreylau ping

[ribose-jeffreylau]

@dewyatt Hi, sorry for taking so long.

I had a look at the code at https://github.com/riboseinc/packaging/blob/f08025a96b58fc996dd299b3930500e7e79b807d/scripts/_common.sh#L109 and traced it back to https://github.com/riboseinc/packaging/blob/f08025a96b58fc996dd299b3930500e7e79b807d/scripts/_common.sh#L22 . I believe the intention of check_if_newer_than_published() was to skip the following actions iff the Yum repo already has a SHA record (e.g., see here) that is newer than or equal to [*] the one from which we are trying to build the RPM:

1. update_yum_srpm . signs the built SRPMs and moves them to the SRPMS/ folder of the Yum repo;
2. update_yum_rpm . signs the built RPMs and moves them to the RPMS/ folder of the Yum repo;
3. commit_repo "${package_name}" "${package_spec_commit}" . updates the SHA record in the Yum repo for the current package, and . commits and pushes the changes to the Yum repo.

With this system, there is no change necessary to any rpmspec fields (i.e. no need to change the Version / Release tag). The so-called "versions" are simply the SHA of the respective RPM spec repos, which are recorded under the respective files commits/${package_name} of the Yum repo.

Hope it helps :p

[*] The script determines "newness" by checking whether a commit is an ancestor of another.

[dewyatt]

I think I'm still not clear on some things: 1) What are the shortcomings in this PR? Are changes needed, or is a different approach required? I've tested this system and it works well from my perspective but I'm not sure we're on the same page. 2) > With this system, there is no change necessary to any rpmspec fields (i.e. no need to change the Version / Release tag). The so-called "versions" are simply the SHA of the respective RPM spec repos, which are recorded under the respective files commits/${package_name} of the Yum repo.

This part didn't make sense to me because surely you still have to edit the spec to update the `Version` or `Release` tag when a new upstream release comes out (like libgsf 1.14.45). (

[ronaldtse]

@dewyatt for part 2, the point of storing the commit (of the RPM spec definition) is so that the software packages aren’t built and signed again. The “software” itself it not changed across iterations, this is purely used to allow us to iterate the package definitions (or replace a package) of the packaged software of the same version.

[dewyatt]

@ronaldtse I went ahead and added support for the previous behavior with commits/. Can you take another look?

[ronaldtse]

Thanks @dewyatt ! @ribose-jeffreylau can you help verify?