Build automation (RPM+ISO)

[axk4545]

Currently RPMs are updated, built and added to the repo manually. It would be nice to script this funtionality so that we can have changes pushed to github trigger rebuilding of RPMs and updating of the repo. The GH sources affected are those for the scripts and possibly the branding.

For help ask about Fedora SOP in #fedora-admin and #fedora-devel on freenode

[axk4545]

Maybe rework GH repo to funtion similar to the fedora lookaside cache

[ct-martin]

I can help with this, hit me up another time and we can brainstorm

[ct-martin]

From discussion:

1. Someone commits to GitHub
2. Pull on the build box (using git pull)
3. Make srpm + rpm
4. Copy to /var/builds-to-sign or somewhere that has "chmod g+w" and "chown build-signers:build-signers" w/ people allowed to sign in that group
5. (long-term) send a message via email and/or IRC to notify things need signing
6. Things get signed manually
7. Run after-signing script which does the following:
8. Copy items to mirrors folders
9. (long-term) Git commit to the GitHub repo w/ a RITlug-BuildBox account
10. (long-term) Post a message via email and/or IRC announcing new items (tigeros-announce?)

[axk4545]

@ct-martin also assigning you though I can't since you aren't on the repo yet. poke me if you want to be added and I will discuss with team and eboard.

[axk4545]

Here are a few option: Self hosted COPR Bash scripts and git hooks mock-scm config options to make a tarball

[jwflory]

Self hosted COPR

For what it's worth, I'm 👎 to a self-hosted COPR because it will be difficult for someone to maintain later on. I'd be concerned about what would happen in four years if we hosted our own COPR.

[ct-martin]

@jflory7 tbh, git+jenkins/ci+bash (aka, what we have now) seems like the best option

[axk4545]

True. It is likely though that the build system will be on our infra unless we can find a good hosted way to do RPM builds. Also needs to be able to function as a dnf repo or put the result in a valid dnf repo. Also signing.

Aidan Kahrs abkahrs@gmail.com

On Oct 16, 2017 4:57 PM, "Justin W. Flory" notifications@github.com wrote:

Self hosted COPR

For what it's worth, I'm 👎 to a self-hosted COPR because it will be difficult for someone to maintain later on. I'd be concerned about what would happen in four years if we hosted our own COPR.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/RITlug/TigerOS/issues/29#issuecomment-337040206, or mute the thread https://github.com/notifications/unsubscribe-auth/ADUyLFHaZT6kJpzwLQzJaerBDPmtXFtZks5ss8MygaJpZM4NJJ9h .

[jwflory]

It is likely though that the build system will be on our infra unless we can find a good hosted way to do RPM builds.

Well, if we're hosting it on our own, I'd figure it would have to be hosted on our infrastructure, right? 😉

If COPR is something we want to use, I have a strong preference for using Fedora's COPR and creating a group for the TigerOS team to maintain any packages there. Hosting on our own introduces a significant amount of work, and it would require all of the team to contribute documentation for how to manage that. I encourage all of you to think too about how this will be maintained in five years from now.

[axk4545]

@jflory7 Licensing might be an issue. Also COPR may want to have one repo per package.

[ct-martin]

Status update:

• We have an Ansible playbook that installs all the needed programs and does part of the configuration.
• We (@axk4545) are rethinking CI, will continue once that is decided.

[axk4545]

@ct-martin Travis is a no go. This is gonna be interesting with COPR. Biggest concern is licensing. I would go with automating our self hosted stuff (maybe ala this substituting fedpkg for rpmbuild ) and the process there but be able to migrate to COPR with tito ala this

@jflory7 I would like to get your input on how to make either of those more sustainable

[axk4545]

for help with using tito as describe above in hosted COPR and getting COPR to work with our subdir based approach talk to clime in #fedora-buildsys on freenode

[axk4545]

ideas:

• Lightweight build system section here https://www.hogarthuk.com/?q=node/11
• https://wiki.jenkins-ci.org/display/JENKINS/RPM+Sign+Plugin
• https://github.com/jhrcz/jenkins-rpm-builder
• https://schneide.wordpress.com/2013/03/18/1-click-deployment-of-rpms-with-jenkins/ @ct-martin

[axk4545]

ok. so we should probably use tito in some way to get around the subdir part. we can run tito with jenkins to trigger. signing can likely use the jenkins plugin and deployment is copying in some way from the output dir to the mirror/repo dir and fixing perms. tito will handle making the tarballs and bascially going from flat source files to RPMs and SRPMs.

[axk4545]

Test with standard web server directory since that should have proper selinux context

[axk4545]

New idea for RPMs, GitLab runner on buildbox that talks to the GitLab.com instance of the project and does per branch rebuilds. we can use the only option to restrict the branches and then jsut script the generation of tarballs and the building.

Process might be like the following:

1. git clone the branch or git checkout and pull
2. enter proper directory and create tarball with tar -cf
3. run fedpkgto rebuild the package and email the admin or a team member or emit a notification in slack that the packages are done
4. user goes in and signs the RPMs
5. RPMs are then copied to the download host and permissions and repo fixed and updated. (maybe script this part)

thoughts @jflory7 @RITlug/tigeros-team ?

References

• https://gitlab.com/gitlab-org/gitlab-runner
• https://docs.gitlab.com/ce/ci/yaml/README.html#only-and-except-simplified
• https://docs.gitlab.com/runner/install/linux-repository.html

[jwflory]

We briefly discussed in the TigerOS Slack channel.

I am +1 to creating an individual repo for each unique package that we create.

This follows best practice in the same way that Fedora does packaging with dist-git. Packages are easier to find and maintain individually – putting each unique package into a new branch is an uncommon practice and I think is liable to leave things forgotten over time. Following the dist-git model also helps us follow the upstream policy of packaging for potential inclusion of our packages in Fedora or other repositories, like RPMFusion.

Additionally, the tools available in Fedora that we are considering for our own use expect individual packages to be in their own repo – using unique branches for each package may create more future headaches since branches aren't well-designed for this type of workflow.

[axk4545]

So I am talking with @ritjoe a little and he was favoring the idea of one package per repo. We would still need to workout a way to hook our builder up to github or gitlab or w/e so that it can trigger tasks on a push. Maybe we also take this as an opportunity to establish a RITlug presence on gitlab.com

[axk4545]

Current Plan

Ok. so I know this is getting stupid. anyway I found http://frostyx.cz/posts/copr-stack-dockerized and https://pagure.io/copr/copr/blob/master/f/rpmbuild. I talked to the maintainer and he is willing to add a feature to copr-rpmbuild to allow it to talk directly to the builder container and run a build. so we can use that to trigger a build on a push and then we just make sure that each package has it's own repo... @ct-martin @jflory7 input?

Upstream tracking issue for allowing copr-rpmbuild to talk directly to builder: https://pagure.io/copr/copr/issue/102

[jwflory]

@axk4545 I'm not informed enough to add much here. I'm still opposed to having us run our own Copr instance, but I trust you all to make the right decisions for the project.

[axk4545]

Playbook to setup a COPR builder can be found at: https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/copr/backend/files/provision/provision_builder_tasks.yml

[axk4545]

THIS IS NO LONGER ACCURATE. THE CHANGES HAVE BEEN ADDED TO THE MASTER BRANCH AND ARE NOW IN THE OFFICIAL FEDORA REPOS Here are the links to the COPR for copr-rpm-build-cmdline and a git repo with the source:

COPR: https://copr.fedorainfracloud.org/coprs/clime/copr-rpmbuild-cmdline/

git: https://github.com/axk4545/copr-rpmbuild-cmdline

[axk4545]

Working on automating deployment of builder with Ansible. May revert to Docker if needed since that is premade. Attaching relevant IRC logs COPR_plan.txt

[axk4545]

Command example on builder: copr-rpmbuild scm --clone-url https://pagure.io/rpkg-client --chroot fedora-27-x86_64 the output RPM will be in /var/lib/copr-rpmbuild/results

[ct-martin]

Having issues with Jenkins, looking into GitLab Runner

[axk4545]

We now have IP addresses and can provision the webhost

[axk4545]

It will trigger based on push to master

[Tjzabel]

I think this is going to take a lot longer than the Beta release to complete. Thoughts on moving this to the final release?

[ct-martin]

@Tjzabel I'm not going to block the beta on it, but I would like to keep it on the beta milestone due to its importance

[Tjzabel]

@ct-martin sounds good.

Also, it is probably a good idea to split this issue into smaller issues so it is much easier to track. It would be a good idea to have an issue for updating the builder + mirror Ansible playbooks, an issue for the GitLab ISO CI, and an issue for the RPM building CI

[ct-martin]

@Tjzabel I actually have a draft issue for updating the playbooks right now with some other stuff as well. I have a couple thoughts on how to split up the issue, let's talk about them in the meeting in a couple minutes

[Tjzabel]

Next steps are to do the write-up on how TigerOS CI is going to work. Current draft is located on my GitLab repo.

The goal is to implement CI on a single RPM package to make sure the process goes smoothly. Then, a TigerOS mirror will be set up on the GitLab public site, where we will be hosting CI on the builder, and possibly the mirror.

The draft documentation will be added into the wiki at some point.

[Tjzabel]

Furthermore, I have an RPM CI script in the works to do the RPM building.

However, this is still a rough script, and needs to be ironed out in order to make the RPM building process fully automated. At the moment, the release version and package name need to be added as parameters to the script, which are not yet found automatically.

In order to build an RPM package, the specific release is needed. We need to figure out a way to grep for the release version without manually putting it in.

[Tjzabel]

In addition, RPM CI and ISO CI are very different from one another, and so I believe these two CI issues should be separated to create a more easy-to-follow workflow.

[ct-martin]

@Tjzabel could you clarify the following? I have no idea what your mean, and my best guess doesn't get close to what I had planned.

Then, a TigerOS mirror will be set up on the GitLab public site, where we will be hosting CI on the builder, and possibly the mirror.

Also, I have access to the GitHub credentials now so I can proceed when I have time (probably next weekend)

[Tjzabel]

@ct-martin If it helps, I also have no idea what I was trying to write there :smile:

I am going to update that comment now with what I actually meant to say.

[Tjzabel]

@ct-martin actually, I do know what I was saying there.

Essentially, there will be a GitLab instance that is mirroring what the GitHub repo has, so when there is a change to the repo, the GitLab CI will pull the change and do the automation. Correct me if this line of thinking isn't what you had in mind.

[ct-martin]

@Tjzabel ok, yes, that is what I had in mind. I was confused by "mirror" having multiple meanings. What do you think about using terminology along the lines of "TigerOS mirror" or "the mirror" for mirrors.ritlug.com (to maintain backwards-compatibility in terminology) and "GitLab repo mirror" for the GitLab CI/CD for external repositories? Wording doesn't have to be exact.

Thanks for the clarification

[Tjzabel]

@ct-martin I got the same confusion when I re-read my own comment.

I definitely think it is a good idea to create uniform terminology when referring to the different parts. Even GitLab mirror is enough imo, and TigerOS mirror for the packages mirror.

[ct-martin]

@Tjzabel I think it would be a good idea to use "GitLab repo mirror" (with "repo") since GitLab Runner will be automating things on the mirror soon. I'll file an issue to start a glossary and we can discuss on Wednesday

[ct-martin]

As per meeting we have decided to do Continuous Delivery to a new development mirror based on the devel branches of the respective repos. This means that for devel builds instead of getting put into a queueing folder, they will be signed and put straight on a development mirror separate from the current one. This will also require standardizing branch protection, which I will figure out and document.

[ct-martin]

I have installed GitLab Runner on the builder, will set up the pipelines another time.

[Tjzabel]

This is no longer in scope for a beta release.

[Tjzabel]

Update

Currently testing out the ISO-building Dockerfile in order to build the F29 ISO. We will be one step closer to adding build automation if this succeeds :+1: