openSUSE / openSUSE-release-tools

Tools to aid in staging and release work for openSUSE/SUSE
GNU General Public License v2.0
62 stars 96 forks source link

Create standardized service files for use by IBS and OBS #1006

Closed jberry-suse closed 5 years ago

jberry-suse commented 7 years ago

Given there are two variables for most services a single template variable for a service file is likely not enough. As such pointing to an environment variable file/config file might make more sense. It would be nice if the user could be a parameter based on that file, but guessing that it would not be possible to load environment file in that manor. Alternatively, a wrapper script could be provided to start the desired service using the config file as the appropriate user.

As a different approach the users themselves could also be standardized and created by the package so then they simply need to have their credentials setup for the appropriate OBS instance and user. I would imagine the bots/scripts that run as different users on OBS should also be run separately on IBS so this may simplify things.

jberry-suse commented 7 years ago

Performed an inventory of the services on packagelists machine.

services

A slight lack of consistency in naming and the template argument being partial or full project name or built into service file. I would suggest the full project name be used for all templated services to allow for easy usage by IBS for SUSE: projects as well.

As for naming and additional configuration, it would seem to make sense to prefix services with an abbreviation of openSUSE-release-tools so osrt- as the prefix, otherwise the full name. A common config location like /etc/openSUSE-release-tools with sub-directories for services requiring them and the project name can be included for templated service files requiring different config per project. This will merge all the random config paths into one location with a standard path setup.

I may also rename some of the services that are a bit misleading or obfuscated to make their role more clear. As an example, I plan to do the official repo_checker deployment via the package from OBS and will submit the PRs for the service files before then. Something along the lines of openSUSE-release-tools-repo-checker@openSUSE:Factory or prefixed if preferred as osrt-repo-checker@openSUSE:Factory.

users

$ grep -hoPR 'User=\K.*' systemd/packagelists/ | sort | uniq

The subtle name differences between system and OBS are rather annoying and obfuscating in addition to the lack of consistency on multiple levels.

permissions

factory-staging

Not sure why factory-auto is in that list other than perhaps to have access to dashboard, but should likely be demoted. staging-bot being the only tool that should have staging access.

openSUSE:Factory:Staging

<group groupid="factory-staging" role="maintainer"/>

openSUSE:Factory:Staging/dashboard

<person userid="ancorgs" role="maintainer"/>
<person userid="aplanas" role="maintainer"/>
<person userid="factory-repo-checker" role="maintainer"/>
<person userid="leaper" role="maintainer"/>
<person userid="totest-manager" role="maintainer"/>

The first two users seem like they should be unnecessary at this point. The last three are part of tooling with factory-staging being inherited from the project.

It would seem to make sense to create a new group for tools that need to write to the dashboard, but do not need staging access. The group can then easily be applied to new Leap projects and have an understood meaning. This same workflow can then be used for IBS as well.

Perhaps name the group release-tools since none of the other groups are prefixed with openSUSE or similar. The new repo-checker needs to have dashboard access, but currently does not, but we should add via the new group rather than directly.

jberry-suse commented 7 years ago

One unfortunate note, is that since the current packagelists setup installs itself in global osc plugins dir this switch will not work properly until all services can be redeployed.

jberry-suse commented 7 years ago

Since we are looking to enable a variety of new services for Leap 15.0 and SLE 15 it would be preferable to do so with this rather than manually maintaining two different machines.

jberry-suse commented 7 years ago

Mapping of services to machine users:

$ grep -RoP 'User=\K.*' .
./maintbot.service:maintbot
./osc-check_dups.service:repochecker
./gnome-review-bot.service:gnome-review-bot
./staging-bot-regular@.service:_opensuse.org-staging-bot
./ttm-leap.service:_opensuse.org-totest-manager
./pkglist@.service:repochecker
./suppkg_rebuild@.service:repochecker
./repochecker.service:repochecker
./leap-423-announcer.service:_opensuse.org-factoryauto
./factoryauto.service:_opensuse.org-factoryauto
./staging-bot-reminder.service:_opensuse.org-staging-bot
./update-crawler.service:leaper
./build-fail-reminder@.service:leaper
./factory-auto-submit.service:factorymaintainer
./create-test-dvds.service:repochecker
./ttm-tumbleweed.service:_opensuse.org-totest-manager
./repo-checker@.service:_opensuse.org-repo-checker
./staging-bot-daily@.service:_opensuse.org-staging-bot
./openqa-comments@.service:_opensuse.org-factory-bot
./ttm-arm.service:_opensuse.org-totest-manager
./factory-announcer.service:_opensuse.org-factoryauto
./source-checker.service:factory-source
./trigger-rebuilds.service:repochecker
./manager_42.service:leaper
./leaperbot.service:leaper
./factory-dvl-prj-list.service:leaper
./ttm-s390.service:_opensuse.org-totest-manager
grep -hRoP 'User=\K.*' . | sort | uniq
factorymaintainer
factory-source
gnome-review-bot
leaper
maintbot
_opensuse.org-factoryauto
_opensuse.org-factory-bot
_opensuse.org-repo-checker
_opensuse.org-staging-bot
_opensuse.org-totest-manager
repochecker
jberry-suse commented 7 years ago

Will need to change the locations that are invoking ttm-* services for new name.

jberry-suse commented 7 years ago

factory-auto-submit.service uses obs-autosubmit which should likely be handled there or left not packaged.

jberry-suse commented 7 years ago

After the PRs open as of now the remaining services to be handled are:

./pkglist@.service:repochecker
./leap-423-announcer.service:_opensuse.org-factoryauto
./build-fail-reminder@.service:leaper
./create-test-dvds.service:repochecker
./openqa-comments@.service:_opensuse.org-factory-bot
./factory-announcer.service:_opensuse.org-factoryauto
./source-checker.service:factory-source
./trigger-rebuilds.service:repochecker

The announcer already has service files, but currently uses two different forms and the service file does not handle a project argument so it needs to be reworked. The biggest decision that needs to be made is whether to move services requiring maintainer permission to staging-bot as it seems to make sense more-so than tacked on to a review bot user.

The services requiring maintainer access should be annotated for future reference.

jberry-suse commented 7 years ago

@DimStar77 @lnussel @nilxam @coolo any opposition to merging ./openqa-comments@.service:_opensuse.org-factory-bot (factory-comment-bot) under staging-bot umbrella as I am starting to think all "staging" related services should just run as that user since it will need all the required permissions. In the end also inheriting the non repo-checker related services that still run under factory-repo-checker and then retiring the user. The code relies on the marker in the comment not the user that posted so no code changes should be needed.

Currently, staging-bot is making the reminder comments and other such bits. My thinking is review bots should remain under their own individual users with reviews assigned to them, while generalized staging services run under staging-bot.

Seems to make sense to run pkglist generation under a different user since it's rather different, but I could see merging it as well since currently it runs as factory-repo-checker.

The announcer should remain a separate user and I am not sure it even needs obs credentials, but it currently runs as factory-auto which being a review bot does not make sense short of convenience.

jberry-suse commented 6 years ago

After the last two pull requests the list that need to be generalized and included in packaging are as follows.

Several run as old repo checker user which will require decision on what user to place them under and the appropriate permission changes. The source-checker will require a new user and auto-submit handling of the wrapper scripts.

jberry-suse commented 6 years ago

Presumably the other openqa service lives somewhere else and should be have a service file, but I'll leave that to those familiar with it's deployment.

coolo commented 5 years ago

As the deployment moved to gocd config, this ticket is obsolete. The packaging of systemd services is still in place, but majorly unused. But that's a different ticket for a different time

jberry-suse commented 5 years ago

The large amount of work to standardize them to the point they could be used for both is still in use, but seems fair to close.