timja commented 2 years ago

There is a PR here where inspiration can be taken from: https://github.com/jenkinsci/packaging/pull/93

basil commented 2 years ago

JENKINS-41218 FTR

serianox commented 2 years ago

JENKINS-65809 too, Jenkins restart on Debian no longer works with the current SysVInit script

fe80 commented 2 years ago

Hello,

I think you can use the voxpopuli puppet code for create the systemd unit file.

They truncate your init script https://github.com/voxpupuli/puppet-jenkins/blob/master/templates/jenkins-run.erb (without PARAMS+=("--daemon"), deprecated since 2.313)

After you just need to create the systemd file like that:

systemctl cat jenkins.service 
# /etc/systemd/system/jenkins.service
[Unit]
Description=Jenkins master service
Requires=network.target
After=network.target

[Service]
Type=simple
User=jenkins
Group=jenkins
EnvironmentFile=/etc/sysconfig/jenkins
ExecStart=/usr/lib/jenkins/jenkins-run
Restart=always

[Install]
WantedBy=multi-user.target

EnvironmentFile=/etc/sysconfig/jenkins change between RedHat and Debian family, but everything else is the same.

In https://github.com/jenkinsci/packaging/pull/93/files, they define some option like SyslogIdentifier or UMask. But everyone, and I think it's better, can change the value with systemd override hierarchy https://www.freedesktop.org/software/systemd/man/systemd.unit.html

systemctl cat jenkins.service 
# /etc/systemd/system/jenkins.service
[Unit]
Description=Jenkins master service
Requires=network.target
After=network.target

[Service]
Type=simple
User=jenkins
Group=jenkins
EnvironmentFile=/etc/sysconfig/jenkins
ExecStart=/usr/lib/jenkins/jenkins-run
Restart=always

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/jenkins.service.d/override.conf
[Service]
SyslogIdentifier=jenkins
UMask=0002

timja commented 2 years ago

Feel like submitting a PR? (doesn't have to do debian vs redhat in the same PR either, can be gradual support)

ekohl commented 2 years ago

Would that be applied in https://github.com/jenkinsci/packaging/tree/master/deb/build/debian & https://github.com/jenkinsci/packaging/tree/master/rpm/build? What are the OS constraints? Is there a matrix of supported envs?

timja commented 2 years ago

Would that be applied in master/deb/build/debian & master/rpm/build? What are the OS constraints? Is there a matrix of supported envs?

Yes

I think @MarkEWaite has had an action item to propose an official approved OS matrix but I don't think it currently exists.

I would follow the vendor's standard support life cycle for LTS or whatever is appropriate for OS, with aiming to support recent releases too.

So would require:

i.e. Ubuntu 18,20 Debian Bullseye

Centos/Rhel 7 Rhel 8 / whatever non rhel derivative

MarkEWaite commented 2 years ago

I think @MarkEWaite has had an action item to propose an official approved OS matrix but I don't think it currently exists.

I think that's a good idea as a way to state the operating systems that we actively test. I hadn't captured that action item, but have put it into the Platform SIG meeting notes for future action.

We have a "Windows support policy" page. A "Linux support policy" page would be a good addition to describe the approach used for Linux variants.

I would follow the vendor's standard support life cycle for LTS or whatever is appropriate for OS, with aiming to support recent releases too.

So would require:

i.e. Ubuntu 18,20 Debian Bullseye

I am confident that we have active Debian package users running on all actively maintained versions of Debian (oldstable - "buster" - Debian 10, stable - "bullseye" - Debian 11, testing - "bookworm", and unstable) and on some unmaintained versions ("stretch" - Debian 9).

I think that we need to assure that the systemd implementation works on oldstable, stable, and testing.

Centos/Rhel 7 Rhel 8 / whatever non rhel derivative

I am confident that we have active rpm package users on CentOS 7, CentOS 8, Red Hat Enterprise Linux 7, Red Hat Enterprise Linux 8, Oracle Linux 7, Oracle Linux 8, and Amazon Linux 2.

I think that we need to assure that the systemd implementation works on those variants. The Amazon Linux 2 variant caused some interesting issue reports when we added the requirement for a package from the EPEL repository.

ekohl commented 2 years ago

Thanks for that. Something to keep in mind while taking this on is that the puppet-jenkins module has some code to stop the sysvinit service before installing the systemd service. I'm not sure if it will be needed in packaging, but upgrades should be seamless for users.

ekohl commented 2 years ago

I took a stab at this and https://github.com/jenkinsci/packaging/pull/224 has some of the initial RPM work. I haven't had a chance to try it yet so it's still a draft.

basil commented 2 years ago

Thanks @MarkEWaite for posting the solution requirements above. It occurs to me that in order to avoid regression, a prerequisite for working on this type of thing is to get some type of automated testing in place for this repository. For local usage, this could be done with something like Vagrant. For CI usage, this would require cooperation from the Jenkins infrastructure team to allow jobs to provision one-shot VM-based agents of various OSes in order to run tests. To orchestrate the whole thing, we can look into using tools like Kitchen. What do you think?

MarkEWaite commented 2 years ago

I implemented a partial automated test of the Debian installer (mis)using Docker images as part of https://github.com/MarkEWaite/packaging/commits/INFRA-910-core-release-automation . It helped meet my needs for test automation while we were developing INFRA-910 last year.

The same technique does not work for Red Hat images / CentOS images because they don't seem to include the components of the init system in the Docker images I was using.

We have an automated test of the Debian and Red Hat installers that runs separately as https://ci.jenkins.io/job/Infra/job/acceptance-tests/ . It is not integrated into this repository, but it might be some help.

I didn't find a way to automate the testing of upgrades from one version to the next. I suspect it is possible, but I'm not sure that it will be enough of a help to justify the energy to create it.

basil commented 2 years ago

One of the reasons I mentioned Kitchen is that they have drivers for either using real VMs in e.g. EC2 or using "fat" Docker containers (cf. kitchen-dokken) which are a bit less realistic but perhaps a bit more practical. It's been years since I've used Kitchen, so I have no idea if it's the current best-of-breed tool for this sort of thing or not, but it seems to me that if a well-maintained framework exists that it would be preferable to rolling our own.

MarkEWaite commented 2 years ago

if a well-maintained framework exists that it would be preferable to rolling our own.

Agreed.

basil commented 2 years ago

I've split out the testing discussion into #225. If we were using Jira as our issue tracker, I would have linked the two issues together with #216 depending on #225.

basil commented 2 years ago

Distribution-specific systemd packaging guidelines:

The Debian and Fedora guides provide a useful pattern for maintaining backward compatibility with /etc/{default,sysconfig} files: the use of EnvironmentFile=-. Since our users may have customized these files, we can likely use this pattern to preserve compatibility with existing customizations.

basil commented 2 years ago

Apart from migration of legacy configuration to systemd, which is still a work in progress, #266 is otherwise complete from my perspective, with a working systemd implementation for all supported target platforms.

basil commented 2 years ago

266 is ready from my perspective. I'll keep doing additional testing, but the PR is feature complete, passes CI consistently, and passes manual upgrade/migration testing for me.

basil commented 2 years ago

As Bryan Cantrill once so eloquently wrote:

Have you reviewed and tested your change every way that you know how? You should not even contemplate pushing until your answer to this is an unequivocal YES.

I am getting to that point where I have tested #266 every way that I know how. In my opinion this change is ready for merge.

darxriggs commented 2 years ago

Now that systemd is supported, its features could be used to further improve security. For example see:

timja commented 2 years ago

Feel free to raise issues with any suggestions https://github.com/jenkinsci/packaging/issues :)

jenkinsci / packaging

Systemd support #216

266 is ready from my perspective. I'll keep doing additional testing, but the PR is feature complete, passes CI consistently, and passes manual upgrade/migration testing for me.