search

theforeman / foreman-infra

Puppet modules and scripts to manage Foreman project infrastructure
https://theforeman.github.io/foreman-infra
Apache License 2.0
23 stars 51 forks source link

Decomission Koji setup #2006

Open ekohl opened 6 months ago

ekohl commented 6 months ago

With the move to COPR (#1795) the Koji setup is no longer needed. Today we still build Foreman 3.7 & 3.8, but after those are EOL the infrastructure can be decommissioned.

ehelms commented 6 months ago

If I do the math correctly, that means with the 3.11 release we can take on this work.

evgeni commented 6 months ago

I think that's correct.

Technically we also have to consider Katello 4.10 and whatever Pulpcore was before 3.39, but luckily all those align together.

ekohl commented 3 months ago

I think these get rid of everything that requires Koji:

ekohl commented 2 months ago

When we've removed the VMs, we should make sure koji.katello.org is also removed.

ekohl commented 2 months ago

Expanding on my previous comment, I think it's a good time to consider what the full plan is. First of all, can we start turning off koji itself?

http://koji.katello.org/koji shows the last build was a month and a half ago, so that's good. Then the question is whether we need anything from http://koji.katello.org/releases/.

Currently we have:

/dev/xvda1      8.0G  3.5G  4.6G  44% /
/dev/nvme0n1p2  505G  274G  206G  58% /mnt/tmp
/dev/xvdx1     1008G  939G   19G  99% /mnt/koji

So that's not easy to just dump somewhere. Another thing to note is that /mnt/tmp is empheral and will be deleted. Quoting /mnt/tmp/README:

This volume is ephemeral, stopping the VM immediately deletes it.

It only contains temporary and work files, but we keep backups at
/mnt/koji/backups/ephemeral (which is AWS EBS volume). The backup
script is at:

/etc/cron.weekly/koji-backup

To restore do this:

duplicity restore file:///mnt/koji/backups/ephemeral /mnt/tmp --force --no-encryption

The backup skips filenames with RPM extension to keep the backup
clean and small - the most important are directories and permissions.

Perhaps we can copy over the backups:

# du -sh /mnt/koji/backups/
209G    /mnt/koji/backups/

But it should be noted that those have file timestamps back to 2017 so it's probably sufficient to take the latest ones.

As for storage to place it: we have 650 GB of unallocated storage at OSUOSL.

evgeni commented 2 months ago

Expanding on my previous comment, I think it's a good time to consider what the full plan is. First of all, can we start turning off koji itself?

I think so, yes.

http://koji.katello.org/koji shows the last build was a month and a half ago, so that's good.

That correlates with https://github.com/theforeman/jenkins-jobs/pull/436 that dropped the support of being able to build there ;)

Then the question is whether we need anything from http://koji.katello.org/releases/.

I'd argue we have everything we need on yum.theforeman.org?

Currently we have:

/dev/xvda1      8.0G  3.5G  4.6G  44% /
/dev/nvme0n1p2  505G  274G  206G  58% /mnt/tmp
/dev/xvdx1     1008G  939G   19G  99% /mnt/koji

So that's not easy to just dump somewhere. Another thing to note is that /mnt/tmp is empheral and will be deleted. Quoting /mnt/tmp/README:

This volume is ephemeral, stopping the VM immediately deletes it.

It only contains temporary and work files, but we keep backups at
/mnt/koji/backups/ephemeral (which is AWS EBS volume). The backup
script is at:

/etc/cron.weekly/koji-backup

To restore do this:

duplicity restore file:///mnt/koji/backups/ephemeral /mnt/tmp --force --no-encryption

The backup skips filenames with RPM extension to keep the backup
clean and small - the most important are directories and permissions.

Perhaps we can copy over the backups:

# du -sh /mnt/koji/backups/
209G  /mnt/koji/backups/

But it should be noted that those have file timestamps back to 2017 so it's probably sufficient to take the latest ones.

As for storage to place it: we have 650 GB of unallocated storage at OSUOSL.

If we only take things from 2024, it's 13G. But given it does not include the RPMs, I don't see much value in the backups outside the existing install?

ekohl commented 2 months ago

But given it does not include the RPMs, I don't see much value in the backups outside the existing install?

I was leaning the same way. So a concrete plan:

evgeni commented 4 weeks ago

@pcreech has shut down the machine last Friday.

evgeni commented 1 week ago

@pcreech you can go ahead and delete it

TODO: