Purge does not work - Githubissues

tomposmiko commented 3 years ago

hi,

I probably do something wrong with my environment, but do not find the reason.

g10k.yaml:

---
#postrun:
#  - '/postrun.sh'
cachedir: '/root/.g10k'
deploy:
  purge_levels: ['deployment', 'puppetfile', 'environment']
sources:
  puppet:
    remote: 'git@gitlab.com:<URL>'
    basedir: '/var/lib/docker/volumes/compose-services_puppetserver-code/_data/environments/'
    invalid_branches: 'correct_and_warn'
    prefix: p
git:
forge:

command:

./g10k -config g10k.yaml -verbose

My understanding is that this should purge the basedir by removing stale directories.

version:

g10k  v0.8.15  Build time: 0.8.15 UTC

Any suggestion, what's wrong with my setup? Thanks in advance,

walkamongus commented 3 years ago

this seems likely related to #155. Do you have any branches that don't have their name corrected and if so are they purged?

xorpaul commented 3 years ago

@tomposmiko Can you confirm that it only affects branches that are renamed/autocorrected by g10k?

tomposmiko commented 3 years ago

Oops, sorry, did not see your answer. I will try to check it at this weekend.

tomposmiko commented 3 years ago

hi @xorpaul ,

I cannot confirm. This is what I did:

Go to the environments directory. Copy one branch dir to another. Run g10k.

The copied directory remained in place.

Am I correct, it should be removed? It was not autocorrected by g10k.

xorpaul commented 3 years ago

Yes, it should be removed and I can't reproduce your problem:

$ ./g10k -version
g10k  v0.8.15  Build time: 0.8.15 UTC
$ cat test.yaml
---
:cachedir: '/tmp/g10k'
use_cache_fallback: false

deploy:
  purge_levels: [ 'deployment', 'environment', 'puppetfile' ]

sources:
  example:
    remote: 'https://github.com/xorpaul/g10k-environment.git'
    basedir: '/tmp/example/'
    prefix: true
    force_forge_versions: false
$ ./g10k -config test.yaml -branch benchmark -info
Need to sync /tmp/example/example_benchmark
WARN: Forge module puppetlabs-azure has been deprecated by its author since 2020-04-15 01:31:33 -0700
WARN: Forge module puppetlabs-corosync has been deprecated by its author since 2017-03-21 11:28:39 -0700 The author has suggested puppet-corosync as its replacement
WARN: Forge module vshn-gitlab has been deprecated by its author since 2018-02-08 09:25:47 -0800 The author has suggested puppet-gitlab as its replacement
WARN: Forge module locp-cassandra has been deprecated by its author since 2018-01-05 10:00:09 -0800 The author has suggested puppet-cassandra as its replacement
WARN: Forge module arioch-redis has been deprecated by its author since 2019-03-15 14:56:45 -0700 The author has suggested puppet-redis as its replacement
WARN: Forge module camptocamp-kmod has been deprecated by its author since 2021-06-14 03:15:07 -0700 The author has suggested puppet-kmod as its replacement
WARN: Forge module puppetlabs-dsc has been deprecated by its author since 2021-05-19 16:09:30 -0700
WARN: Forge module example42-logstash has been deprecated by its author since 2019-10-28 03:29:25 -0700 The author has suggested elastic-logstash as its replacement
Need to sync /tmp/example/example_benchmark/external_modules/stdlib
Need to sync /tmp/example/example_benchmark/external_modules/gitlab
Need to sync /tmp/example/example_benchmark/external_modules/supervisord
Need to sync /tmp/example/example_benchmark/external_modules/kmod
Need to sync /tmp/example/example_benchmark/external_modules/accounts
Need to sync /tmp/example/example_benchmark/external_modules/redis
Need to sync /tmp/example/example_benchmark/external_modules/openssl
Need to sync /tmp/example/example_benchmark/external_modules/ntp
Need to sync /tmp/example/example_benchmark/external_modules/unattended_upgrades
Need to sync /tmp/example/example_benchmark/external_modules/corosync
Need to sync /tmp/example/example_benchmark/external_modules/cassandra
Need to sync /tmp/example/example_benchmark/external_modules/xinetd
Need to sync /tmp/example/example_benchmark/external_modules/locales
Need to sync /tmp/example/example_benchmark/external_modules/vcsrepo
Need to sync /tmp/example/example_benchmark/external_modules/resolv_conf
Need to sync /tmp/example/example_benchmark/external_modules/lvm
Need to sync /tmp/example/example_benchmark/external_modules/vmwaretools
Need to sync /tmp/example/example_benchmark/external_modules/apt
Need to sync /tmp/example/example_benchmark/external_modules/firewall
Need to sync /tmp/example/example_benchmark/external_modules/aws
Need to sync /tmp/example/example_benchmark/external_modules/azure
Need to sync /tmp/example/example_benchmark/external_modules/apache
Need to sync /tmp/example/example_benchmark/external_modules/dsc
Need to sync /tmp/example/example_benchmark/external_modules/inifile
Need to sync /tmp/example/example_benchmark/external_modules/concat
Need to sync /tmp/example/example_benchmark/external_modules/postfix
Need to sync /tmp/example/example_benchmark/external_modules/logstash
Need to sync /tmp/example/example_benchmark/external_modules/haproxy
Need to sync /tmp/example/example_benchmark/external_modules/mysql
Synced test.yaml with branch benchmark with 5 git repositories and 25 Forge modules in 4.8s with git (1.4s sync, I/O 0.2s) and Forge (3.2s query+download, I/O 6.5s) using 50 resolve and 20 extract workers
$ ./g10k -config test.yaml -branch master -info
Need to sync /tmp/example/example_master
Need to sync /tmp/example/example_master/external_modules/postfix
Need to sync /tmp/example/example_master/external_modules/stdlib
Need to sync /tmp/example/example_master/external_modules/concat
Need to sync /tmp/example/example_master/external_modules/inifile
Need to sync /tmp/example/example_master/external_modules/ntp
Need to sync /tmp/example/example_master/external_modules/resolv_conf
Need to sync /tmp/example/example_master/external_modules/firewall
Need to sync /tmp/example/example_master/external_modules/aws
Need to sync /tmp/example/example_master/external_modules/apt
Need to sync /tmp/example/example_master/external_modules/apache
Synced test.yaml with branch master with 5 git repositories and 6 Forge modules in 2.7s with git (0.4s sync, I/O 0.1s) and Forge (2.2s query+download, I/O 1.1s) using 50 resolve and 20 extract workers
$ cp -pr /tmp/example/example_benchmark/ /tmp/example/example_master/
$ ls -l /tmp/example/example_master/
total 12
drwxrwxr-x  3 xorpaul xorpaul 4096 Jun 25 13:52 example_benchmark
drwxrwxr-x 12 xorpaul xorpaul 4096 Jun 25 13:52 external_modules
-rw-rw-r--  1 xorpaul xorpaul  611 Aug 21  2020 Puppetfile
$ ./g10k -config test.yaml -branch master -info
Removing unmanaged path /tmp/example/example_master/example_benchmark
Removing unmanaged path /tmp/example/example_master/example_benchmark/.g10k-deploy.json
Removing unmanaged path /tmp/example/example_master/example_benchmark/Puppetfile
Removing unmanaged path /tmp/example/example_master/example_benchmark/external_modules
Synced test.yaml with branch master with 5 git repositories and 6 Forge modules in 0.9s with git (0.5s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolve and 20 extract workers
$ ls -l /tmp/example/example_master/
total 8
drwxrwxr-x 12 xorpaul xorpaul 4096 Jun 25 13:52 external_modules
-rw-rw-r--  1 xorpaul xorpaul  611 Aug 21  2020 Puppetfile

Can you provide more information or try to reproduce with my test.yaml and g10k commands? The control repository is public on Github

tomposmiko commented 1 year ago

@xorpaul Finally, I got here:) Sorry for the delay...really. I cannot reproduce it with your test case either. Can I help here with something? BTW, I don't use Puppetifle, just deploy various branches of code to a puppet server.

xorpaul commented 1 year ago

@tomposmiko Have you also tried a more recent version like https://github.com/xorpaul/g10k/releases/tag/v0.9.7 ? Because I've reworked the purge logic quite a lot since 2021 :grin:

tomposmiko commented 1 year ago

Yes, I tested it with this version: g10k v0.9.7 Build time: 2023-02-01_10:14:16 UTC Do you want me to run another test?

CommanderTso commented 1 year ago

Hi @xorpaul. Also having issues with getting purge to work. Unless I'm screwing it up, I'm managing to reproduce this on one of our Rocky 8 puppet masters. Happy to test more on this end. Thanks.

[root@b-puppet-compile01 test]# pwd
/tmp/test
[root@b-puppet-compile01 test]# cat test.yaml
:cachedir: '/tmp/g10k'
use_cache_fallback: false

deploy:
  purge_levels: [ 'deployment', 'environment', 'puppetfile' ]

sources:
  example:
    remote: 'https://github.com/xorpaul/g10k-environment.git'
    basedir: '/tmp/example/'
    prefix: true
    force_forge_versions: false
[root@b-puppet-compile01 test]# g10k --version
g10k  v0.9.8  Build time: 2023-05-25_13:02:05 UTC
[root@b-puppet-compile01 test]# g10k -config test.yaml -branch benchmark -info
Need to sync /tmp/example/example_benchmark
WARN: Forge module camptocamp-postfix has been deprecated by its author since 2021-08-26 05:20:19 -0700 The author has suggested puppet-postfix as its replacement
WARN: Forge module camptocamp-openssl has been deprecated by its author since 2022-05-12 14:31:08 -0700
WARN: Forge module camptocamp-kmod has been deprecated by its author since 2021-06-14 03:15:07 -0700 The author has suggested puppet-kmod as its replacement
WARN: Forge module puppetlabs-dsc has been deprecated by its author since 2021-05-19 16:09:30 -0700
WARN: Forge module vshn-gitlab has been deprecated by its author since 2018-02-08 09:25:47 -0800 The author has suggested puppet-gitlab as its replacement
WARN: Forge module example42-logstash has been deprecated by its author since 2019-10-28 03:29:25 -0700 The author has suggested elastic-logstash as its replacement
WARN: Forge module puppetlabs-azure has been deprecated by its author since 2020-04-15 01:31:33 -0700
WARN: Forge module puppetlabs-corosync has been deprecated by its author since 2017-03-21 11:28:39 -0700 The author has suggested puppet-corosync as its replacement
WARN: Forge module arioch-redis has been deprecated by its author since 2019-03-15 14:56:45 -0700 The author has suggested puppet-redis as its replacement
WARN: Forge module locp-cassandra has been deprecated by its author since 2018-01-05 10:00:09 -0800 The author has suggested puppet-cassandra as its replacement
Need to sync /tmp/example/example_benchmark/external_modules/stdlib
Need to sync /tmp/example/example_benchmark/external_modules/redis
Need to sync /tmp/example/example_benchmark/external_modules/logstash
Need to sync /tmp/example/example_benchmark/external_modules/mysql
Need to sync /tmp/example/example_benchmark/external_modules/gitlab
Need to sync /tmp/example/example_benchmark/external_modules/kmod
Need to sync /tmp/example/example_benchmark/external_modules/vmwaretools
Need to sync /tmp/example/example_benchmark/external_modules/azure
Need to sync /tmp/example/example_benchmark/external_modules/cassandra
Need to sync /tmp/example/example_benchmark/external_modules/ntp
Need to sync /tmp/example/example_benchmark/external_modules/aws
Need to sync /tmp/example/example_benchmark/external_modules/haproxy
Need to sync /tmp/example/example_benchmark/external_modules/inifile
Need to sync /tmp/example/example_benchmark/external_modules/locales
Need to sync /tmp/example/example_benchmark/external_modules/postfix
Need to sync /tmp/example/example_benchmark/external_modules/vcsrepo
Need to sync /tmp/example/example_benchmark/external_modules/openssl
Need to sync /tmp/example/example_benchmark/external_modules/xinetd
Need to sync /tmp/example/example_benchmark/external_modules/apt
Need to sync /tmp/example/example_benchmark/external_modules/supervisord
Need to sync /tmp/example/example_benchmark/external_modules/apache
Need to sync /tmp/example/example_benchmark/external_modules/firewall
Need to sync /tmp/example/example_benchmark/external_modules/lvm
Need to sync /tmp/example/example_benchmark/external_modules/dsc
Need to sync /tmp/example/example_benchmark/external_modules/accounts
Need to sync /tmp/example/example_benchmark/external_modules/concat
Need to sync /tmp/example/example_benchmark/external_modules/unattended_upgrades
Need to sync /tmp/example/example_benchmark/external_modules/corosync
Need to sync /tmp/example/example_benchmark/external_modules/resolv_conf
Synced test.yaml with branch benchmark with 5 git repositories and 25 Forge modules in 35.9s with git (20.0s sync, I/O 5.4s) and Forge (5.7s query+download, I/O 14.2s) using 50 resolve and 20 extract workers
[root@b-puppet-compile01 test]# g10k -config test.yaml -branch master -info
Need to sync /tmp/example/example_master
WARN: Forge module camptocamp-postfix has been deprecated by its author since 2021-08-26 05:20:19 -0700 The author has suggested puppet-postfix as its replacement
Need to sync /tmp/example/example_master/external_modules/postfix
Need to sync /tmp/example/example_master/external_modules/stdlib
Need to sync /tmp/example/example_master/external_modules/inifile
Need to sync /tmp/example/example_master/external_modules/resolv_conf
Need to sync /tmp/example/example_master/external_modules/concat
Need to sync /tmp/example/example_master/external_modules/ntp
Need to sync /tmp/example/example_master/external_modules/firewall
Need to sync /tmp/example/example_master/external_modules/aws
Need to sync /tmp/example/example_master/external_modules/apt
Need to sync /tmp/example/example_master/external_modules/apache
Synced test.yaml with branch master with 5 git repositories and 6 Forge modules in 6.7s with git (0.5s sync, I/O 0.1s) and Forge (1.3s query+download, I/O 0.9s) using 50 resolve and 20 extract workers
[root@b-puppet-compile01 test]# cp -pr /tmp/example/example_benchmark/ /tmp/example/example_master/
[root@b-puppet-compile01 test]# ls -l /tmp/example/example_master/
total 4
drwxr-xr-x  3 root root  73 Aug  7 13:00 example_benchmark
drwxr-xr-x 12 root root 146 Aug  7 13:01 external_modules
-rw-rw-r--  1 root root 611 Aug 21  2020 Puppetfile
[root@b-puppet-compile01 test]# g10k -config test.yaml -branch master -info
WARN: Forge module camptocamp-postfix has been deprecated by its author since 2021-08-26 05:20:19 -0700 The author has suggested puppet-postfix as its replacement
Synced test.yaml with branch master with 5 git repositories and 6 Forge modules in 0.7s with git (0.4s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolve and 20 extract workers

xorpaul commented 1 year ago

g10k checks the /tmp/example/example_master/.g10k-deploy.json file if something changed and a sync is needed.

It does not detect if some folder was copied into the Puppet environment, when neither the control repository commit hash or the Puppetfile checksum was changed.

To force a check/sync which will detect the copied folder you need to force a sync by deleting or modifying the .g10k-deploy.json file:

$ ./g10k -version
g10k  v0.9.8  Build time: 2023-05-25_13:02:05 UTC
$ cp -pr /tmp/example/example_benchmark/ /tmp/example/example_master/
$ ls -l /tmp/example/example_master/
total 12
drwxrwxr-x  3 xorpaul xorpaul 4096 Aug  9 17:07 example_benchmark
drwxrwxr-x 12 xorpaul xorpaul 4096 Aug  9 17:07 external_modules
-rw-rw-r--  1 xorpaul xorpaul  611 Aug 21  2020 Puppetfile
$ rm /tmp/example/example_master/.g10k-deploy.json
$ ./g10k -config test.yaml -branch master -debug  2>&1 | grep example_master/example_benchmark
2023/08/09 17:08:09 DEBUG purgeControlRepoExceptModuledir(): deleting /tmp/example/example_master/example_benchmark
2023/08/09 17:08:09 DEBUG purgeDir(): Trying to remove: /tmp/example/example_master/example_benchmark called from purgeControlRepoExceptModuledir
$ ls -l /tmp/example/example_master/
total 8
drwxrwxr-x 12 xorpaul xorpaul 4096 Aug  9 17:07 external_modules
-rw-rw-r--  1 xorpaul xorpaul  611 Aug 21  2020 Puppetfile

g10k then does delete the copied folder as you can see with the -debug parameter

2023/08/09 17:08:09 DEBUG purgeDir(): Trying to remove: /tmp/example/example_master/example_benchmark called from purgeControlRepoExceptModuledir

CommanderTso commented 1 year ago

Gotcha. Above I was trying to mimic a situation where r10k would delete environments that don't have a corresponding remote branch. Going to test things a bit more on my side and will repost with hopefully a good repo case if I can't get things working. Thanks.

CommanderTso commented 1 year ago

OK, I dug into this a good bit more. For the life of me, I can't get g10k to purge environments. Can you please let me know if I'm doing something wrong?

Config g10k_test.yaml:

:cachedir: '/tmp/g10k_cache_commandertso'

deploy:
  purge_levels: [ 'deployment', 'environment', 'puppetfile' ]

sources:
  example:
    # remote: 'https://github.com/xorpaul/g10k-environment.git'
    remote: 'https://github.com/CommanderTso/g10k_test.git'
    basedir: '/tmp/g10k_deployments_commandertso/'
    prefix: true

Repro Steps:

Create repo g10k_test on my github account as a public repo with a minimal Puppetfile and a readme
Deploy main branch via command line with g10k -config /tmp/g10k_test.yaml -environment example_main -debug
Create branch on github for g10k_test repo called test_branch
Deploy test_branch with g10k -config /tmp/g10k_test.yaml -environment example_test_branch -debug
Delete test_branch branch on github
Deploy main branch again with same command as the first time

Result:

The example_test_branch is not deleted as expected based on my config file.

2023/08/24 13:42:50 DEBUG purgeUnmanagedContent(): Glob'ing with path /tmp/g10k_deployments_commandertso/example_*
2023/08/24 13:42:50 DEBUG purgeUnmanagedContent(): Checking if environment should exist: example_main
2023/08/24 13:42:50 DEBUG purgeUnmanagedContent(): Not purging environment example_main
2023/08/24 13:42:50 DEBUG purgeUnmanagedContent(): Skipping purging unmanaged content for Puppet environment 'example_test_branch', because -environment parameter is set to example_main

I have to assume I'm doing something wrong here, but damned if I can figure out what it is. The equivalent setup worked for us with r10k previously. Please let me know if you need any more detail or testing done. Thanks.

xorpaul commented 1 year ago

I mean g10k is telling you why it does not delete the non-existing branch:

Skipping purging unmanaged content for Puppet environment 'example_test_branch', because -environment parameter is set to example_main

Why not sync all branches of your control repository and not using -environment or -branch parameters, then g10k should delete the Puppet environment of the non-existing branch?

For me if a user specifies that g10k should only sync one specific branch either by using -environment or -branch parameters, then it does not modify or touch any other branches/Puppet environments.

I'm not sure if I want to copy the behaviour of r10k here :thinking:

CommanderTso commented 1 year ago

Thanks for the response.

I was able to now get this to work by having my compile masters all sync the entire control repo every time. That's not optimal - the build, while still fast, is 2-3x slower now.

If I'm understanding how g10k works correctly, the current g10k behavior is a big, undocumented divergence from how r10k works. When deploying via r10k with r10k deploy environment, any environments without a corresponding git remote branch are deleted from the master. I expected g10k to work in a similar fashion, given this portion of the g10k readme:

Starting with v.0.9.0 g10k supports the r10k-like purge behaviour of stale content with the different configuration settings purge_level and purge_allowlist as documented here for purge_levels and here for purge_allowlist

The r10k behavior seems excellent to me, since if you are creating dynamic environments on the regular and not deleting them, your build masters will quickly get loaded up with old, unnecessary environments.

If you can confirm for me that the current behavior is what you are looking for, I'd be happy to put in a PR with a change to the readme.

alexjfisher commented 1 month ago

I'm not sure if I want to copy the behaviour of r10k here 🤔

Would it be possible to make this behaviour configurable? I've got around 500 environments (partially because users are really bad at housekeeping), and checking/syncing them all just to get the purge behaviour isn't going to work well for me.

xorpaul / g10k

Purge does not work #182