search

oscar-stack / puppet-debugging-kit

A batteries-included environment for debugging Puppet powered infrastructure.
29 stars 16 forks source link

Purge PuppetDB #9

Open cdenneen opened 7 years ago

cdenneen commented 7 years ago

@Sharpie Looks like I've just found a bug in PE that I was only able to reproduce by modifying the vms.yaml and append new digit to the agent hostname.

Apparently even with purge clean deactivate there is still leftover fact data in PuppetDB for the previous, now destroyed agent, and a second creation of that agent won't produce the bug due to leftover fact data for that same hostname. Incrementing the hostname allows the bug to show but requires me to keep incrementing the hostname, for further testing. Since my test requires me to create some classification I cannot simply do a vagrant destroy and vagrant up for the master.

So looking for a way to incorporate in the puppet-debugging-kit a FULL purge of a destroyed nodes info from PuppetDB since puppet node deactivate|purge|clean don't seem to be doing it, this maybe be crude but maybe even direct psql delete statements to be issued.

Sharpie commented 7 years ago

The current purge behavior comes from the vagrant-pe_build plugin and is to run a puppet node purge on the agent's certname when the agent VM is destroyed:

https://github.com/oscar-stack/vagrant-pe_build/blob/0.17.2/lib/pe_build/provisioner/pe_agent.rb#L254-L259

The purge command is equivalent to a cert clean + node deactivate + a call to the classifier API than unpins the certname from any groups. As you noted above, node deactivation doesn't actually remove data from the database -- a GC sweep has to occur before the data is truly gone. Until this GC happens, any activity (submitted facts, report, catalog, etc) from that certname will restore the old data. I've filed a ticket for a "delete node" command as I think this would be useful:

https://tickets.puppetlabs.com/browse/PDB-3300

Without such a thing, PSQL statements would be the way to go. However, I'm a bit conflicted about adding this to pe_build. On the one hand, Vagrant testing environments are very prone to certname re-use, so thoroughly cleaning data is important. On the other hand, the interaction between PuppetDB node deactivation and GC can be surprising and it is better to be surprised by it during testing than in production because the test tooling smoothed over the issue.