[Req] Test templates changes between releases

[san7ket]

It would be great to have automated test for verifying that an existing template with content "a" e.g. in 6.2 is automatically updated to content "b" after update to 6.3. The most important template would be the kickstart one which I'm I think changed between versions. Maybe it would be best to verify that after update, the template in database is 1:1 to the version that is on filesystem. Templates can be found at ~foreman/app/views/unattended/provisioning_templates and /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-*/app/views/foreman/unattended. The later directory contains the kickstart template. The same behavior should be observable for partition tables. It would be also to good to verify that templates that are shipped in 6.3 by default are locked. Note that whenever foreman-rake db:seed is run (usually from satellite installer), the templates in DB should be updated to versions on filesystem.

@san7ket is there such test or could it be added to the upgrade suite? @rplevka could you please add the right flag to this issue please?

[rplevka]

I don't think this got to exist here, we might have kept the robottelo issue opened. Opened a robottelo RFT issue Since we're about to compare db vs fs, this test won't require any 'pre-upgrade checks' and will make sense even in regular (non-upgraded environment.

[sghai]

@rplevka cool, please feel free to close this issue and linked it up w/ the one opened on robottelo.

[rplevka]

@san7ket, do you agree?

[san7ket]

yea, I think so, but still we can also have this here , to check the content changes were done to the templates. Currently, we have tests to check the template entity exists (pre/post comparison), but we do not check the content changes that were done to templates

[san7ket]

@rplevka If we are going to have this in robottelo, please feel free to close this issue here.

[jyejare]

@san7ket , Can we close this else we have to find way to compare template data and compare pre and post either by existence or scenario tests.

[rplevka]

Or, maybe we should come up with the tests that actually use the templates - this way we wouldn't test whether some templates got changed, but whether they continue to work instead. I know this would require a number of tests, but maybe we can cover most of them by few end-to-ends

[jyejare]

@rplevka , That will be a huge effort to put. Better we can ask dev's to fetch template contents either from API or CLI and compare that data with Upgrade Existence Tests.

@sghai , @san7ket @ntkathole Thoughts ?

[sghai]

A Template contains many statements apart from plain os provisioning, so with provisioning, we may not be able to identify gaps and moreover covering all possible scenarios required more efforts. So existence tests would be a good way to go, however,lets see how much efforts needed for existence tests as well. @jyejare : could you please check efforts required to automate via existence tests ?

[san7ket]

One implementation:

hammer -u admin -p changeme template dump --id 30 >>a.txt
>>> import hashlib
>>> pre_upgrade_fhash = hashlib.md5()
>>> print(pre_upgrade_fhash)
<md5 HASH object @ 0x7f0a677e6ad0>
>>> print(pre_upgrade_fhash.hexdigest())
cf1cd8d7b3743ebb6bc305df018d27b1

Now, we can check the hash of all the files with what's expected and if the assert fails for particular file, the subtest will fail and we go and check it manually, if the change was expected. If, intended we change the hash of file for expected hash. This would be fast, rather than comparing the content one character by character. @jyejare @sghai I think we can do it with existence tests.

compare(after_upgrade_hash_dict: {puppet.conf: <hash>, file2: <hash>}, expected_hash_dict: {puppet.conf: <hash>, file2: <hash>})