search

PaloAltoNetworks / ansible-pan

Ansible modules for Palo Alto Networks NGFWs
Other
229 stars 159 forks source link

Testing only in Localhost can create problem #508

Closed Talismanic closed 3 years ago

Talismanic commented 3 years ago

Greetings, I am an engineer who is interested in infrastructure as code testing. Currently, I am looking for testing anti-patterns in iac test scripts. I noticed Local-only Testing occurring in test instances. This can give happy test results in the local environment but in a real production system test can fail due to the difference in the environment. My recommendation is to test IaC code in an isolated non-local environment which needs to be similar to the production environment.

So I have the following queries:

Do you agree that this is an IaC testing anti-pattern? Do you want to fix this?

Any feedback is appreciated.

Source Files: https://github.com/PaloAltoNetworks/ansible-pan/blob/develop/tests/test.yml

btorresgil commented 3 years ago

Greetings!

This Ansible module leverages another library to connect to live devices called pan-os-python. That library is tested against live devices with each change. So by leveraging separation of concerns, we are effectively testing the Ansible modules in a live environment. The only thing we don't do is full e2e testing of Ansible in a live environment, but the coverage we have makes that kind of testing less valuable. It's unlikely we would detect any issues with that testing g that we wouldn't detect with existing testing.

Thanks for reaching out!

Talismanic commented 3 years ago

So by leveraging separation of concerns, we are effectively testing the Ansible modules in a live environment.

That is really cool. So by principle, do you agree that Ansible codes needs to be testing in a 'live-like' environment, right?

btorresgil commented 3 years ago

I can't speak for all Ansible projects as Ansible can be leveraged in a wide variety of ways. For our situation, we try to come as close as we can to an end to end test.