TELIC Paper Content Discussion

[Talismanic]

1. TELIC identifies a test script to include a test play if a play within a script includes (i) one of the following keywords: ‘check’, ‘determine’, ‘ensure’,‘test’, ‘validate’, and ‘verify’. Actually TELIC classifies a test script if it is under "tests" directory and have yml/yaml extension.

2. I have not determined Total Test Plays and LOC. Have you calculated those bhaiya from the scripts?

3. For selecting the oracle dataset I used RAND() function of MySQL to detect 100 random scripts from our anti-pattern database.

4. In Listing 6, our example of adding the yum repositories from external URL is actually the only way adding new repository. But if we do installation of a package from external repository that will be anti-pattern. For example, following is a hypothetical example of anti-pattern:

   
   - name: Downloading nginx rpm,
     get_url:
       url: http://nginx.org/packages/centos/{{ansible_distribution_major_version}}/noarch/RPMS/nginx-release-centos-{{ansible_distribution_major_version}}-0.el{{ansible_distribution_major_version}}.ngx.noarch.rpm
       dest: /tmp/ngx.noarch.rpm
   
   - name: Install nginx
      yum:
           name: /tmp/ngx.noarch.rpm
           state: present

The right way to do this would have been simply taking full advantage fro yum module:

• name: install nginx yum: name=nginx state=present ```

[akondrahman]

The latter ... stuff like this: https://chrisbergeron.com/2018/06/08/ansible_performance_tuning/

[Talismanic]

The latter ... stuff like this: https://chrisbergeron.com/2018/06/08/ansible_performance_tuning/

In practical scenario,practitioners do not write the whole playbook by themselves. 90% use cases are out there somewhere. In most cases, we actually dont bother with the time ansible takes. For example, if an ansible playbook takes 20 mins to set up a kubernetes cluster, it is acceptable to us as this is far better than manually provisioning the system.

For some quick performance what I used do is disabling gathering facts parts where those are not required. Once or twice I had tuned the ssh pipelining as far as I remember.

Also in case of installation type codes, I used mirrors in local network as an artifact repository.

But I hardly noticed coding bugs have created any performance issue. But I am not dismissing the probability that coding bug can introduce performance problems. For example, if we download a rpm file and install it, playbook may take more time than installing it through yum, as yum repo-cache may be locally available.

[akondrahman]

Great insight @Talismanic ... thanks! I need to look for instances of these on GitHub or Stack Overflow

• we download a rpm file and install it, playbook may take more time than installing it through yum, as yum repo-cache may be locally available.
• I used mirrors in local network as an artifact repository
• disabling gathering facts parts where those are not required

[Talismanic]

I need to look for instances of these on GitHub or Stack Overflow

Bhaiya, is this for any other project?

[akondrahman]

@Talismanic it is for another set of projects. Currently thinking of what research avenues my team can explore in the next 4-5 years: if you have any feel free to share!

[Talismanic]

if you have any feel free to share!

Let me think for some day Bhaiya. I will share you if anything crosses my mind.

On a different note bhaiya, while submitting the bug reports, I am going through the open source repositories and seeing some plain-text mysql password in the test scripts. We have this sensitive data leakage category in our exploration suite initially, but discarded this later as our openstack data source did not have anything like this.

[akondrahman]

Those are security smells ... already addressed in our security smell papers.

[akondrahman]

@Talismanic

I have added the abstract ... please check and see if there are interesting things of the paper that are not mentioned in the abstract.

[akondrahman]

@Talismanic

Do you think any of the five test smell categories have implications related to reproducible deployments?

[Talismanic]

@akondrahman Bhai, I am assuming below things:

By reproducible deployments, you meants a codebase which will create similar infrastructure if they are run multiple times in the same environment.

Based on this assumption, if we can examine case by case:

1. Assertion Confusion: If the script is truely test script, this should not have any impact in reproducibility.
2. Mystery Guest: If can have issue with reproducibility if the environments are not completely identical where the code will run. For example, one environment can have some data in one location, but another may not have the same.
3. Local Only Test: It can impact
4. Leftover Installation: It should not have any impact
5. Linter Strangler: Similar to 2.

However, being said those, reproducibility may be more impacted/directly impacted by production code smells (similar to your prior researches) than the test smells.

[akondrahman]

Thanks @Talismanic

Based on your observation I would request you to write a paragraph or two on the potential impact of test smells on reproducible deployments. Share the paragraph(s) here when ready. This content will go into the discussion section.

[Talismanic]

@akondrahman Bhaiya, I could not finish it today. I will get back on this by tomorrow.

[Talismanic]

Implication on Reproducible Deployment One of the important aspect of IaC is it's reproduciblity. That is IaC scripts should be able to generate similar target infrastructure and IaC testing should ensure that. However, some of our test smells can create false positive and false negative test result. For example, external dependency which Mystery Guest introduces can lead to test failure in some environments whereas the actual production code might have been able to create the intended infrastructure. So this smell may give a false negative result about the reproducible deployment. On tha other hand, Local Only Testing can make some test cases passed in local environment whereas those test cases may fail in production environment along with the failure of production IaC code to deploy the target infrastructure. In these case, this smell generates a false positive notion about the reproducible deployment. Principally, IaC scripts should be idempotent and hence leftover installation should not have any impact on the reproducibility of the deployment. Similar logic is applicable for Assertion Roulette and Linter Strangler.

@akondrahman Bhai, I withdraw my opinion that Linter Strangler can have any impact on Reproducible Deployment. Above is my understanding. Those are not validated by anyone till now.

[akondrahman]

@Talismanic ^ your writing is a good start ... but you need to focus on local-only testing only, and talk more about what is a reproducible deployment and why reproducible deployments are important, and how local-only testing is strongly connected with reproducible deployments. Find some examples from blog posts used in your workshop paper to make the case stronger.

[Talismanic]

Reproducible Deployment is one of the important benefits of IaC. Especially with the advent of Cloud Services, practitioners are adopting IaC rapidly to reach an expected end state of software infrastructure for similar needs. For example, setting up a database server is a common need for all software. As soon as practitioners develop the IaC script to create a database server from code, they can reuse it in multiple software deployments with a similar cloud service provider. IaC scripts are expected to be idempotent. Idempotence ensures that IaC scripts will produce the same end infrastructure irrespective of the initial infrastructure where it has been run. This behavior makes the deployments reproducible. Based on our analysis, we can say that the Local Only Test can create false-positive test results endangering the reproducibility of the deployments. Practitioners have already emphasized that "to properly test your infrastructure code, you typically have to deploy it to a real environment, run real infrastructure, validate that it does what it should, and then tear it all down [1]". To analyze how Local Only Test can impact Reproducible Deployment we can reconsider the database setup task mentioned above. For database installation, sometimes extra dependencies are required to be installed e.g java or OpenJDK. An IaC script may not implement the dependency installation tasks properly. If this script is tested on a local machine that already has necessary dependencies installed, it will produce a false-positive test result. However, if the scripts run in a remote environment where these dependencies are not already present, those will fail to reproduce the expected end infrastructure. So Local Only Test can reduce the generalization of the IaC scripts on different environment and result in the development of IaC scripts which cannot create reproducible deployments.

1 : Lesson 4

Will it work @akondrahman Bhai?

[akondrahman]

@Talismanic Looks good. Thanks.

[Talismanic]

@akondrahman Bhai, I was a bit busy for since last weekend due to some deadlines. Now, I am back to research work. Please let me know if I need to do anything.

[akondrahman]

@Talismanic

No worries. I am trying to get more survey responses at this point. When will you be available for reading group?

[Talismanic]

When will you be available for reading group?

Bhaiya, any day from Monday to Thursday from 10:30 pm Dhaka time onwards.

[akondrahman]

@Talismanic

Sent you a meeting invite

[akondrahman]

@Talismanic

Need your immediate help on the following issues:

Issue-1

For the following code we write If any of the tests fail then a practitioner may find it hard to understand which test is failing, as Ansible will generated one common test failure message for the three elements.

Can you give me the exact error message when a test case with assertion roulette fails.

- name: Bond check
  assert:
    that:
      - ansible_bond0['active'] == true
      - ansible_bond0['type'] == 'bonding'
      - ansible_bond0['mtu'] == 9000

Issue-2

I need a reference that backs up the following statement Behavior of device drivers, such as network device drivers are dependent on the type of operating system.

[Talismanic]

Can you give me the exact error message when a test case with assertion roulette fails.

Bhaiya, I can not reproduce this exact case due to unavailabilty of resources. But I have ran a sample case about what is the response when one of the cases in an assertion roulette fails and kept it in this issue https://github.com/akondrahman/IaCTesting/issues/23#issue-784775262 .

reference that backs up the following statement

Different Operating System has different framework for making device drivers. For example, Microsoft has Windows Driver Foundation which defines how device drivers should work in an windows environment. Similarly, Apple has I/O Kit framework which guides the behavior of device drivers in MacOS and for Linux based systems device drivers comes with kernel.

Bhaiya, I have not actually read what is inside those referrence in detail. Mostly they are development guide on how to develop device drivers on specific opearting system. Is this sufficient?

[akondrahman]

@Talismanic

I just need a citation to use. So share the development guide(s) please

[Talismanic]

Windows Driver Foundation Developer Guide Citation

@book{orwick2007developing,
  title={Developing Drivers with the Windows Driver Foundation: Dev Driver Win Driver Fou\_p1},
  author={Orwick, Penny and Smith, Guy},
  year={2007},
  publisher={Microsoft Press}
}

I/O Kit for MacOS Developer Guide Citation

@article{lee2005kit,
  title={I/O kit drivers for L4},
  author={Lee, Geoffrey},
  journal={BE thesis, School of Computer Science and Engineering, University of NSW, Sydney},
  volume={2052},
  year={2005},
  publisher={Citeseer}
}

Linux Device Driver Developer Guide Citation

@book{rubini2001linux,
  title={Linux device drivers},
  author={Rubini, Alessandro and Corbet, Jonathan},
  year={2001},
  publisher={" O'Reilly Media, Inc."}
}

Also I found below lines in Dingo's paper which can also be used as the support:

Each device has a unique device protocol defined by the manufacturer. Drivers conceal this device diversity 
from the OS by implementing standard software protocols, common to a family of devices, e.g. 
Ethernet, audio, etc. These protocols are  defined by the OS. The OS also defines protocols for 
accessing the support services that it provides.

From this we can decide that each OS has separate implementation of Device Drivers.

Citation for Dingo's paper is below:

@inproceedings{ryzhyk2009dingo,
  title={Dingo: Taming device drivers},
  author={Ryzhyk, Leonid and Chubb, Peter and Kuz, Ihor and Heiser, Gernot},
  booktitle={Proceedings of the 4th ACM European conference on Computer systems},
  pages={275--288},
  year={2009}
}

@akondrahman Bhai, these are my findings.

[akondrahman]

Great list. Thanks!

[akondrahman]

@Talismanic

In the discussion we have a sentence Researchers have documented the use of complex steps to provision infrastructure using IaC. Can you please find reff. for this?

[Talismanic]

@akondrahman Bhai, In this, it has been reported that Nodes too deep is a bad practice in IaC stating:

The tree of nodes generated from a single script is too deep

Its citation is:

@inproceedings{guerriero2019adoption,
  title={Adoption, support, and challenges of infrastructure-as-code: Insights from industry},
  author={Guerriero, Michele and Garriga, Martin and Tamburri, Damian A and Palomba, Fabio},
  booktitle={2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)},
  pages={580--589},
  year={2019},
  organization={IEEE}
}

If we rephrase the sentence like : Researchers have documented that, the use of complex steps to provision infrastructure using IaC often creates maintainability problem, than I think this citation can be used.

[akondrahman]

@Talismanic

We will read your paper tomorrow at 11 AM CST. Please join if you have time. I am expecting you to answer questions ... I will keep track fo the changes and later in the day make necessary changes. While making the changes I will seek your help if needed.

[Talismanic]

@akondrahman bhai, surely I will join Inshallah.

[akondrahman]

@Talismanic are you in the meeting?

[akondrahman]

@Talismanic

Thanks for attending today. I think for you there are three major tasks:

1. Find a way to highlight test smell in Latex Listing
2. Write down assertion-related example in simple words
3. Read out loud to find examples are clear

I will take care of stuff in abstract and intro. Let me know when the above are done.

[akondrahman]

@Talismanic

I see you already have some Latex tricks fro code highlighting. To fix the alignment issue I suggest you use the following setting for minted everywhere:

\begin{minted}[ breaklines, fontsize=\small, linenos=true, numbersep=2pt, xleftmargin=6 ]{yaml}

and I also think you need to add what is being tested for Listing 1 using \mintinline ... please add that. See this for reference: http://tug.ctan.org/macros/latex/contrib/minted/minted.pdf

Finally, what actionable activities should we take on to mitigate mystery guest instances?

[Talismanic]

@Talismanic

Thanks for attending today. I think for you there are three major tasks:

1. Find a way to highlight test smell in Latex Listing
2. Write down assertion-related example in simple words
3. Read out loud to find examples are clear

I will take care of stuff in abstract and intro. Let me know when the above are done.

I am on those Bhai. I will sit tonight to work on this.

[Talismanic]

@akondrahman Bhai, I am seeing there is second round of meeting at 11 am CST on 9th Feb-2021. Will it happen?

[akondrahman]

@Talismanic

No meeting today.

As you fix the above issues also think about the following:

Finally, what actionable activities should we take on to mitigate mystery guest instances?

[Talismanic]

1. Find a way to highlight test smell in Latex Listing

Done Bhaiya.

[Talismanic]

Finally, what actionable activities should we take on to mitigate mystery guest instances?

Practitioners can develop test plays so that each test script is self-contained. Practioners can avoid importing playbooks and "use same playbook all the time with embedded tests in development" as suggested in the Ansible documents [1]. Also, practitioners can use roles to achieve the reusability of same test code. For example, Listing 6 task can be rewritten as a role (e.g sparse_file_creation) and Listing 5 can just use that role using a roles tag. In that case one possible update of Listing 5 would have been:

\begin{listing}[h]\centering
% \begin{minted}[   breaklines, fontsize=\small ]{yaml}
\begin{minted}[  breaklines, fontsize=\small, linenos=true, numbersep=2pt, xleftmargin=6 ]{yaml}
- hosts: webservers
  roles:
     - sparse_file_creation
- name: Playbook for role testing
  hosts: webservers
  post\_tasks:
    - name: Ensure mount are mounted
      command: grep -w '{{ item }}' /proc/mounts
      with\_items:
        - /var/lib/sparse-file
\end{minted}

[1]. https://docs.ansible.com/ansible/latest/reference_appendices/test_strategies.html

[akondrahman]

Thanks @Talismanic

Are you done with your edits?

[Talismanic]

Are you done with your edits?

Not yet Bhaiya. Working on easier example of Assertion Roulette for Listing 4. Will update you by next 1 hour.

[Talismanic]

2. Write down assertion-related example in simple words

- name: get output for single command
- assert:
    that:
      - result.changed == false
      - result.stdout is defined
      - result.stdout_lines is defined

We provide an example in Listing 4 to demonstrate how assertion roulette appears in Ansible test plays. In Listing 4 we observe a testcase is setup to test a variable called 'result' using the assert tag. It is testing whether result has changed, if it has the stdout property and whether there are stdout_lines defined in result. Ansible will report 'fatal: FAILED' message if either result is changed or stdout is not defined or stdout_lines are not found in result. However, if result is changed, Ansible will not test whether stdout and stdout_lines are defined and abort the execution there. So to understand, which property will be failing the practitioners will have to comment two of the three properties and test one property at a time.

@akondrahman Bhai, does this sound like a more easy example for assertion roulette? I have taken it from below file:

C:\mined_repos\Inspq\ansible\test\integration\targets\vyos_command\tests\cli\output.yaml

If it is ok, I will update this in the Overleaf.

[akondrahman]

@Talismanic

I think the network example is better. Whatever you wrote here is fine. Use the same style to explain the network-related example.

[Talismanic]

Use the same style to explain the network-related example.

@akondrahman Bhai, updated in Overleaf.

I had another action point. I was supposed to check whether there is any automated way/tool to handle Leftover Installation. I am looking into that.

[akondrahman]

Thanks for the hard work.

[Talismanic]

Leftover Installation: practitioners can apply the following practices: (i) cleaning the installed dependencies afterthe test is over by developing custom clean up tasks as conductedby other practitioners[14], and (ii) using dedicated containers for this task as host. Tools like Molecule provides steps like 'dependency'[15] where necessary dependencies are installed in dedicated docker containers for testing and finally cleans up the container with step 'destroy'. [16]

[15] https://molecule.readthedocs.io/en/latest/configuration.html#dependency [16] https://molecule.readthedocs.io/en/latest/getting-started.html#run-test-sequence-commands

@akondrahman Bhai, should we use these extra lines and references for tool based leftover installation clean up?

[akondrahman]

@Talismanic

Thanks. The only thing we have left is mystery guest

[Talismanic]

The only thing we have left is mystery guest

Bhaiya, aint this (https://github.com/akondrahman/IaCTesting/issues/21#issuecomment-776024629) is sufficient for Mystery Guest?

[akondrahman]

Yes. Thanks!

[akondrahman]

@Talismanic : I think we are done. Get some sleep. I will ping you tomorrow or later this week if needed.

[Talismanic]

I will ping you tomorrow or later this week if needed

Sure Bhaiya. Good Night for today.

[akondrahman]

@Talismanic

Thanks again for the great writing related to mitigation. Out of curiosity I would like to know, do you think all five test smell categories can be repaired automatically?