TELIC Paper Content Discussion

[Talismanic]

1. TELIC identifies a test script to include a test play if a play within a script includes (i) one of the following keywords: ‘check’, ‘determine’, ‘ensure’,‘test’, ‘validate’, and ‘verify’. Actually TELIC classifies a test script if it is under "tests" directory and have yml/yaml extension.

2. I have not determined Total Test Plays and LOC. Have you calculated those bhaiya from the scripts?

3. For selecting the oracle dataset I used RAND() function of MySQL to detect 100 random scripts from our anti-pattern database.

4. In Listing 6, our example of adding the yum repositories from external URL is actually the only way adding new repository. But if we do installation of a package from external repository that will be anti-pattern. For example, following is a hypothetical example of anti-pattern:

   
   - name: Downloading nginx rpm,
     get_url:
       url: http://nginx.org/packages/centos/{{ansible_distribution_major_version}}/noarch/RPMS/nginx-release-centos-{{ansible_distribution_major_version}}-0.el{{ansible_distribution_major_version}}.ngx.noarch.rpm
       dest: /tmp/ngx.noarch.rpm
   
   - name: Install nginx
      yum:
           name: /tmp/ngx.noarch.rpm
           state: present

The right way to do this would have been simply taking full advantage fro yum module:

• name: install nginx yum: name=nginx state=present ```

[Talismanic]

do you think all five test smell categories can be repaired automatically?

Technically it may be possible. But I am thinking whether that is feasible from ecosystem point of view. As the ansible world is still growing, there can be scenario where anti_patterns can not be avoided. I had a discussion with one practitioner in this Github issue https://github.com/redhat-performance/satperf/issues/144 where we finally agreed that this is how ansible works currently. So ecosystem is still not matured for automated repairing in my opinion.

[akondrahman]

Great insights @Talismanic ... I will do some thinking and let you know my thoughts after FSE.

[akondrahman]

@Talismanic fun trivia question: what is the probability that your FSE paper will be accepted?

[Talismanic]

@Talismanic fun trivia question: what is the probability that your FSE paper will be accepted?

Well, it depends on the reviewers always. If the reviewers have high level knowledge on IaC and Testing, there are good chances of acceptance. From story point of view, I think you told the best possible story.

[akondrahman]

I am glad you like the story. But there is always room for improvement. So let me know if you think of anything.

[Talismanic]

@akondrahman Bhai, I have added two comments in the overleaf. Please have a look when you are free.

Apart from that, I submitted bugs to 18 repositories and got response from 8 practitioners. Do you want me to summarize their insight to check whether anything useful is ther?

[akondrahman]

@Talismanic
Before answering your question, let me ask you one: do you think the current content for RQ2 is interesting?

[Talismanic]

do you think the current content for RQ2 is interesting?

With my very small reading experience of research questions, I think the question is ok. But in your scale, the question has touch of literature I would day. In the whole story of the paper (title and everything) there is a subtle sense of literature. But the findings of the question is really interesting to me.

[akondrahman]

there is a subtle sense of literature

what do you mean?

[Talismanic]

@akondrahman Bhaiya, What I meant is actually that there is a certain type of narrative (something like story) which is good to read. I am not sure whether it is academically interesting, but it feels good to read.

[akondrahman]

@Talismanic I have a task for you: create a figshare (https://figshare.com/) private repository that will have datasets and source code of TELIC. Please remove any identifiers that can reveal your name or identity. Once you have the repository share the link with me.

Another thing: the reviewers may ask about why we did not do validation with empirical data ... what could be a good response?

[Talismanic]

Please remove any identifiers that can reveal your name or identity.

Bhaiya, I need some time for this as my IDE automatically tags my identity with the scripts. I will remove those and prepare a copy. Also about data, are you meaning openstack files or 100 files which we used to verify in oracle dataset?

the reviewers may ask about why we did not do validation with empirical data

Testing Infrastructure as Code is still very young as a research topic and best practices for this are still under exploration. What we reported are our findings and the mentioned anti-patterns are our proposal. There are future scope to adjust the anti-patterns and tune the rules of TELIC. Hence, validating it with empirical dataset poses the risk of over-generalization. Hence we decided to validate the tool with Oracle dataset where we could manually label the anti-patterns within a limited time.

[akondrahman]

@Talismanic

Also about data, are you meaning openstack files or 100 files which we used to verify in oracle dataset?

I am referring to the output CSV files that you created as part of RQ3 as well as the output for the 100 files used in the oracle dataset

[akondrahman]

@Talismanic I am noticing some space between Listing code and the caption, and also after caption. Are there tricks to reduce that?

[Talismanic]

@Talismanic I am noticing some space between Listing code and the caption, and also after caption. Are there tricks to reduce that?

I will explore that tonight Bhaiya.

[akondrahman]

@Talismanic no need to explore space reduction: I was able to shrink the paper to almost 10 pages. It is a little over, but I will fix it over the week.

If I need sth. I will let you know. Closing the issue for the time being.