search

paser-group / iac-compiler-quality

Placeholder for all materials related to studying bugs and vulnerabilities in the Ansible compiler
MIT License
1 stars 1 forks source link

Plans for July 2023 #9

Closed akondrahman closed 1 year ago

akondrahman commented 1 year ago

@sibat119 and @jmsalvador2395 will implement the following heuristics to discover latent type-related bugs in the Ansible compiler. For speed please split the workload. Attempt 1-9 first, then move on to 10 - 14.

Heuristics:

  1. Mixed case sensitivity for string configuration values can trigger bugs in the Ansible compiler
  2. Pass in byte string values instead of regular strings
  3. Generate configuration values that includes both bytes and strings
  4. Generate random MAC addresses
  5. Generate random port numbers
  6. Genrates strings with different encodings using https://stackoverflow.com/questions/1728376/get-a-list-of-all-the-encodings-python-can-encode-to
  7. Generate random IP addresses for recently developed networking modules
  8. Perform division-based operations for assigning limit-related values
  9. Use !null as configuration values in manifests
  10. Exploit regex bugs by generating random regexes
  11. Relatively newly developed modules will have more type-related bugs
  12. Track the flow of data where with_items and authorized_key is used, and then pass in empty strings
  13. Use digits with become_password in an inventory file
  14. Track the flow of data where int() and other explicit casting is used, and then pass in the sample strings and other generated strings

I think these heuristics follow the tempalte pattern as described in "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT" by White et al. (2023).

akondrahman commented 1 year ago

@sibat119 , what components are we targeting for applying the heuristics? I know one is https://github.com/ansible-collections/community.general, what are the others? For example, are we doing anything with https://github.com/ansible/ansible ?

sibat119 commented 1 year ago

@sibat119 , what components are we targeting for applying the heuristics? I know one is https://github.com/ansible-collections/community.general, what are the others? For example, are we doing anything with https://github.com/ansible/ansible ?

We are targeting the following collection for the heuristic:

akondrahman commented 1 year ago

Good list. I am hoping we will catch some bugs in ansible.builtin.

sibat119 commented 1 year ago

Qualitative Study between GPT 3.5 Turbo and GPT 4

Assumptions

  1. We will conduct our test on randomly chosen 10 modules inside ansible.builtin collection.
  2. We will do a comparative analysis on first 3 (total 30 examples) heuristic presented in Intro of this issue.
  3. We will have same set of prompt for both model.
  4. We will conduct the study based on the following criteria: i. Syntactical correctness on first try. ii. Number of second request needed. iiii. Total percentage of syntactical correctness. iv. Soundness of final produced ansible-playbook (Qualitative).

Prompts

System Role

" Role: Ansible Latent Type-Related Bug Explorer

Description: Your prerequisites include comprehensive expertise in Ansible and excellent debugging skills. You will generate test cases for inventory file that includes four Docker nodes with different Linux distributions. These nodes are connected to a network (node-net) with a subnet of 10.1.1.0/24 and a gateway of 10.1.1.254. The nodes and their corresponding IP addresses are ubuntu1 - 10.1.1.1, alpine1 - 10.1.1.2, centos1 - 10.1.1.3, and redhat1 - 10.1.1.4. Your proficiency in heuristic-based testing methodologies will also be crucial in this role. Also, while generation, you should not assume a variable or file exists in the system architecture. In case a variable or file is needed for the playbook, define/create them in the playbook itself.

Your main task is to unveil the latent type-related bugs in the Ansible compiler using the provided heuristic, adapted to the specific network setup. "

Directive

" As an Ansible Latent Bug Finder & Heuristic Test Engineer, your key task is to develop a comprehensive, self-contained Ansible playbook for the '{{module}}' module which {{desc_short}}. This playbook aims to reveal latent type-based bugs, informed by a detailed understanding of the module and its unique options and attributes, including {{attributes}}.

Your playbook should also incorporate test cases based on a specific heuristic: '{{heuristic}}'. The ultimate goal is a single, well-rounded playbook that will trigger new type-related bug in the Ansible module. "

Results

Overall

Model Total valid second request syntax (%) AVG Soundness (0-5) Time
GPT 3.5 29 27 16 93.10 2.47 15 minutes
GPT 4 30 26 12 86.66 3.28 35 minutes