Plan For May

[jmsalvador2395]

TODO

Goals

• [ ] 1) get 100 samples with valid syntax
  • [ ] a) count how many bugs you can generate using dumb prompts
  • [ ] b) grade relevance
• [ ] 2) try to confirm 5-10 new bugs

Current Plan

• [x] 1) create prompt generator
  • [x] a) convert dataset into huggingface format
  • [x] b) build prompt generator class
• [x] 2) create evaluation pipeline
  • [x] a) validate syntax
  • [ ] b) mark files that require sudo access
• [x] 3) Build Testing Infrastructure
  • [x] a) Deploy ubuntu containers
  • [x] b) Automate deployment and redeployment strategies
  • [ ] c) deploy database
• [ ] 4) repeat for 3 weeks
  • [ ] a) generate list of prompts and narrow down to ~100 samples
  • [ ] b) try to implement Ideas To Create New Bugs

Ideas To Create New Bugs

1. Take examples from RQ1 and ask chatgpt to generate code that would expose categorically similar vulnerabilities
   • generate codes but without presenting examples
2. Using current bugs ask chatgpt to create new bugs that would emerge because of this bug
3. use fixed code/commit messages and prompt chatgpt to expose bugs that could potentially emerge from the fix
4. Try to expose new bugs in ansible using bugs that exist in other IaC languages
5. Check if previous bugs exist in new ansible version
6. recreate existing bugs and hopefully stumble upon new ones (not the best idea)

[jmsalvador2395]

I'm working on Current Plan 1a

[jmsalvador2395]

loading dataset and saving results with huggingface pipeline implemented.

[sibat119]

finished plan 1b and now working on plan 2a

[jmsalvador2395]

added checkboxes for infrastructure deployment and marked completed

[akondrahman]

Do we need to keep this issue open ?

[sibat119]

Do we need to keep this issue open ?

We are currently not perusing this direction. For now we can close this issue and reopen when we come back to this.