search

iacsecurity / tool-compare

MIT License
276 stars 59 forks source link

Add makefile and targetted runs #18

Closed owenrumney closed 3 years ago

owenrumney commented 3 years ago

Just a selfish convenience really, won't be offended if you opt to reject

yi2020 commented 3 years ago

@owenrumney looks good. Did you test all the options? We don't have CI yet for this repo, so it would need to be done manually.

yi2020 commented 3 years ago

@owenrumney also note this branch now has a conflict due to a separate PR that I merged.

owenrumney commented 3 years ago

No probs, I'm doing a full run of all tests so I've pull from upstream. Each individual works, just testing the full end to end then will push

owenrumney commented 3 years ago

@yi2020 - I'm struggling to see what updateresults_summary.json in each test-case folder? I can see it being read for the creation of README.md but the creation of it is eluding me. Any pointers?

yi2020 commented 3 years ago

@yi2020 - I'm struggling to see what updateresults_summary.json in each test-case folder? I can see it being read for the creation of README.md but the creation of it is eluding me. Any pointers?

It's manual right now unfortunately (as part of the PR). What I do is I look at the diff of the *_results.txt files and see what new issues are captured that weren't before. If you think about it, there's no way really to know if a tool catches a specific issue programatically. I thought of using regex, but different tools have different naming for issues, so it doesn't work well.

owenrumney commented 3 years ago

Okay, tenth times a charm. I've cut this PR back to just the Makefile and script changes. I'll do the tfsec updates in a separate, dedicated PR. Apologies for all the back and forth required on this one

I've tested each target of the Makefile and they all work

yi2020 commented 3 years ago

Merged, thank you!