feat: add generate-config command and add support for adding noisy field (global and local) in config file

[AkashKumar7902]

Related Issue

• Info about Issue or bug

Closes: #1012

Describe the changes you've made

added generate-config command and added support for adding noisy fields in config file.

noisy field example:

globalNoise: |
   {
      "global": {
          "body": {
            "url": ["https?://\S+", "http://\S+"]
          },
          "header": {
            "Date": []
          }
      }
     "test-sets": {
         "test-set-name": {
            "body": {},
            "header": {}
         }
      }
   }

============= USAGE============= If the user only wants some of the values of a field to be labelled as noisy, they can provide a corresponding array of regex expression. If an empty array is provided for a key, no matter what the corresponding value is, it would always be ignored as noisy.

Type of change

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Code style update (formatting, local variables)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update

Please let us know if any test cases are added

Please describe the tests(if any). Provide instructions how its affecting the coverage.

Describe if there is any unusual behaviour of your code(Write NA if there isn't)

NA

Checklist:

• [ ] My code follows the style guidelines of this project.
• [ ] I have performed a self-review of my own code.
• [ ] I have commented my code, particularly in hard-to-understand areas.
• [ ] I have made corresponding changes to the documentation.
• [ ] My changes generate no new warnings.
• [ ] I have added tests that prove my fix is effective or that my feature works.
• [ ] New and existing unit tests pass locally with my changes.

Screenshots (if any)

when i pass a correct regex expression for the Date, it successfully gets ignored:

but when i pass a regex, say "test", it behaves as expected and consider the date value in header as not noisy as it doesn't get matched with "test":

[AkashKumar7902]

Hey @Sarthak160 Can you please review it ?

P.S.: I haven't implemented the regex logic yet. Will implement it once a round of review is completed.

[sweep-ai[bot]]

Apply Sweep Rules to your PR?

• [ ] Apply: Leftover TODOs in the code should be handled.
• [ ] Apply: All new business logic should have corresponding unit tests in the tests/ directory.
• [ ] Apply: Any clearly inefficient or repeated code should be optimized or refactored.

[Sarthak160]

Hey @AkashKumar7902 , Can you please fix the conflicts so that I could test and get this merge ?

[Sarthak160]

@AkashKumar7902 Please fix the build fail !

[charankamarapu]

Hey @AkashKumar7902 please fix the error

Error: pkg/service/test/output.go:32:113: cannot use map[string]string{} (value of type map[string]string) as map[string][]string value in struct literal
Error: pkg/service/test/output.go:40:42: cannot use exp (variable of type string) as map[string]string value in assignment
Error: pkg/service/test/output.go:40:47: cannot use act (variable of type string) as map[string]string value in assignment
Error: pkg/service/test/test.go:568:76: too many arguments in call to logDiffs.PushHeaderDiff
    have (string, string, string, map[string][]string)
    want (string, string, map[string][]string)

[charankamarapu]

and Please start working on this PR after merging this PR only, becuase you have built this feature on top of config PR right! So conflicts will be cleared once you merge that 982 PR.

[charankamarapu]

Can you also add the screenshots of testing you have done in PR template?

[AkashKumar7902]

Can you also add the screenshots of testing you have done in PR template?

done!

[Sarthak160]

Hey @AkashKumar7902, Currently if keploy-config.yaml is not present , it logs an error for that . I think that should not be the situation as config file is an optional file not necessity for now .

[AkashKumar7902]

@Sarthak160 hey I have made the changes!

[charankamarapu]

Hey @AkashKumar7902 will review the changes by EOD.

[charankamarapu]

Please update readme with the config feature and create an issue in the docs for this feature update.

[AkashKumar7902]

Hey @charankamarapu

in match.go, we have:

for i := 0; i < expSlice.Len(); i++ {

            isMatchedElement := false
            for j := 0; j < actSlice.Len(); j++ {
                if x, err := jsonMatch(key, expSlice.Index(i).Interface(), actSlice.Index(j).Interface(), noiseMap); err == nil && x {
                    isMatchedElement = true
                    break
                }
            }
            isMatched = isMatchedElement && isMatched

        }

I think this will yield unexpected output in some cases.

Consider a case: expslice = ['a', 'a', 'b', 'c'] actslice = ['a', 'b', 'c', 'c']

Dry run: when i == 0, then actslice will be searched for expslice[i] which is 'a', it will found at j = 0. i will be incremented and actslice will be searched for expslice[i] which is 'a', it will be found again at j = 0. This seems wrong as j = 0 is already matched with i = 0.

[charankamarapu]

will review it by nov 6th EOD.

[charankamarapu]

@AkashKumar7902 Please resolve the merger conflicts.

[charankamarapu]

Hey @AkashKumar7902 we can have connect offline too if you have any doubts are questions.

[charankamarapu]

actslice = ['a', 'b', 'c', 'c']

I have missed this comment. Will analyse this and reply soon.

[charankamarapu]

Regarding slice error which have mentioned. Yes there is an error . At first we gave a thought that order of elements in slice is not important and we have written that logic although that logic is also false for that useCase. Now we are considering order is important so we can use just use one index and one for loop and write the logic . If you are interested you can create a separate issue on it and work it.

[charankamarapu]

Hi @AkashKumar7902 Please inform me once all the changes are done for review.

[AkashKumar7902]

Hi @AkashKumar7902 Please inform me once all the changes are done for review.

Hey @charankamarapu, all the changes have been made !

[charankamarapu]

cool will review it by EOD.

[AkashKumar7902]

do we need to include keploy-config.yaml in .gitignore file ?

[charankamarapu]

In users filebase : No! They may use it in testing step in production or before sending to production

[charankamarapu]

Good work @AkashKumar7902 !