Form RequestFieldMatcher can not be constructed from Simulation

matthiastz commented 11 months ago

Description of the bug

With bumping our Dependencies to ("io.specto:hoverfly-java-junit5:0.15.0") ("io.specto:hoverfly-java:0.15.0") one of our test breaks, because the form request is not matched anymore.

Steps to reproduce the issue

Observed result

Hoverfly Java and Hoverfly error messages seen (If none, say none)


io.specto.hoverfly.junit.api.HoverflyClientException: Failed to get simulation: Cannot construct instance of `io.specto.hoverfly.junit.core.model.RequestFieldMatcher$MatcherType`, problem: No enum constant io.specto.hoverfly.junit.core.model.RequestFieldMatcher.MatcherType.FORM
 at [Source: (String)"{"data":{"pairs":[{"request":{"path":[{"matcher":"exact","value":"/oauth2/token"}],"method":[{"matcher":"exact","value":"POST"}],"destination":[{"matcher":"exact","value":"www.my-test.com"}],"body":[{"matcher":"form","value":{"client_id":[{"matcher":"exact","value":"some-id"}],"client_secret":[{"matcher":"exact","value":""}],"code":[{"matcher":"exact","value":"some-code-value"}],"grant_type":[{"matcher":"exact","value":"authorization_code"}],"redirect_uri":[{"m"[truncated 588 chars]; line: 1, column: 211] (through reference chain: io.specto.hoverfly.junit.core.model.Simulation["data"]->io.specto.hoverfly.junit.core.model.HoverflyData["pairs"]->java.util.LinkedHashSet[0]->io.specto.hoverfly.junit.core.model.RequestResponsePair["request"]->io.specto.hoverfly.junit.core.model.Request["body"]->java.util.ArrayList[0]->io.specto.hoverfly.junit.core.model.RequestFieldMatcher["matcher"])

    at io.specto.hoverfly.junit.api.OkHttpHoverflyClient.getSimulation(OkHttpHoverflyClient.java:118)
    at io.specto.hoverfly.junit.core.Hoverfly.verifyAll(Hoverfly.java:507)

If possible, add screenshots to help explain your problem

So I read about the schema change for form and created a json file for my test:

{
  "data": {
    "pairs": [
      {
        "request": {
          "body": [
            {
              "matcher": "form",
              "value": {
                "grant_type": [
                  {
                    "matcher": "exact",
                    "value": "authorization_code"
                  }
                ],
                "code": [
                  {
                    "matcher": "exact",
                    "value": "some-code"
                  }
                ],
                "client_secret": [
                  {
                    "matcher": "exact",
                    "value": ""
                  }
                ],
                "client_id": [
                  {
                    "matcher": "exact",
                    "value": "some-client-id"
                  }
                ],
                "redirect_uri": [
                  {
                    "matcher": "exact",
                    "value": "https://example.org/callback"
                  }
                ]
              }
            }
          ],
          "path": [
            {
              "value": "/oauth2/token",
              "matcher": "exact"
            }
          ],
          "method": [
            {
              "value": "POST",
              "matcher": "exact"
            }
          ],
          "destination": [
            {
              "value": "www.my-test.com",
              "matcher": "exact"
            }
          ]
        },
        "response": {
          "body": "{\"access_token\":\"eyJ0.foo.bar\",\"expires_in\":9999999999,\"refresh_token\":\"eyJ0.foo.bar\",\"id_token\":\"eyJ0.foo.bar\",\"token_type\":\"Bearer\"}",
          "status": 200,
          "headers": {
            "Content-Type": [
              "application/json"
            ]
          },
          "templated": false,
          "encodedBody": false
        }
      }
    ],
    "globalActions": {
      "delays": [],
      "delaysLogNormal": []
    }
  },
  "meta": {
    "timeExported": "2023-10-23T14:13:54Z",
    "schemaVersion": "v5.2",
    "hoverflyVersion": "v1.5.0"
  }
}

my simplified test code (Kotlin and Junit5):

hoverfly.simulate(SimulationSource.classpath("sso/file.json"))

// do something

hoverfly.verifyAll()

before the update I used a slightly different setup (with ("io.specto:hoverfly-java-junit5:0.14.4") and ("io.specto:hoverfly-java:0.14.4")):

14.4. version

``` hoverfly.simulate( dsl( service(baseUrl) .post(startsWith(oauthTokenPath)) .header("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8") .anyBody() .willReturn(success(tokenResponse, "application/json")), ), ) // do something hoverfly.verify( service(baseUrl) .post(startsWith(oauthTokenPath)) .header("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8") .anyQueryParams() // need to do regex based matching since those are not "real" query params: https://github.com/SpectoLabs/hoverfly/issues/903 .body( RequestFieldMatcher.newRegexMatcher( "code=$codeParam.*grant_type=authorization_code.*client_secret=.*redirect_uri=${ URLEncoder.encode( redirectUri, "UTF-8", ) }.*client_id=$fakeClientId", ), ), ) ```

Expected result

Additional relevant information

the underlying http request/framework that is responsible to construct and send the HTTP request (which is observed in this tests) uses oauth2-oidc-sdk-5.45. Complete project is Kotlin/Java with Junit5.

tommysitu commented 9 months ago

I've raised a PR which should fix this issue: https://github.com/SpectoLabs/hoverfly-java/pull/292

tommysitu commented 8 months ago

it's fixed in 0.16.0

matthiastz commented 8 months ago

Hey @tommysitu 👋 , I tried to update the test in our Repo today to make use of the new form matcher (body).

Unfortunately, it still does not work.

My simplified test code is still the same (Kotlin and Junit5):

hoverfly.simulate(SimulationSource.classpath("sso/file.json"))

// do something

hoverfly.verifyAll()

I am 100% certain that the issue is related to the body verification (if I leave out the body matcher my test works fine - but thats obviously not what I need), unfortunately I do not know how to debug this issue further, as it seem to happen in the underlying Go service. Error:

WARN  2024-01-08T16:04:49,253 io.specto.hoverfly.junit.api.HoverflyClient - Failed to search journal: Unexpected response (code=500, message=PANIC: interface conversion: interface {} is nil, not func(interface {}, string) bool
goroutine 19 [running]:
github.com/codegangsta/negroni.(*Recovery).ServeHTTP.func1()
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/recovery.go:34 +0x94
panic({0x1052b5b80, 0x1400046cff0})
    /usr/local/go/src/runtime/panic.go:838 +0x204
github.com/SpectoLabs/hoverfly/core/matching.isMatching({{0x140004470f4, 0x4}, {0x1052aa480, 0x1400046cf30}, 0x0, 0x0}, {0x0?, 0x1400007e000?})
    /home/circleci/hoverfly/core/matching/field_matcher.go:43 +0x1d8
github.com/SpectoLabs/hoverfly/core/matching.FieldMatcher({0x1400046cfc0?, 0x1, 0x1052cae00?}, {0x0, 0x0})
    /home/circleci/hoverfly/core/matching/field_matcher.go:19 +0xc8
github.com/SpectoLabs/hoverfly/core/journal.(*Journal).GetFilteredEntries(0x140002ed860, {0x105263f00?})
    /home/circleci/hoverfly/core/journal/journal.go:191 +0x354
github.com/SpectoLabs/hoverfly/core/handlers/v2.(*JournalHandler).Post(0x14000267c10, {0x12c9c1b58, 0x14000402900}, 0x140003f4c40?, 0x0?)
    /home/circleci/hoverfly/core/handlers/v2/journal_handler.go:86 +0x80
github.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0x14000425598?, {0x12c9c1b58?, 0x14000402900?}, 0x0?, 0x0?)
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/negroni.go:24 +0x40
github.com/codegangsta/negroni.middleware.ServeHTTP({{0x105366a98?, 0x140003fe150?}, 0x1400009f9b0?}, {0x12c9c1b58, 0x14000402900}, 0xa3?)
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/negroni.go:33 +0xc4
github.com/SpectoLabs/hoverfly/core/handlers.(*AuthHandler).RequireTokenAuthentication(0x105a805b8?, {0x12c9c1b58?, 0x14000402900?}, 0x140003f4c00?, 0x0?)
    /home/circleci/hoverfly/core/handlers/auth_handler.go:50 +0x158
github.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0x1400006a480?, {0x12c9c1b58?, 0x14000402900?}, 0x1058d5ae8?, 0x14000425698?)
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/negroni.go:24 +0x40
github.com/codegangsta/negroni.middleware.ServeHTTP({{0x105366a98?, 0x140003fe140?}, 0x1400009f998?}, {0x12c9c1b58, 0x14000402900}, 0x104c36754?)
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/negroni.go:33 +0xc4
github.com/codegangsta/negroni.(*Negroni).ServeHTTP(0x140003f6b10, {0x12c9c1b58?, 0x140004028c0}, 0x140003ee168?)
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/negroni.go:73 +0xd0
github.com/go-zoo/bone.(*Route).parse(0x140003fc9a0, {0x12c9c1b58, 0x140004028c0}, 0x14000310500)
    /home/circleci/hoverfly/vendor/github.com/go-zoo/bone/route.go:171 +0x1b8
github.com/go-zoo/bone.(*Mux).parse(0x1400026bbc0, {0x12c9c1b58, 0x140004028c0}, 0x14000310500)
    /home/circleci/hoverfly/vendor/github.com/go-zoo/bone/helper.go:23 +0x9c
github.com/go-zoo/bone.(*Mux).DefaultServe(0x14000404320?, {0x12c9c1b58, 0x140004028c0}, 0x104b53f00?)
    /home/circleci/hoverfly/vendor/github.com/go-zoo/bone/bone.go:71 +0x30
github.com/go-zoo/bone.(*Mux).ServeHTTP(0x105a805b8?, {0x12c9c1b58?, 0x140004028c0?}, 0x140003f4be0?)
    /home/circleci/hoverfly/vendor/github.com/go-zoo/bone/bone.go:90 +0xb0
github.com/codegangsta/negroni.Wrap.func1({0x12c9c1b58, 0x140004028c0}, 0x1052bd601?, 0x140003f4be0)
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/negroni.go:41 +0x58
github.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0x140000b1560?, {0x12c9c1b58?, 0x140004028c0?}, 0x1049c0a50?, 0x5000?)
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/negroni.go:24 +0x40
github.com/codegangsta/negroni.middleware.ServeHTTP({{0x105366a98?, 0x1400009fe60?}, 0x1400009fe90?}, {0x12c9c1b58, 0x140004028c0}, 0x12c9998a8?)
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/negroni.go:33 +0xc4
github.com/codegangsta/negroni.(*Recovery).ServeHTTP(0x1400006a480?, {0x12c9c1b58?, 0x140004028c0?}, 0x0?, 0x14000425a08?)
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/recovery.go:45 +0x78
github.com/codegangsta/negroni.middleware.ServeHTTP({{0x1053654b8?, 0x1400009fe18?}, 0x1400009fe78?}, {0x12c9c1b58, 0x140004028c0}, 0x104a06c9c?)
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/negroni.go:33 +0xc4
github.com/codegangsta/negroni.(*Negroni).ServeHTTP(0x140003f6ed0, {0x10536ab78?, 0x140004a81c0}, 0x1049c0a50?)
    /home/circleci/hoverfly/vendor/github.com/codegangsta/negroni/negroni.go:73 +0xd0
net/http.serverHandler.ServeHTTP({0x1053689d8?}, {0x10536ab78, 0x140004a81c0}, 0x14000310500)
    /usr/local/go/src/net/http/server.go:2916 +0x3fc
net/http.(*conn).serve(0x140002f1f40, {0x10536b280, 0x140003f7110})
    /usr/local/go/src/net/http/server.go:1966 +0x56c
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:3071 +0x450

I think the most important lines here are:

field_matcher.go:43 +0x1d8
field_matcher.go:19 +0xc8
journal.go:191

field_matcher.go:43 then panics, since it somehow tries to convert nil to func(interface {}, string) bool which is not possible. I have no clue where the nil comes from. https://github.com/SpectoLabs/hoverfly/blob/d8b55cae005caec655a6b9440dd878d8b6e47af7/core/matching/field_matcher.go#L43

my simplified json schema, with which you hopefully can reproduce the issue:

{
  "data": {
    "pairs": [
      {
        "request": {
          "method": [
            {
              "value": "POST",
              "matcher": "exact"
            }
          ],
          "body": [
            {
              "matcher": "form",
              "value": {
                "grant_type": [
                  {
                    "matcher": "exact",
                    "value": "authorization_code"
                  }
                ]
              }
            }
          ]
        },
        "response": {
          "body": "{\"access_token\":\"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.foo.bar\",\"expires_in\":3600,\"refresh_token\":\"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.foo.bar\",\"id_token\":\"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.foo.bar\",\"token_type\":\"Bearer\"}",
          "status": 200,
          "headers": {
            "Content-Type": [
              "application/json"
            ]
          }
        }
      }
    ],
    "globalActions": {
      "delays": []
    }
  },
  "meta": {
    "schemaVersion": "v5.2"
  }
}

Is there any way with hoverfly + junit to somehow debug what is going on in the go service while running? Any extended logging, e.g. on the current field/currentMatcher? This would help.

Ty!

tommysitu commented 8 months ago

Hi @matthiastz thanks for the info. I will take a look as soon as I can!

tommysitu commented 8 months ago

debugging time: https://github.com/SpectoLabs/hoverfly-java/pull/294 😬

tommysitu commented 8 months ago

If you wanna debug hoverfly, you will need to download the go project, run it in debug mode in your IDE (I use Intellij for both java and go), and then configure the hoverfly rule in your junit test to point to the local instance on the default ports like this:

@ClassRule
    public static HoverflyRule hoverflyRule = HoverflyRule.inSimulationMode(simulationSource, 
HoverflyConfig.remoteConfigs().host("localhost").proxyPort(8500).adminPort(8888));

so when you run your junit test, it will pause the execution when it hits the breakpoint on the golang project (which is hoverfly running in debug mode).

tommysitu commented 8 months ago

I've raised a PR on hoverfly to fix this issue: https://github.com/SpectoLabs/hoverfly/pull/1110

tommysitu commented 8 months ago

fixed in v0.16.1

matthiastz commented 8 months ago

Hey @tommysitu - thx so much for your support and the hint to debug hoverfly!! Can confirm that its working now as expected :)

tommysitu commented 8 months ago

🎉

SpectoLabs / hoverfly-java