Duplicated query params

[mlundela]

Upgrading from 0.10.3 to 0.11.0 is causing tests to fail, and from what I can understand it is because the request now has duplicates, and no longer matches my simulation files.

09:51:02.697 [Thread-2] ERROR hoverfly - There was an error when matching error=Could not find a match for request, create or record a valid matcher first!

The following request was made, but was not matched by Hoverfly:

{
    "Path": "/",
    "Method": "POST",
    ...,
    "Query": {
        "foo": [
            "bar"
        ],
        "foo": [
            "bar"
        ]
    },...

[tommysitu]

@mlundela I am trying to understand your issue. Did you send the request with query contains duplicate key-value? This is usually not possible with HTTP clients.

Could you please give me an example on how to reproduce this issue? Thanks!

[mlundela]

No, all I did was to upgrade from 0.10.3 to 0.11.0.

Deleting the simulation.json, and running the tests again in capture mode is a workaround.

[cazpr]

In my case, when the request doesn't match the captured request, Hoverfly seems to duplicate each query parameter that contains capitals when outputting the request that was made.

An example (java 8, macOs Mojave, io.specto:hoverfly-java:0.11.0):

import io.specto.hoverfly.junit.core.Hoverfly;
import io.specto.hoverfly.junit.core.HoverflyMode;
import io.specto.hoverfly.junit.core.SimulationSource;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import static io.specto.hoverfly.junit.core.HoverflyConfig.localConfigs;

public class HoverflyDuplicateQueryParams {
    public static void main(String[] args) throws IOException {
        try (Hoverfly hoverfly = new Hoverfly(localConfigs(), HoverflyMode.SIMULATE)) {
            hoverfly.start();
            hoverfly.simulate(SimulationSource.classpath("simulation.json"));

            try (InputStream inputStream = new URL("https://website.com/mismatch?CaseSensitive=something").openStream()) {
                inputStream.skip(Long.MAX_VALUE);
            }
        }
    }
}

The example above will fail because the path doesn't match, which is reported correctly but when you look at the query param output of the captured request, you'll notice that the 'CaseSensitive' parameter is outputted twice: once with it's original name and once with the lowercase name.

Not sure if this is intentional or not but I think this can be quite confusing, especially when the query parameters sections doesn't match.

2018-10-19 21:29:26  INFO  TempFileManager: Selecting the following binary based on the current operating system: hoverfly_OSX_amd64
2018-10-19 21:29:26  INFO  TempFileManager: Storing binary in temporary directory /var/folders/j6/18ncf42s05j9pp168jpg6c944r52nj/T/hoverfly.4592300079021866205/hoverfly_OSX_amd64
2018-10-19 21:29:26  INFO  Hoverfly: Executing binary at /var/folders/j6/18ncf42s05j9pp168jpg6c944r52nj/T/hoverfly.4592300079021866205/hoverfly_OSX_amd64
2018-10-19 21:29:26  INFO  hoverfly: Default proxy port has been overwritten port=62495
2018-10-19 21:29:26  INFO  hoverfly: Default admin port has been overwritten port=62496
2018-10-19 21:29:26  INFO  hoverfly: Using memory backend 
2018-10-19 21:29:26  INFO  hoverfly: Proxy prepared... Destination=. Mode=simulate ProxyPort=62495
2018-10-19 21:29:26  INFO  hoverfly: current proxy configuration destination=. mode=simulate port=62495
2018-10-19 21:29:26  INFO  hoverfly: serving proxy 
2018-10-19 21:29:26  INFO  hoverfly: Admin interface is starting... AdminPort=62496
2018-10-19 21:29:26  INFO  hoverfly: Mode has been changed mode=simulate
2018-10-19 21:29:26  INFO  ProxyConfigurer: Setting proxy host to localhost
2018-10-19 21:29:26  INFO  ProxyConfigurer: Setting proxy proxyPort to 62495
2018-10-19 21:29:26  INFO  Hoverfly: Importing simulation data to Hoverfly
2018-10-19 21:29:26  INFO  hoverfly: payloads imported failed=0 successful=1 total=1
2018-10-19 21:29:26  WARN  hoverfly: Failed to find matching request from simulation destination=website.com error=No match found method=GET path=/mismatch query={CaseSensitive=[something], casesensitive=[something]}
2018-10-19 21:29:26  INFO  Hoverfly: Destroying hoverfly process
2018-10-19 21:29:26  ERROR hoverfly: There was an error when matching error=Could not find a match for request, create or record a valid matcher first!

The following request was made, but was not matched by Hoverfly:

{
    "Path": "/mismatch",
    "Method": "GET",
    "Destination": "website.com",
    "Scheme": "https",
    "Query": {
        "CaseSensitive": [
            "something"
        ],
        "casesensitive": [
            "something"
        ]
    },
    "Body": "",
    "Headers": {
        "Accept": [
            "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"
        ],
        "Connection": [
            "keep-alive"
        ],
        "User-Agent": [
            "Java/1.8.0_152"
        ]
    }
}

Whilst Hoverfly has the following state:

{}

The matcher which came closest was:

{
    "path": [
        {
            "matcher": "exact",
            "value": "/query"
        }
    ],
    "method": [
        {
            "matcher": "exact",
            "value": "GET"
        }
    ],
    "destination": [
        {
            "matcher": "exact",
            "value": "website.com"
        }
    ],
    "scheme": [
        {
            "matcher": "exact",
            "value": "https"
        }
    ],
    "query": {
        "CaseSensitive": [
            {
                "matcher": "exact",
                "value": "something"
            }
        ]
    }
}

But it did not match on the following fields:

[path]

Which if hit would have given the following response:

{
    "status": 200,
    "body": "\u003chtml\u003e\u003c/html\u003e",
    "encodedBody": false,
    "headers": {
        "Content-Type": [
            "text/html"
        ]
    },
    "templated": false
} mode=simulate request={body=, destination=website.com, headers={Accept=[text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2], Connection=[keep-alive], User-Agent=[Java/1.8.0_152]}, method=GET, path=/mismatch, query={CaseSensitive=[something], casesensitive=[something]}, scheme=https} response={error=nil response}
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 502 for URL: https://website.com/mismatch?CaseSensitive=something

simulation.json {"data":{"pairs":[{"request":{"path":[{"matcher":"exact","value":"/query"}],"method":[{"matcher":"exact","value":"GET"}],"destination":[{"matcher":"exact","value":"website.com"}],"scheme":[{"matcher":"exact","value":"https"}],"query":{"CaseSensitive":[{"matcher":"exact","value":"something"}]}},"response":{"status":200,"body":"<html></html>","encodedBody":false,"templated":false,"headers":{"Content-Type":["text/html"]}}}],"globalActions":{"delays":[]}},"meta":{"schemaVersion":"v5","hoverflyVersion":"v0.17.1","timeExported":"2018-10-19T20:31:23+02:00"}}

[tommysitu]

@cazpr Those are really useful info. I can now locate the bug here https://github.com/SpectoLabs/hoverfly/blob/master/core/matching/query_matching.go#L35 where it duplicates the query when doing the matching.

[tommysitu]

fixed in version 0.11.1