Deserialization of suggest responses not working

[royteeuwen]

Elasticsearch version 7.7.0

Plugins installed: /

JVM version (java -version): 11.0.6

OS version (uname -a if on a Unix-like system): macOS Catalina / Redhat 7

Description of the problem including expected versus actual behavior:

I am trying to create unit tests for classes that use the java high level rest client. To make this easier I am mocking the response that is given by elasticsearch so that all the business logic that happens can then be tested without an actual running elasticsearch. The problem is that this mocking is not working.

A sample query to Elasticsearch that we are testing:

{
  "suggest": {
    "suggestion": {
      "prefix": "all-possib",
      "completion": {
        "field": "pageTitle.completion",
        "skip_duplicates": true,
        "contexts": {
          "siteName": [
            {
              "context": "nl",
              "boost": 1,
              "prefix": false
            }
          ]
        }
      }
    }
  }
}

A sample rest response of Elasticsearch:

{
  "took": 7,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": 0.0,
    "hits": []
  },
  "suggest": {
    "suggestion": [
      {
        "text": "all-possib",
        "offset": 0,
        "length": 10,
        "options": [
          {
            "text": "All possible",
            "_index": "aem_pages_nl",
            "_type": "page",
            "_id": "allpossible",
            "_score": 1.0,
            "_source": {
              "pageTitle": "All possible",
              "pageDescription": "Nullam id dolor id nibh ultricies vehicula ut id elit.",
              "path": "/content/my-site/nl/testpage/allpossible",
              "SEOTitle": "All possible",
              "SEODescription": "Nullam id dolor id nibh ultricies vehicula ut id elit.",
              "language": "nl",
              "offTimeString": "",
              "onTimeString": "",
              "lastModifiedRFC822": "Tue, 10 Mar 2020 11:06:59 +0100",
              "tagsString": "Gezin",
              "tagIds": [
                "top-navigation:gezin"
              ],
              "siteName": "siteA",
              "resourceType": "my\/components\/page\/all-possible-page",
              "shortResourceType": "all-possible-page",
              "shortDescription": "Nullam id dolor id nibh ultricies vehicula ut id elit.",
              "author": "",
              "shortTitle": "All possible",
              "extraPageInfo": "",
              "boosting": ""
            },
            "contexts": {
              "siteName": [
                "siteA"
              ]
            }
          }
        ]
      }
    ]
  }
}

Parsing the SearchResponse from the JSON string:

public static SearchResponse getSearchResponse(String json) {
        try {
            NamedXContentRegistry registry = new NamedXContentRegistry(getDefaultNamedXContents());
            XContentParser parser = JsonXContent.jsonXContent.createParser(registry, DeprecationHandler.IGNORE_DEPRECATIONS, json);
            return SearchResponse.fromXContent(parser);
        } catch (IOException e) {
            System.out.println("exception " + e);
        }
        return null;
    }

    public static List<NamedXContentRegistry.Entry> getDefaultNamedXContents() {
        Map<String, ContextParser<Object, ? extends Aggregation>> map = new HashMap<>();
        map.put(TopHitsAggregationBuilder.NAME, (p, c) -> ParsedTopHits.fromXContent(p, (String) c));
        map.put(StringTerms.NAME, (p, c) -> ParsedStringTerms.fromXContent(p, (String) c));
        List<NamedXContentRegistry.Entry> entries = map.entrySet().stream()
                .map(entry -> new NamedXContentRegistry.Entry(Aggregation.class, new ParseField(entry.getKey()), entry.getValue()))
                .collect(Collectors.toList());
        return entries;
    }

Exception being thrown:

ParsingException[Could not parse suggestion keyed as [suggestion]
]
    at org.elasticsearch.search.suggest.Suggest.fromXContent(Suggest.java:197)
    at org.elasticsearch.action.search.SearchResponse.innerFromXContent(SearchResponse.java:308)
    at org.elasticsearch.action.search.SearchResponse.fromXContent(SearchResponse.java:265)
    at my.application.SearchResponseUtil.getSearchResponse(SearchResponseUtil.java:29)

What would be a correct / good way to make this work? All our integration tests with a suggest in json response are failing because of this.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Java High Level REST Client)

[javanna]

heya, at first glance the response you are trying to parse cannot be parsed by the high level REST client as it requires the typed_keys parameter set to true. That way the response will contain the same suggestion, but its name will be composed of the name of the suggestion as well as its type separated by #. This way the parser knows which class to create for that type of suggestion.

[royteeuwen]

Hey @javanna , how is it possible then that I actually got that response first by using the high level client to get it? Thats how I created the mock json to test, by first using the real logic and saving the SearchResponse.toString() in a test resource file

[javanna]

toString calls toXContent. You should call toXContent manually and provide the typed_keys param to it. Even if you are printing a response obtained through the client, you still need to print it out the same way that the client expects it or it cannot be parsed. Otherwise the client does not know which class to create.

On Fri, May 29, 2020, 21:54 Roy Teeuwen notifications@github.com wrote:

Hey @javanna https://github.com/javanna , how is it possible then that I actually got that response first by using the high level client to get it? Thats how I created the mock json to test, by first using the real logic and saving the SearchResponse.toString() in a test resource file

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/elastic/elasticsearch/issues/57297#issuecomment-636158926, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGLHTFMOFEEIEZ4KHCCDILRUAHFPANCNFSM4NNFPKLQ .

[javanna]

hi @royteeuwen I hope you don't mind if I close this issue given that it's not a bug.

[royteeuwen]

Hey @javanna, sorry I have two days off so couldnt test it out yet. I still dont see how to call the toxcontent manually so that its parseable afterwards. Can you provide an example?

[javanna]

@royteeuwen an example is here: https://github.com/elastic/elasticsearch/blob/master/server/src/test/java/org/elasticsearch/action/search/SearchResponseTests.java#L157 .