Structural Type as Map Key cannot be rendered by CLI

[wenleix]

Presto has weird behavior when I use histogram in a wrong way:

presto> select histogram(ARRAY[1, 2, 3]);
Query is gone (server restarted?)

The correct way:

presto> select histogram(x) from UNNEST(ARRAY[1, 2, 3]) as t(x);
      _col0
-----------------
 {1=1, 2=1, 3=1}

However, it shouldn't response "Query is gone", which is very confusing?

[martint]

That can happen when the client can't deserialize the response from the server. It's possible that the result is not being encoded correctly.

This is the detailed error:

java.lang.RuntimeException: Error fetching next at http://localhost:8080/v1/statement/20170309_072058_00007_wchv4/1 returned an invalid response: JsonResponse{statusCode=200, statusMessage=OK, headers={Vary=[Accept-Encoding, User-Agent], Content-Length=[1681], Date=[Thu, 09 Mar 2017 07:20:58 GMT], X-Content-Type-Options=[nosniff], Content-Type=[application/json]}, hasValue=false, value=null} [Error: {"id":"20170309_072058_00007_wchv4","infoUri":"http://localhost:8080/query.html?20170309_072058_00007_wchv4","partialCancelUri":"http://127.0.0.1:8080/v1/stage/20170309_072058_00007_wchv4.0","nextUri":"http://localhost:8080/v1/statement/20170309_072058_00007_wchv4/2","columns":[{"name":"_col0","type":"map(array(integer),bigint)","typeSignature":{"rawType":"map","typeArguments":[{"rawType":"array","typeArguments":[{"rawType":"integer","typeArguments":[],"literalArguments":[],"arguments":[]}],"literalArguments":[],"arguments":[{"kind":"TYPE_SIGNATURE","value":{"rawType":"integer","typeArguments":[],"literalArguments":[],"arguments":[]}}]},{"rawType":"bigint","typeArguments":[],"literalArguments":[],"arguments":[]}],"literalArguments":[],"arguments":[{"kind":"TYPE_SIGNATURE","value":{"rawType":"array","typeArguments":[{"rawType":"integer","typeArguments":[],"literalArguments":[],"arguments":[]}],"literalArguments":[],"arguments":[{"kind":"TYPE_SIGNATURE","value":{"rawType":"integer","typeArguments":[],"literalArguments":[],"arguments":[]}}]}},{"kind":"TYPE_SIGNATURE","value":{"rawType":"bigint","typeArguments":[],"literalArguments":[],"arguments":[]}}]}}],"data":[[{"[1, 2, 3]":1}]],"stats":{"state":"RUNNING","queued":false,"scheduled":true,"nodes":1,"totalSplits":6,"queuedSplits":6,"runningSplits":0,"completedSplits":0,"userTimeMillis":0,"cpuTimeMillis":0,"wallTimeMillis":0,"processedRows":0,"processedBytes":0,"rootStage":{"stageId":"0","state":"RUNNING","done":false,"nodes":1,"totalSplits":6,"queuedSplits":6,"runningSplits":0,"completedSplits":0,"userTimeMillis":0,"cpuTimeMillis":0,"wallTimeMillis":0,"processedRows":0,"processedBytes":0,"subStages":[]}}}
]
    at com.facebook.presto.client.StatementClient.requestFailedException(StatementClient.java:349)
    at com.facebook.presto.client.StatementClient.advance(StatementClient.java:299)
    at com.facebook.presto.cli.StatusPrinter.printInitialStatusUpdates(StatusPrinter.java:122)
    at com.facebook.presto.cli.Query.renderQueryOutput(Query.java:123)
    at com.facebook.presto.cli.Query.renderOutput(Query.java:107)
    at com.facebook.presto.cli.Console.process(Console.java:318)
    at com.facebook.presto.cli.Console.runConsole(Console.java:258)
    at com.facebook.presto.cli.Console.run(Console.java:144)
    at com.facebook.presto.cli.Presto.main(Presto.java:32)
Caused by: java.lang.IllegalArgumentException: Unable to create class com.facebook.presto.client.QueryResults from JSON response:
[{"id":"20170309_072058_00007_wchv4","infoUri":"http://localhost:8080/query.html?20170309_072058_00007_wchv4","partialCancelUri":"http://127.0.0.1:8080/v1/stage/20170309_072058_00007_wchv4.0","nextUri":"http://localhost:8080/v1/statement/20170309_072058_00007_wchv4/2","columns":[{"name":"_col0","type":"map(array(integer),bigint)","typeSignature":{"rawType":"map","typeArguments":[{"rawType":"array","typeArguments":[{"rawType":"integer","typeArguments":[],"literalArguments":[],"arguments":[]}],"literalArguments":[],"arguments":[{"kind":"TYPE_SIGNATURE","value":{"rawType":"integer","typeArguments":[],"literalArguments":[],"arguments":[]}}]},{"rawType":"bigint","typeArguments":[],"literalArguments":[],"arguments":[]}],"literalArguments":[],"arguments":[{"kind":"TYPE_SIGNATURE","value":{"rawType":"array","typeArguments":[{"rawType":"integer","typeArguments":[],"literalArguments":[],"arguments":[]}],"literalArguments":[],"arguments":[{"kind":"TYPE_SIGNATURE","value":{"rawType":"integer","typeArguments":[],"literalArguments":[],"arguments":[]}}]}},{"kind":"TYPE_SIGNATURE","value":{"rawType":"bigint","typeArguments":[],"literalArguments":[],"arguments":[]}}]}}],"data":[[{"[1, 2, 3]":1}]],"stats":{"state":"RUNNING","queued":false,"scheduled":true,"nodes":1,"totalSplits":6,"queuedSplits":6,"runningSplits":0,"completedSplits":0,"userTimeMillis":0,"cpuTimeMillis":0,"wallTimeMillis":0,"processedRows":0,"processedBytes":0,"rootStage":{"stageId":"0","state":"RUNNING","done":false,"nodes":1,"totalSplits":6,"queuedSplits":6,"runningSplits":0,"completedSplits":0,"userTimeMillis":0,"cpuTimeMillis":0,"wallTimeMillis":0,"processedRows":0,"processedBytes":0,"subStages":[]}}}
]
    at io.airlift.http.client.FullJsonResponseHandler$JsonResponse.<init>(FullJsonResponseHandler.java:121)
    at io.airlift.http.client.FullJsonResponseHandler.handle(FullJsonResponseHandler.java:68)
    at io.airlift.http.client.FullJsonResponseHandler.handle(FullJsonResponseHandler.java:37)
    at io.airlift.http.client.jetty.JettyHttpClient.execute(JettyHttpClient.java:383)
    at com.facebook.presto.client.StatementClient.advance(StatementClient.java:286)
    ... 7 more
Caused by: java.lang.IllegalArgumentException: Invalid JSON bytes for [simple type, class com.facebook.presto.client.QueryResults]
    at io.airlift.json.JsonCodec.fromJson(JsonCodec.java:201)
    at io.airlift.http.client.FullJsonResponseHandler$JsonResponse.<init>(FullJsonResponseHandler.java:118)
    ... 11 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of com.facebook.presto.client.QueryResults, problem: Cannot cast java.lang.String to java.util.List
 at [Source: [B@5911e990; line: 1, column: 1680]
    at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:268)
    at com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:1405)
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:468)
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:487)
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:276)
    at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:224)
    at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:135)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:471)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1194)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:314)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3789)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2950)
    at io.airlift.json.JsonCodec.fromJson(JsonCodec.java:198)
    ... 12 more
Caused by: java.lang.ClassCastException: Cannot cast java.lang.String to java.util.List
    at java.lang.Class.cast(Class.java:3369)
    at com.facebook.presto.client.QueryResults.fixValue(QueryResults.java:239)
    at com.facebook.presto.client.QueryResults.fixValue(QueryResults.java:249)
    at com.facebook.presto.client.QueryResults.fixData(QueryResults.java:221)
    at com.facebook.presto.client.QueryResults.<init>(QueryResults.java:93)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:124)
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:274)
    ... 21 more

[haozhun]

This is a known issue in Presto. Serialization for query results of map type was not carefully considered when map type was introduced. JSON maps can only have varchar keys. Jackson uses toString for map keys during serialization. There are some hacks in client library to reverse engineer certain types. But structural types in map keys are not handled.

[GaryHo34]

Hi, i am interested in this issue, can someone give me some advice where to start with?

[tdcmeehan]

@GaryHo34 please start with the stack trace.

To simply reproduce, follow the README on how to build the project.

Then run in IntelliJ TpchQueryRunner.

Once it is running, execute the following from the root of your project directory: ./presto-cli/target/presto-cli-*-SNAPSHOT-executable.jar --debug --execute "select histogram(ARRAY[1, 2, 3]);"

You'll observe a stack trace, and from there you can read the source code and deduce how to fix this issue.

[GaryHo34]

Hello,

I have a few questions.

What is the expected behavior of the following query: select histogram(ARRAY[1,2,3]);?

{1=1, 2=1, 3=1}
{[1,2,3] = 1}

It seems that this issue might arise when the client calls the fixValue function in the FixJsonDataUtils.java file. Presto attempts to parse a string "[1,2,3]" into a list, which leads to an error.

Should I modify the fixValue function to produce the second result mentioned above? Or should I focus on fixing the part of the code where Jackson parses the key of the map into a string?

Update:

I modified the fixValue function like this:

  if (signature.getBase().equals(ARRAY)) {
      List<Object> fixedValue = new ArrayList<>();
      // if the object is a string, parse it again to get the object
      if(value.getClass()==String.class)
          value = jsonCodec(Object.class).fromJson(value.toString());
      for (Object object : List.class.cast(value)) {
          fixedValue.add(fixValue(signature.getTypeParametersAsTypeSignatures().get(0), object));
      }
      return fixedValue;
  }

The histogram works for array type after this change:

Am I doing it right?