search

apache / drill

Apache Drill is a distributed MPP query layer for self describing data
https://drill.apache.org/
Apache License 2.0
1.93k stars 984 forks source link

REST API jackson exception when join more than one files #2837

Open lenahi opened 9 months ago

lenahi commented 9 months ago

Lab Configuration: Drill query files in Minio bucket "test". drill version: 1.21.1 minio verison: RELEASE.2023-01-25T00-19-54Z

drill storage configuration:

{
  "type": "file",
  "connection": "s3a://test",
  "config": {
    "fs.s3a.endpoint": "http://nginx:9000",
    "fs.s3a.connection.ssl.enabled": "false",
    "fs.s3a.path.style.access": "true",
    "fs.s3a.impl.disable.cache": "true"
  },
  "workspaces": {
    "root": {
      "location": "/",
      "writable": false,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    }
  },
  "formats": {
    "json": {
      "type": "json",
      "extensions": [
        "json"
      ]
    },
    "csv": {
      "type": "text",
      "extensions": [
        "csv"
      ],
      "lineDelimiter": "\n",
      "fieldDelimiter": ",",
      "quote": "\"",
      "escape": "\"",
      "comment": "#",
      "extractHeader": true
    }
  },
  "authMode": "SHARED_USER",
  "credentialsProvider": {
    "credentialsProviderType": "VaultCredentialsProvider",
    "secretPath": "drill/test",
    "propertyNames": {
      "fs.s3a.access.key": "usernameSecret",
      "fs.s3a.secret.key": "passwordSecret"
    }
  },
  "enabled": true
}

In Minio bucket "test" there include two csv files (attached in ticket):

"2021-01-14T08-30-00_sftp.csv"
"2022-02-28T00-00-00_sftp.csv"

2021-01-14T08-30-00_sftp.csv 2022-02-28T00-00-00_sftp.csv

Problem:

Success: Run in drill Portal (PORT: 8047): It is success when run below query in portal, the result is expected.

select * from test.`.`

Run with Drill Rest API with limit, can be much larger than expected rows (sample python code): Result is success with expected.

    headers = {'Content-Type': 'application/json'}
    query = {"queryType": "SQL", "query": "select * from test.`.` limit 2000"}
    r = drill_session.post(drill_url, headers=headers, json=query)
    print(r.json())

Run with Drill Rest API with selecting each column instead of using the asterisk(sample python code): Result is success with expected.

    headers = {'Content-Type': 'application/json'}
    query = {"queryType": "SQL", "query": "select year,industry_code_ANZSIC,industry_name_ANZSIC,rme_size_grp,variable,value,unit from test.`.` "}
    r = drill_session.post(drill_url, headers=headers, json=query)
    print(r.json())

Fail: Run with Drill Rest API with using the asterisk(sample python code): Result failed with exception.

    headers = {'Content-Type': 'application/json'}
    query = {"queryType": "SQL", "query": "select * from test.`.`"}
    r = drill_session.post(drill_url, headers=headers, json=query)
    print(r.json())

Exception in drill:

Certificate was added to keystore
Apache Drill 1.21.1
"In Drill We Trust."
apache drill> 02:29:25.608 [qtp1517434953-28] ERROR o.g.j.server.ServerRuntime$Responder - An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: com.fasterxml.jackson.core.JsonGenerationException: Current context not Array but Object
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
        at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
        at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:635)
        at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373)
        at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:363)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:258)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
        at org.apache.drill.exec.server.rest.header.ResponseHeadersSettingFilter.doFilter(ResponseHeadersSettingFilter.java:71)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.drill.exec.server.rest.CsrfTokenInjectFilter.doFilter(CsrfTokenInjectFilter.java:54)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:571)
        at org.apache.drill.exec.server.rest.auth.DrillHttpSecurityHandlerProvider.handle(DrillHttpSecurityHandlerProvider.java:163)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.lang.Thread.run(Thread.java:750)
Caused by: com.fasterxml.jackson.core.JsonGenerationException: Current context not Array but Object
        at com.fasterxml.jackson.core.JsonGenerator._reportError(JsonGenerator.java:2733)
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeEndArray(UTF8JsonGenerator.java:359)
        at org.apache.drill.exec.vector.complex.fn.BasicJsonOutput.writeEndArray(BasicJsonOutput.java:85)
        at org.apache.drill.exec.server.rest.stream.StreamingHttpConnection.finish(StreamingHttpConnection.java:236)
        at org.apache.drill.exec.server.rest.stream.QueryRunner.sendResults(QueryRunner.java:66)
        at org.apache.drill.exec.server.rest.QueryResources$1.write(QueryResources.java:127)
        at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:55)
        at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:37)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:242)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:227)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
        at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:85)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
        at org.glassfish.jersey.server.mvc.internal.TemplateMethodInterceptor.aroundWriteTo(TemplateMethodInterceptor.java:53)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:61)
        ... 58 common frames omitted

Thanks for support.