search

devinrsmith / deephaven-parquet-viewer

A browser-based Parquet file viewer
41 stars 2 forks source link

No such object: deephaven-parquet-viewer #9

Open alvaro1728 opened 1 year ago

alvaro1728 commented 1 year ago

Just cloned this on my Mac and started Docker. Then I ran it like this:

./deephaven-parquet-viewer.sh some-file.parquet

It installed the docker image and then it started showing errors like this:

Starting Deephaven Parquet Viewer......Error: No such object: deephaven-parquet-viewer
.Error: No such object: deephaven-parquet-viewer
.Error: No such object: deephaven-parquet-viewer
.Error: No such object: deephaven-parquet-viewer
.Error: No such object: deephaven-parquet-viewer

The "No such object" errors kept coming, about once a second.

Please help, thanks.

devinrsmith commented 1 year ago

@alvaro1728, it looks like the container failed to start. It might be that there was a port conflict, or some other issue.

Can you run:

docker run \
    --rm \
    --name deephaven-parquet-viewer \
    -p "10000:10000" \
    --mount type=bind,source=/absolute/path/to/some-file.parquet,target=/file.parquet,readonly \
    ghcr.io/devinrsmith/deephaven-parquet-viewer:latest

and paste any resulting error messages here?

devinrsmith commented 1 year ago

I've created a follow-up issue to improve the container startup error handling, https://github.com/devinrsmith/deephaven-parquet-viewer/issues/10

luisyng commented 1 year ago

Hi!

I also got the same error. I've run the command you posted and this is the result: 

Initiating shutdown due to: Uncaught exception in thread main
java.lang.RuntimeException: Error encountered at line 3: parquet_table = ParquetTools.readTable('/file.parquet')
    at io.deephaven.engine.util.GroovyDeephavenSession.maybeRewriteStackTrace(GroovyDeephavenSession.java:361)
    at io.deephaven.engine.util.GroovyDeephavenSession.wrapAndRewriteStackTrace(GroovyDeephavenSession.java:339)
    at io.deephaven.engine.util.GroovyDeephavenSession.evaluate(GroovyDeephavenSession.java:330)
    at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:147)
    at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:149)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:160)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:148)
    at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:147)
    at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:218)
    at io.deephaven.server.appmode.ApplicationFactory.evaluateScripts(ApplicationFactory.java:98)
    at io.deephaven.server.appmode.ApplicationFactory.lambda$visit$0(ApplicationFactory.java:66)
    at io.deephaven.appmode.ApplicationContext.runUnderContext(ApplicationContext.java:36)
    at io.deephaven.server.appmode.ApplicationFactory.visit(ApplicationFactory.java:66)
    at io.deephaven.appmode.ScriptApplication.walk(ScriptApplication.java:49)
    at io.deephaven.server.appmode.ApplicationFactory.create(ApplicationFactory.java:27)
    at io.deephaven.server.appmode.ApplicationInjector.loadApplication(ApplicationInjector.java:90)
    at io.deephaven.server.appmode.ApplicationInjector.run(ApplicationInjector.java:78)
    at io.deephaven.server.runner.DeephavenApiServer.run(DeephavenApiServer.java:167)
    at io.deephaven.server.jetty.JettyMain.main(JettyMain.java:24)
Caused by: java.lang.UnsupportedOperationException: Encountered unsupported multi-column field meta: found columns meta/versionId and meta/lastUpdated
    at io.deephaven.parquet.table.ParquetSchemaReader.lambda$readParquetSchema$1(ParquetSchemaReader.java:173)
    at java.base/java.util.HashMap.compute(HashMap.java:1316)
    at io.deephaven.parquet.table.ParquetSchemaReader.readParquetSchema(ParquetSchemaReader.java:168)
    at io.deephaven.parquet.table.ParquetTools.convertSchema(ParquetTools.java:842)
    at io.deephaven.parquet.table.ParquetTools.readTableInternal(ParquetTools.java:551)
    at io.deephaven.parquet.table.ParquetTools.readTable(ParquetTools.java:80)
    at io.deephaven.parquet.table.ParquetTools$readTable.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
    at io.deephaven.dynamic._opt_deephaven_config_app_d_parquet_viewer_groovy_6.run(_opt_deephaven_config_app_d_parquet_viewer_groovy_6.groovy:4)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:427)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:461)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:436)
    at io.deephaven.engine.util.GroovyDeephavenSession.evaluateCommand(GroovyDeephavenSession.java:302)
    at io.deephaven.engine.util.GroovyDeephavenSession.lambda$evaluate$0(GroovyDeephavenSession.java:325)
    at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
    at io.deephaven.engine.util.GroovyDeephavenSession.evaluate(GroovyDeephavenSession.java:325)
    ... 16 more
2023-11-07T13:56:05.424Z | Thread-0             |  WARN | d.u.p.ShutdownManagerImpl | Initiating shutdown processing
2023-11-07T13:56:05.425Z | Thread-0             |  WARN | d.u.p.ShutdownManagerImpl | Starting to invoke FIRST shutdown tasks
2023-11-07T13:56:05.427Z | Thread-0             |  WARN | d.u.p.ShutdownManagerImpl | Done invoking FIRST shutdown tasks
2023-11-07T13:56:05.427Z | Thread-0             |  WARN | d.u.p.ShutdownManagerImpl | Starting to invoke MIDDLE shutdown tasks
2023-11-07T13:56:05.427Z | Thread-0             |  WARN | d.u.p.ShutdownManagerImpl | Done invoking MIDDLE shutdown tasks
2023-11-07T13:56:05.427Z | Thread-0             |  WARN | d.u.p.ShutdownManagerImpl | Starting to invoke LAST shutdown tasks
2023-11-07T13:56:05.429Z | Thread-0             |  WARN | d.u.p.ShutdownManagerImpl | Done invoking LAST shutdown tasks
2023-11-07T13:56:05.429Z | Thread-0             |  WARN | d.u.p.ShutdownManagerImpl | Finished shutdown processing

Hope this helps to find the cause of the problem.

Thanks in advance.

devinrsmith commented 1 year ago

Thanks for the report @luisyng - in this case, the parquet file appears to have a nested structure (meta/versionId and meta/lastUpdated). Unfortunately, we don't currently have support for nested structures. There's a ticket filed in the upstream project: https://github.com/deephaven/deephaven-core/issues/871.

Are you able to produce a version of this file with a "flat" definition?

Personally, I've used https://duckdb.org/docs/archive/0.9.1/guides/import/parquet_import and https://duckdb.org/docs/archive/0.9.1/guides/import/parquet_export to massage / flatten files in the past; I'm happy to help you walk through this tooling if you find it helpful.

luisyng commented 1 year ago

Thank you very much for the helpful reply. :)

I will have a look to the tools and try to flatten them. Thanks a lot!