search

deephaven / deephaven-core

Deephaven Community Core
Other
256 stars 80 forks source link

where() fails after renaming columns on table read from parquet file #2701

Closed rbasralian closed 1 year ago

rbasralian commented 2 years ago

If I read a table from Parquet, then rename one of its columns with a view(), then run a where() on the renamed column, I get an exception while compiling the filter. However the code works fine if the view() and where() are run in separate commands in the console.

DH Version 0.14.1

Reproducing code:

SIMULATED_STREAKS_PATH = '/dh/data/shared/baseball/simulation/bts_simulation_results_2021.parquet'
contestants_with_streaks = io.deephaven.parquet.table.ParquetTools.readTable(SIMULATED_STREAKS_PATH)
tst = contestants_with_streaks.view(
        'Batter=Pick1Batter'
    )
println tst.getDefinition().getColumnNames()
// this works fine if you run it separately, but not when you copy/paste
// it at the same time as the .view() above
tst2 = tst.where('!isNull(Batter)')

Exception:

r-Scheduler-Serial-1 | i.d.s.s.SessionState      | Internal Error 'c020825d-73f1-4a6d-9b05-05877c59386f' io.deephaven.engine.table.impl.select.FormulaCompilationException: Formula compilation error for: !isNull(Batter)
        at io.deephaven.engine.table.impl.select.AbstractConditionFilter.init(AbstractConditionFilter.java:200)
        at io.deephaven.engine.table.impl.QueryTable.lambda$whereInternal$28(QueryTable.java:1023)
        at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:518)
        at io.deephaven.engine.table.impl.QueryTable.whereInternal(QueryTable.java:993)
        at io.deephaven.engine.table.impl.QueryTable.where(QueryTable.java:989)
        at io.deephaven.engine.table.impl.TableWithDefaults.where(TableWithDefaults.java:212)
        at io.deephaven.engine.table.impl.DeferredViewTable.getResultTableWithWhere(DeferredViewTable.java:103)
        at io.deephaven.engine.table.impl.DeferredViewTable.where(DeferredViewTable.java:73)
        at io.deephaven.engine.table.impl.TableWithDefaults.where(TableWithDefaults.java:218)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:191)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
        at io.deephaven.dynamic.Script_32.run(Script_32.groovy:45)
        at groovy.lang.GroovyShell.evaluate(GroovyShell.java:427)
        at groovy.lang.GroovyShell.evaluate(GroovyShell.java:461)
        at groovy.lang.GroovyShell.evaluate(GroovyShell.java:436)
        at io.deephaven.engine.util.GroovyDeephavenSession.evaluateCommand(GroovyDeephavenSession.java:228)
        at io.deephaven.engine.util.GroovyDeephavenSession.lambda$evaluate$2(GroovyDeephavenSession.java:251)
        at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:49)
        at io.deephaven.engine.util.GroovyDeephavenSession.evaluate(GroovyDeephavenSession.java:251)
        at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:166)
        at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:81)
        at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:106)
        at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$8(ConsoleServiceGrpcImpl.java:188)
        at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1299)
        at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:847)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at io.deephaven.server.runner.DeephavenApiServerModule$ThreadFactory.lambda$newThread$0(DeephavenApiServerModule.java:162)
        at java.lang.Thread.run(Thread.java:829)
caused by:
java.lang.RuntimeException: Error compiling class io.deephaven.temp.c1359401156n730523876v55_0.GeneratedFilterKernel:
/io/deephaven/temp/c1359401156n730523876v55_0/GeneratedFilterKernel.java:87: error: cannot find symbol
            if (not(isNull(Batter))) {
                           ^
  symbol:   variable Batter
  location: class GeneratedFilterKernel
1 error

        at io.deephaven.compilertools.CompilerTools.maybeCreateClassHelper(CompilerTools.java:789)
        at io.deephaven.compilertools.CompilerTools.maybeCreateClass(CompilerTools.java:752)
        at io.deephaven.compilertools.CompilerTools.compileHelper(CompilerTools.java:510)
        at io.deephaven.compilertools.CompilerTools.compile(CompilerTools.java:469)
        at io.deephaven.compilertools.CompilerTools.compile(CompilerTools.java:424)
        at io.deephaven.engine.table.impl.select.ConditionFilter.generateFilterCode(ConditionFilter.java:392)
        at io.deephaven.engine.table.impl.select.AbstractConditionFilter.init(AbstractConditionFilter.java:197)
        at io.deephaven.engine.table.impl.QueryTable.lambda$whereInternal$28(QueryTable.java:1023)
        at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:518)
        at io.deephaven.engine.table.impl.QueryTable.whereInternal(QueryTable.java:993)
        at io.deephaven.engine.table.impl.QueryTable.where(QueryTable.java:989)
        at io.deephaven.engine.table.impl.TableWithDefaults.where(TableWithDefaults.java:212)
        at io.deephaven.engine.table.impl.DeferredViewTable.getResultTableWithWhere(DeferredViewTable.java:103)
        at io.deephaven.engine.table.impl.DeferredViewTable.where(DeferredViewTable.java:73)
        at io.deephaven.engine.table.impl.TableWithDefaults.where(TableWithDefaults.java:218)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:191)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
        at io.deephaven.dynamic.Script_17.run(Script_17.groovy:43)
        at groovy.lang.GroovyShell.evaluate(GroovyShell.java:427)
        at groovy.lang.GroovyShell.evaluate(GroovyShell.java:461)
        at groovy.lang.GroovyShell.evaluate(GroovyShell.java:436)
        at io.deephaven.engine.util.GroovyDeephavenSession.evaluateCommand(GroovyDeephavenSession.java:228)
        at io.deephaven.engine.util.GroovyDeephavenSession.lambda$evaluate$2(GroovyDeephavenSession.java:251)
        at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:49)
        at io.deephaven.engine.util.GroovyDeephavenSession.evaluate(GroovyDeephavenSession.java:251)
        at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:166)
        at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:81)
        at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:106)
        at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$8(ConsoleServiceGrpcImpl.java:188)
        at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1299)
        at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:847)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at io.deephaven.server.runner.DeephavenApiServerModule$ThreadFactory.lambda$newThread$0(DeephavenApiServerModule.java:162)
        at java.lang.Thread.run(Thread.java:829)
rcaudy commented 2 years ago

@rbasralian is this the same as #1683?

rbasralian commented 2 years ago

Yes, definitely the same as #1683

nbauernfeind commented 1 year ago

This was related to #1683, but ultimately required additional work.