search

deephaven / deephaven-core

Deephaven Community Core
Other
257 stars 80 forks source link

Python array type inference improperly infers the type of single element tuples #4657

Closed chipkent closed 1 year ago

chipkent commented 1 year ago

DH queries seem to have a hard time correctly inferring the type if the return value is a tuple containing a single element, even if the type hint is correct.

from typing import List, Sequence, Tuple, Any
from deephaven import empty_table
import numpy as np

class C:
    def __init__(self):
        print("C")

def f1() -> Tuple[C]:
    return (C(),C())

def f2() -> Tuple[C]:
    return (C())

t = empty_table(10).update(["X1 = f1()","X2 = f2()"])

t2 = t.ungroup()

m = t.meta_table
image

Similarly, if this code is run, an exception is triggered:


from typing import List, Sequence, Tuple, Any
from deephaven import empty_table
import numpy as np

class C:
    def __init__(self):
        print("C")

def f1() -> Tuple[C]:
    return (C(),C())

def f2() -> Tuple[C]:
    return (C())

def f3(i) -> Tuple[C]:
    if i%0 == 0:
        return (C())
    else:
        return (C(),C())

t = empty_table(10).update(["X1 = f1()","X2 = f2()","X3 = f3()"])

t2 = t.ungroup()

m = t.meta_table

With the error:

r-Scheduler-Serial-1 | .c.ConsoleServiceGrpcImpl | Error running script: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'deephaven.dherror.DHError'>
Value: table update operation failed. : java.lang.RuntimeException: Error in Python interpreter:
Traceback (most recent call last):
  File "/private/tmp/my-dh-venv/lib/python3.11/site-packages/deephaven/table.py", line 893, in update
    return Table(j_table=self.j_table.update(*formulas))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: io.deephaven.engine.table.impl.select.FormulaEvaluationException: In formula: X3 = f3.call()
    at io.deephaven.temp.c_ed181becdc254500dc3be552d413d90a27c20b2323f9c8d17eb8d61bd41934e4v55_0.Formula.applyFormulaPerItem(Formula.java:149)
    at io.deephaven.temp.c_ed181becdc254500dc3be552d413d90a27c20b2323f9c8d17eb8d61bd41934e4v55_0.Formula.lambda$fillChunkHelper$4(Formula.java:138)
    at io.deephaven.engine.rowset.RowSequence.lambda$forAllRowKeys$0(RowSequence.java:175)
    at io.deephaven.engine.rowset.impl.singlerange.SingleRangeMixin.forEachRowKey(SingleRangeMixin.java:17)
    at io.deephaven.engine.rowset.RowSequence.forAllRowKeys(RowSequence.java:174)
    at io.deephaven.temp.c_ed181becdc254500dc3be552d413d90a27c20b2323f9c8d17eb8d61bd41934e4v55_0.Formula.fillChunkHelper(Formula.java:136)
    at io.deephaven.temp.c_ed181becdc254500dc3be552d413d90a27c20b2323f9c8d17eb8d61bd41934e4v55_0.Formula.fillChunk(Formula.java:114)
    at io.deephaven.engine.table.impl.sources.ViewColumnSource.fillChunk(ViewColumnSource.java:219)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.doApplyUpdate(SelectColumnLayer.java:412)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.lambda$doSerialApplyUpdate$2(SelectColumnLayer.java:264)
    at io.deephaven.engine.util.systemicmarking.SystemicObjectTracker.executeSystemically(SystemicObjectTracker.java:56)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.doSerialApplyUpdate(SelectColumnLayer.java:263)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.lambda$onAllRequiredColumnsCompleted$1(SelectColumnLayer.java:212)
    at io.deephaven.engine.table.impl.util.ImmediateJobScheduler.submit(ImmediateJobScheduler.java:23)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.onAllRequiredColumnsCompleted(SelectColumnLayer.java:210)
    at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer$SelectLayerCompletionHandler.onLayerCompleted(SelectAndViewAnalyzer.java:589)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.doSerialApplyUpdate(SelectColumnLayer.java:271)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.lambda$onAllRequiredColumnsCompleted$1(SelectColumnLayer.java:212)
    at io.deephaven.engine.table.impl.util.ImmediateJobScheduler.submit(ImmediateJobScheduler.java:23)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.onAllRequiredColumnsCompleted(SelectColumnLayer.java:210)
    at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer$SelectLayerCompletionHandler.onLayerCompleted(SelectAndViewAnalyzer.java:589)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.doSerialApplyUpdate(SelectColumnLayer.java:271)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.lambda$onAllRequiredColumnsCompleted$1(SelectColumnLayer.java:212)
    at io.deephaven.engine.table.impl.util.ImmediateJobScheduler.submit(ImmediateJobScheduler.java:23)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.onAllRequiredColumnsCompleted(SelectColumnLayer.java:210)
    at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer$SelectLayerCompletionHandler.onLayerCompleted(SelectAndViewAnalyzer.java:589)
    at io.deephaven.engine.table.impl.select.analyzers.BaseLayer.applyUpdate(BaseLayer.java:76)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.applyUpdate(SelectColumnLayer.java:151)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.applyUpdate(SelectColumnLayer.java:151)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.applyUpdate(SelectColumnLayer.java:151)
    at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$30(QueryTable.java:1502)
    at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:519)
    at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$31(QueryTable.java:1452)
    at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3458)
    at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1451)
    at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1429)
    at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:109)
    at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
    at org.jpy.PyLib.executeCode(Native Method)
    at org.jpy.PyObject.executeCode(PyObject.java:138)
    at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
    at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:194)
    at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
    at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:194)
    at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:147)
    at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:149)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:160)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:148)
    at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:147)
    at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:84)
    at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:102)
    at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:175)
    at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1384)
    at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:920)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:78)
    at org.jpy.PyLib.callAndReturnObject(Native Method)
    at org.jpy.PyObject.call(PyObject.java:449)
    at io.deephaven.server.console.python.DebuggingInitializer.lambda$createInitializer$0(DebuggingInitializer.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
caused by java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'TypeError'>
Value: f3() missing 1 required positional argument: 'i'
Line: 432
Namespace: wrapper
File: /private/tmp/my-dh-venv/lib/python3.11/site-packages/deephaven/table.py
Traceback (most recent call last):
  File "/private/tmp/my-dh-venv/lib/python3.11/site-packages/deephaven/table.py", line 432, in wrapper

    at org.jpy.PyLib.callAndReturnObject(Native Method)
    at org.jpy.PyObject.callMethod(PyObject.java:432)
    at io.deephaven.engine.util.PyCallableWrapperJpyImpl.call(PyCallableWrapperJpyImpl.java:210)
    at io.deephaven.temp.c_ed181becdc254500dc3be552d413d90a27c20b2323f9c8d17eb8d61bd41934e4v55_0.Formula.applyFormulaPerItem(Formula.java:147)
    ... 62 more

Line: 895
Namespace: update
File: /private/tmp/my-dh-venv/lib/python3.11/site-packages/deephaven/table.py
Traceback (most recent call last):
  File "<string>", line 23, in <module>
  File "/private/tmp/my-dh-venv/lib/python3.11/site-packages/deephaven/table.py", line 895, in update

    at org.jpy.PyLib.executeCode(Native Method)
    at org.jpy.PyObject.executeCode(PyObject.java:138)
    at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
    at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:194)
    at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
    at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:194)
    at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:147)
    at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:149)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:160)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:148)
    at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:147)
    at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:84)
    at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:102)
    at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:175)
    at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1384)
    at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:920)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:78)
    at org.jpy.PyLib.callAndReturnObject(Native Method)
    at org.jpy.PyObject.call(PyObject.java:449)
    at io.deephaven.server.console.python.DebuggingInitializer.lambda$createInitializer$0(DebuggingInitializer.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
jmao-denver commented 1 year ago

The 1st example works as expected, Python array interference only support component types that are either primitive, or String, Boolean, Instant. Everything else will be treated as a scalar org.jpy.PyObject.

The 2nd example contains two user mistakes, specifically, f3 needs to be called with f3(i), and inside the body of f3, i%0 == 0 should probably be i%3 == 0. After correcting them, the code runs through.

chipkent commented 1 year ago

The second example contains a 3rd and bigger mistake. Tuple[C] inciates that the type should only contain a tuple with a single value, which Python simplifies to the single value. Instead, what is being returned does not match the type hint.