search

deephaven / deephaven-core

Deephaven Community Core
Other
251 stars 80 forks source link

An incorrect type hint raises an esoteric error message that depends on table size #4287

Open jjbrosnan opened 1 year ago

jjbrosnan commented 1 year ago

Description

Using the wrong NumPy type in a Python type hint for a function called in a table operation raises confusing errors. The errors themselves don't really indicate an issue with the type hint itself. Additionally, the error message is only somewhat helpful if the table has a single row. If the table has more than one row, it raises an even more confusing error.

I discovered this when I accidentally used the type hint -> np.intc instead of -> np.double for the following function:

def bessel(x):
    return np.i0(x)

Steps to reproduce

The following script will reproduce the error. It should be run with a table with a single row, as well as a table with more than one row, as the error itself will change.

from deephaven import empty_table
import numpy as np

def bessel(x) -> np.intc:
    return np.i0(x)

# Change 1 to 2 for a different error message
t = empty_table(1).update(["X = i", "Y = bessel(X)"])

Expected results

One of the following:

Actual results

The following error for one row:

only integer scalar arrays can be converted to a scalar index

The following error for more than one row:

<built-in function next> returned a result with an exception set

The full stack trace is long. I will paste it for both cases in a thread.

Additional details and attachments

If applicable, add any additional screenshots, logs, or other attachments to help explain your problem.

Versions

jjbrosnan commented 1 year ago

Full stack trace for one row:

r-Scheduler-Serial-1 | .c.ConsoleServiceGrpcImpl | Error running script: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'deephaven.dherror.DHError'>
Value: table update operation failed. : RuntimeError: java.lang.RuntimeException: Error in Python interpreter:
Traceback (most recent call last):
  File "/opt/deephaven/venv/lib/python3.10/site-packages/deephaven/table.py", line 792, in update
    return Table(j_table=self.j_table.update(*formulas))
RuntimeError: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'TypeError'>
Value: only integer scalar arrays can be converted to a scalar index
Line: 418
Namespace: wrapper
File: /opt/deephaven/venv/lib/python3.10/site-packages/deephaven/table.py
Traceback (most recent call last):
  File "/opt/deephaven/venv/lib/python3.10/site-packages/deephaven/table.py", line 418, in wrapper

    at org.jpy.PyLib.callAndReturnValue(Native Method)
    at org.jpy.PyObject.call(PyObject.java:474)
    at io.deephaven.engine.table.impl.select.python.FormulaKernelPythonChunkedFunction.applyFormulaChunk(FormulaKernelPythonChunkedFunction.java:93)
    at io.deephaven.engine.table.impl.select.FormulaKernelTypedBase$ToTypedMethod.visit(FormulaKernelTypedBase.java:138)
    at io.deephaven.chunk.IntChunk.walk(IntChunk.java:130)
    at io.deephaven.engine.table.impl.select.FormulaKernelTypedBase.applyFormulaChunk(FormulaKernelTypedBase.java:42)
    at io.deephaven.engine.table.impl.select.formula.FormulaKernelAdapter.fillChunkHelper(FormulaKernelAdapter.java:299)
    at io.deephaven.engine.table.impl.select.formula.FormulaKernelAdapter.fillChunk(FormulaKernelAdapter.java:229)
    at io.deephaven.engine.table.impl.sources.ViewColumnSource.fillChunk(ViewColumnSource.java:219)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.doApplyUpdate(SelectColumnLayer.java:412)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.lambda$doSerialApplyUpdate$2(SelectColumnLayer.java:264)
    at io.deephaven.engine.util.systemicmarking.SystemicObjectTracker.executeSystemically(SystemicObjectTracker.java:56)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.doSerialApplyUpdate(SelectColumnLayer.java:263)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.lambda$onAllRequiredColumnsCompleted$1(SelectColumnLayer.java:212)
    at io.deephaven.engine.table.impl.util.ImmediateJobScheduler.submit(ImmediateJobScheduler.java:23)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.onAllRequiredColumnsCompleted(SelectColumnLayer.java:210)
    at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer$SelectLayerCompletionHandler.onLayerCompleted(SelectAndViewAnalyzer.java:589)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.doSerialApplyUpdate(SelectColumnLayer.java:271)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.lambda$onAllRequiredColumnsCompleted$1(SelectColumnLayer.java:212)
    at io.deephaven.engine.table.impl.util.ImmediateJobScheduler.submit(ImmediateJobScheduler.java:23)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.onAllRequiredColumnsCompleted(SelectColumnLayer.java:210)
    at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer$SelectLayerCompletionHandler.onLayerCompleted(SelectAndViewAnalyzer.java:589)
    at io.deephaven.engine.table.impl.select.analyzers.BaseLayer.applyUpdate(BaseLayer.java:76)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.applyUpdate(SelectColumnLayer.java:151)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.applyUpdate(SelectColumnLayer.java:151)
    at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$32(QueryTable.java:1477)
    at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:519)
    at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$33(QueryTable.java:1427)
    at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3464)
    at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1426)
    at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1404)
    at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:109)
    at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
    at org.jpy.PyLib.executeCode(Native Method)
    at org.jpy.PyObject.executeCode(PyObject.java:138)
    at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
    at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:197)
    at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
    at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:197)
    at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:150)
    at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:149)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:160)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:148)
    at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:150)
    at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:87)
    at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:105)
    at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:175)
    at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1380)
    at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:921)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:78)
    at org.jpy.PyLib.callAndReturnObject(Native Method)
    at org.jpy.PyObject.call(PyObject.java:449)
    at io.deephaven.server.console.python.DebuggingInitializer.lambda$createInitializer$0(DebuggingInitializer.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)

Line: 794
Namespace: update
File: /opt/deephaven/venv/lib/python3.10/site-packages/deephaven/table.py
Traceback (most recent call last):
  File "<string>", line 7, in <module>
  File "/opt/deephaven/venv/lib/python3.10/site-packages/deephaven/table.py", line 794, in update

        at org.jpy.PyLib.executeCode(PyLib.java:-2)
        at org.jpy.PyObject.executeCode(PyObject.java:138)
        at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
        at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:197)
        at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
        at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:197)
        at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:150)
        at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:149)
        at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:160)
        at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:148)
        at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:150)
        at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:87)
        at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:105)
        at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:175)
        at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1380)
        at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:921)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:78)
        at org.jpy.PyLib.callAndReturnObject(PyLib.java:-2)
        at org.jpy.PyObject.call(PyObject.java:449)
        at io.deephaven.server.console.python.DebuggingInitializer.lambda$createInitializer$0(DebuggingInitializer.java:30)
        at java.lang.Thread.run(Thread.java:833)
jjbrosnan commented 1 year ago

Full stack trace for multiple rows

r-Scheduler-Serial-1 | .c.ConsoleServiceGrpcImpl | Error running script: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'deephaven.dherror.DHError'>
Value: table update operation failed. : RuntimeError: java.lang.RuntimeException: Error in Python interpreter:
Traceback (most recent call last):
  File "/opt/deephaven/venv/lib/python3.10/site-packages/deephaven/table.py", line 792, in update
    return Table(j_table=self.j_table.update(*formulas))
RuntimeError: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'SystemError'>
Value: <built-in function next> returned a result with an exception set
Line: 419
Namespace: wrapper
File: /opt/deephaven/venv/lib/python3.10/site-packages/deephaven/table.py
Traceback (most recent call last):
  File "/opt/deephaven/venv/lib/python3.10/site-packages/deephaven/table.py", line 419, in wrapper

    at org.jpy.PyLib.callAndReturnValue(Native Method)
    at org.jpy.PyObject.call(PyObject.java:474)
    at io.deephaven.engine.table.impl.select.python.FormulaKernelPythonChunkedFunction.applyFormulaChunk(FormulaKernelPythonChunkedFunction.java:93)
    at io.deephaven.engine.table.impl.select.FormulaKernelTypedBase$ToTypedMethod.visit(FormulaKernelTypedBase.java:138)
    at io.deephaven.chunk.IntChunk.walk(IntChunk.java:130)
    at io.deephaven.engine.table.impl.select.FormulaKernelTypedBase.applyFormulaChunk(FormulaKernelTypedBase.java:42)
    at io.deephaven.engine.table.impl.select.formula.FormulaKernelAdapter.fillChunkHelper(FormulaKernelAdapter.java:299)
    at io.deephaven.engine.table.impl.select.formula.FormulaKernelAdapter.fillChunk(FormulaKernelAdapter.java:229)
    at io.deephaven.engine.table.impl.sources.ViewColumnSource.fillChunk(ViewColumnSource.java:219)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.doApplyUpdate(SelectColumnLayer.java:412)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.lambda$doSerialApplyUpdate$2(SelectColumnLayer.java:264)
    at io.deephaven.engine.util.systemicmarking.SystemicObjectTracker.executeSystemically(SystemicObjectTracker.java:56)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.doSerialApplyUpdate(SelectColumnLayer.java:263)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.lambda$onAllRequiredColumnsCompleted$1(SelectColumnLayer.java:212)
    at io.deephaven.engine.table.impl.util.ImmediateJobScheduler.submit(ImmediateJobScheduler.java:23)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.onAllRequiredColumnsCompleted(SelectColumnLayer.java:210)
    at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer$SelectLayerCompletionHandler.onLayerCompleted(SelectAndViewAnalyzer.java:589)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.doSerialApplyUpdate(SelectColumnLayer.java:271)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.lambda$onAllRequiredColumnsCompleted$1(SelectColumnLayer.java:212)
    at io.deephaven.engine.table.impl.util.ImmediateJobScheduler.submit(ImmediateJobScheduler.java:23)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer$1.onAllRequiredColumnsCompleted(SelectColumnLayer.java:210)
    at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer$SelectLayerCompletionHandler.onLayerCompleted(SelectAndViewAnalyzer.java:589)
    at io.deephaven.engine.table.impl.select.analyzers.BaseLayer.applyUpdate(BaseLayer.java:76)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.applyUpdate(SelectColumnLayer.java:151)
    at io.deephaven.engine.table.impl.select.analyzers.SelectColumnLayer.applyUpdate(SelectColumnLayer.java:151)
    at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$32(QueryTable.java:1477)
    at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:519)
    at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$33(QueryTable.java:1427)
    at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3464)
    at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1426)
    at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1404)
    at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:109)
    at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
    at org.jpy.PyLib.executeCode(Native Method)
    at org.jpy.PyObject.executeCode(PyObject.java:138)
    at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
    at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:197)
    at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
    at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:197)
    at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:150)
    at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:149)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:160)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:148)
    at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:150)
    at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:87)
    at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:105)
    at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:175)
    at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1380)
    at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:921)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:78)
    at org.jpy.PyLib.callAndReturnObject(Native Method)
    at org.jpy.PyObject.call(PyObject.java:449)
    at io.deephaven.server.console.python.DebuggingInitializer.lambda$createInitializer$0(DebuggingInitializer.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)

Line: 794
Namespace: update
File: /opt/deephaven/venv/lib/python3.10/site-packages/deephaven/table.py
Traceback (most recent call last):
  File "<string>", line 7, in <module>
  File "/opt/deephaven/venv/lib/python3.10/site-packages/deephaven/table.py", line 794, in update

        at org.jpy.PyLib.executeCode(PyLib.java:-2)
        at org.jpy.PyObject.executeCode(PyObject.java:138)
        at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
        at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:197)
        at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
        at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:197)
        at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:150)
        at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:149)
        at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:160)
        at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:148)
        at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:150)
        at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:87)
        at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:105)
        at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:175)
        at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1380)
        at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:921)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:78)
        at org.jpy.PyLib.callAndReturnObject(PyLib.java:-2)
        at org.jpy.PyObject.call(PyObject.java:449)
        at io.deephaven.server.console.python.DebuggingInitializer.lambda$createInitializer$0(DebuggingInitializer.java:30)
        at java.lang.Thread.run(Thread.java:833)