search

deephaven / deephaven-core

Deephaven Community Core
Other
257 stars 80 forks source link

Python type inference does not work for pybind11 wrapped C++ #6349

Closed chipkent closed 6 days ago

chipkent commented 1 week ago

I generated a set of examples that integrate C++ into Deephaven. pybind11 was used to integrate C++ directly into python.

The code can be found at https://github.com/chipkent/deephaven-example-use-cases/tree/main/examples/cpp/deephaven-example.

There seem to be problems in both of the mechanisms I tried.

  1. In the first case below, a cast is needed to get the right column type. If the cast is not present, the UI says the column is a String.
  2. In the second case below, there is an error. Value: no signature found for builtin <built-in method price of PyCapsule object at 0xfffec56ae760>

from deephaven import empty_table
import blackscholes

v = blackscholes.price(100, 95, 0.05, 1.2, 0.4, True, False) 
print(v)
print(type(v))

t = empty_table(10).update([
    "UnderlyingPrice = 100+i",
    "Strke = 95",
    "RiskFree = 0.05",
    "YearsToExpiry = 0.6",
    "Vol = 0.4",
    "IsCall = true",
    "IsStock = false",
    "BlackScholesPythonCpp = (double) blackscholes.price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
    "BlackScholesJavaCpp = io.deephaven.BlackScholes.price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
])

price = blackscholes.price

tx = empty_table(10).update([
    "UnderlyingPrice = 100+i",
    "Strke = 95",
    "RiskFree = 0.05",
    "YearsToExpiry = 0.6",
    "Vol = 0.4",
    "IsCall = true",
    "IsStock = false",
    "BlackScholesPythonCpp = price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
    "BlackScholesJavaCpp = io.deephaven.BlackScholes.price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)",
])

Error:

r-Scheduler-Serial-1 | .c.ConsoleServiceGrpcImpl | Error running script: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'deephaven.dherror.DHError'>
Value: table update operation failed. : Error in Python interpreter
Traceback (most recent call last):
  File "/build/venv/lib/python3.12/site-packages/deephaven/table.py", line 994, in update
    return Table(j_table=self.j_table.update(*formulas))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: io.deephaven.engine.table.impl.select.FormulaCompilationException: Formula compilation error for: price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)
    at io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:201)
    at io.deephaven.engine.table.impl.select.SwitchColumn.initDef(SwitchColumn.java:64)
    at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.createContext(SelectAndViewAnalyzer.java:128)
    at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$34(QueryTable.java:1527)
    at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
    at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$35(QueryTable.java:1509)
    at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3646)
    at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1508)
    at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1487)
    at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:98)
    at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
    at org.jpy.PyLib.executeCode(Native Method)
    at org.jpy.PyObject.executeCode(PyObject.java:138)
    at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
    at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:205)
    at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
    at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:205)
    at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:165)
    at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:196)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:207)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:195)
    at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:165)
    at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:72)
    at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:75)
    at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:193)
    at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1537)
    at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:100)
    at java.base/java.lang.Thread.run(Thread.java:829)
caused by io.deephaven.engine.table.impl.lang.QueryLanguageParser$QueryLanguageParseException: 

Having trouble with the following expression:
Full expression           : price(UnderlyingPrice, Strke, RiskFree, YearsToExpiry, Vol, IsCall, IsStock)
Expression having trouble : 
Exception type            : java.lang.RuntimeException
Exception message         : Error in Python interpreter:
Type: <class 'ValueError'>
Value: no signature found for builtin <built-in method price of PyCapsule object at 0xfffec56ae760>
Line: 2361
Namespace: _signature_from_builtin
File: /usr/lib/python3.12/inspect.py
Traceback (most recent call last):
  File "/build/venv/lib/python3.12/site-packages/deephaven/_udf.py", line 529, in _udf_parser
  File "/build/venv/lib/python3.12/site-packages/deephaven/_udf.py", line 496, in _parse_signature
  File "/usr/lib/python3.12/inspect.py", line 3310, in signature
  File "/usr/lib/python3.12/inspect.py", line 3054, in from_callable
  File "/usr/lib/python3.12/inspect.py", line 2571, in _signature_from_callable
  File "/usr/lib/python3.12/inspect.py", line 2361, in _signature_from_builtin

    at org.jpy.PyLib.callAndReturnObject(Native Method)
    at org.jpy.PyObject.call(PyObject.java:449)
    at io.deephaven.engine.util.PyCallableWrapperJpyImpl.prepareSignature(PyCallableWrapperJpyImpl.java:195)
    at io.deephaven.engine.util.PyCallableWrapperJpyImpl.parseSignature(PyCallableWrapperJpyImpl.java:206)
    at io.deephaven.engine.table.impl.lang.QueryLanguageParser.pyCallableReturnType(QueryLanguageParser.java:2580)
    at io.deephaven.engine.table.impl.lang.QueryLanguageParser.visit(QueryLanguageParser.java:2389)
    at io.deephaven.engine.table.impl.lang.QueryLanguageParser.visit(QueryLanguageParser.java:133)
    at com.github.javaparser.ast.expr.MethodCallExpr.accept(MethodCallExpr.java:116)
    at io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:319)
    at io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:212)
    at io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:240)
    at io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:122)
    at io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:181)
    ... 32 more

Line: 996
Namespace: update
File: /build/venv/lib/python3.12/site-packages/deephaven/table.py
Traceback (most recent call last):
  File "<string>", line 24, in <module>
  File "/build/venv/lib/python3.12/site-packages/deephaven/table.py", line 996, in update

    at org.jpy.PyLib.executeCode(Native Method)
    at org.jpy.PyObject.executeCode(PyObject.java:138)
    at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
    at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:205)
    at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
    at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:205)
    at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:165)
    at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:196)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:207)
    at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:195)
    at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:165)
    at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:72)
    at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:75)
    at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:193)
    at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1537)
    at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:100)
    at java.base/java.lang.Thread.run(Thread.java:829)
jmao-denver commented 1 week ago
  1. I have done some investigation and it turns out that to add signatures to functions in C extension modules isn't as one might expect. See the following.

https://github.com/python/cpython/issues/123473 https://stackoverflow.com/questions/1104823/python-c-extension-method-signatures-for-documentation https://stackoverflow.com/questions/25847035/what-are-signature-and-text-signature-used-for-in-python-3-4/25847066#25847066

  1. I have also tried creating .pyi files but they don't help runtime inspection via the inspect module.
  2. besides the use of __text_signature__, another alternative way is to wrap C extension modules (making them private) in pure Python code.
  3. numba, numpy choose to store the proprietary encoded signature in the types attribute of the function object
  4. DH UDF signature parser needs to properly handle the case where signature isn't available