deephaven / deephaven-core

Deephaven Community Core
Other
254 stars 80 forks source link

Java streaming functions should work in the query language #2911

Open chipkent opened 2 years ago

chipkent commented 2 years ago

Java streaming functions are very useful, but they do not work properly in the query language. They should be made to work.

For example, this query:

grouping_15min = grouping_15min.update(["TSV_RVS2 =  Arrays.stream((bar_close - bar_open).toArray()).map(Math::signum).toArray()"])

Causes this exception:

grouping_15min = grouping_15min.update(["TSV_RVS2 =  Arrays.stream((bar_close - bar_open).toArray()).map(Math::signum).toArray()"])
r-Scheduler-Serial-1 | i.d.s.s.SessionState      | Internal Error '4c9c0046-fac2-4cb1-8e58-c09317a42cb7' java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'deephaven.dherror.DHError'>
Value: table update operation failed. : Cannot find method toArray() in class [D
Traceback (most recent call last):
  File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven/table.py", line 334, in update
    return Table(j_table=self.j_table.update(*formulas))
RuntimeError: io.deephaven.engine.table.impl.select.FormulaCompilationException: Formula compilation error for: Arrays.stream((bar_close - bar_open).toArray()).map(Math::signum).toArray()
.
.

    at java.base/java.lang.Thread.run(Thread.java:829)
caused by io.deephaven.engine.table.impl.select.FormulaCompilationException: Formula compilation error for: Arrays.stream((bar_close - bar_open).toArray()).map(Math::signum).toArray()
    at io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.analyze(FormulaAnalyzer.java:36)
    at io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:187)
    ... 28 more
caused by io.deephaven.engine.table.impl.lang.QueryLanguageParser$QueryLanguageParseException: 

Having trouble with the following expression:
Full expression           : Arrays.stream((bar_close - bar_open).toArray()).map(Math::signum).toArray()
Expression having trouble : io.deephaven.engine.table.impl.lang.QueryLanguageParser$VisitArgs@7d44c3f5
Exception message         : Cannot find method toArray() in class [D
rcaudy commented 2 years ago

If you're doing this, it points to something that probably needs built-in operational support. This is going to be very, very slow on large data sets, even if we made it work.