search

deephaven / deephaven-core

Deephaven Community Core
Other
230 stars 77 forks source link

Update expressions run in embedded python server error #5426

Closed niloc132 closed 1 week ago

niloc132 commented 2 weeks ago

This appears to be a regression in 0.34.

Code to run in python, with deephaven-core and deephaven-embedded installed:

from deephaven_server import Server
s = Server().start()

from deephaven import time_table

table = time_table("PT1S").update(formulas=["X=i"])

The last line will fail with an error: java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200

Full error message:

Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 109, in shared_lock
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 256, in auto_locking_ctx
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 764, in update
    return Table(j_table=self.j_table.update(*formulas))
RuntimeError: io.deephaven.UncheckedDeephavenException: Compilation failed
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:172)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:141)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:73)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$32(QueryTable.java:1518)
        at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$33(QueryTable.java:1500)
        at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3639)
        at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1499)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1477)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:100)
        at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
caused by java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:825)
        at io.deephaven.engine.context.QueryCompilerImpl.compileHelper(QueryCompilerImpl.java:563)
        at io.deephaven.engine.context.QueryCompilerImpl.compile(QueryCompilerImpl.java:276)
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:156)
        ... 10 more
caused by java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
        at java.base/java.nio.file.Files.createDirectory(Files.java:700)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:134)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:171)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:976)
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:818)
        ... 13 more

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 763, in update
    with _query_scope_ctx(), auto_locking_ctx(self):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 255, in auto_locking_ctx
    with shared_lock(arg.update_graph):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 111, in shared_lock
    raise DHError(e, "exception raised in the enclosed code block.") from e
deephaven.dherror.DHError: exception raised in the enclosed code block. : java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 109, in shared_lock
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 256, in auto_locking_ctx
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 764, in update
    return Table(j_table=self.j_table.update(*formulas))
RuntimeError: io.deephaven.UncheckedDeephavenException: Compilation failed
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:172)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:141)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:73)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$32(QueryTable.java:1518)
        at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$33(QueryTable.java:1500)
        at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3639)
        at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1499)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1477)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:100)
        at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
caused by java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:825)
        at io.deephaven.engine.context.QueryCompilerImpl.compileHelper(QueryCompilerImpl.java:563)
        at io.deephaven.engine.context.QueryCompilerImpl.compile(QueryCompilerImpl.java:276)
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:156)
        ... 10 more
caused by java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
        at java.base/java.nio.file.Files.createDirectory(Files.java:700)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:134)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:171)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:976)
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:818)
        ... 13 more

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 766, in update
    raise DHError(e, "table update operation failed.") from e
deephaven.dherror.DHError: table update operation failed. : java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 109, in shared_lock
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 256, in auto_locking_ctx
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 764, in update
    return Table(j_table=self.j_table.update(*formulas))
RuntimeError: io.deephaven.UncheckedDeephavenException: Compilation failed
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:172)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:141)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:73)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$32(QueryTable.java:1518)
        at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$33(QueryTable.java:1500)
        at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3639)
        at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1499)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1477)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:100)
        at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
caused by java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:825)
        at io.deephaven.engine.context.QueryCompilerImpl.compileHelper(QueryCompilerImpl.java:563)
        at io.deephaven.engine.context.QueryCompilerImpl.compile(QueryCompilerImpl.java:276)
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:156)
        ... 10 more
caused by java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
        at java.base/java.nio.file.Files.createDirectory(Files.java:700)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:134)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:171)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:976)
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:818)
        ... 13 more

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 763, in update
    with _query_scope_ctx(), auto_locking_ctx(self):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 255, in auto_locking_ctx
    with shared_lock(arg.update_graph):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 111, in shared_lock
    raise DHError(e, "exception raised in the enclosed code block.") from e
deephaven.dherror.DHError: exception raised in the enclosed code block. : java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
Traceback (most recent call last):
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 109, in shared_lock
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/update_graph.py", line 256, in auto_locking_ctx
    yield
  File "/home/colin/.pyenv/versions/3.10.13/lib/python3.10/site-packages/deephaven/table.py", line 764, in update
    return Table(j_table=self.j_table.update(*formulas))
RuntimeError: io.deephaven.UncheckedDeephavenException: Compilation failed
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:172)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:141)
        at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.create(SelectAndViewAnalyzer.java:73)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$32(QueryTable.java:1518)
        at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
        at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$33(QueryTable.java:1500)
        at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3639)
        at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1499)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1477)
        at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:100)
        at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
caused by java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:825)
        at io.deephaven.engine.context.QueryCompilerImpl.compileHelper(QueryCompilerImpl.java:563)
        at io.deephaven.engine.context.QueryCompilerImpl.compile(QueryCompilerImpl.java:276)
        at io.deephaven.engine.table.impl.QueryCompilerRequestProcessor$BatchProcessor.compile(QueryCompilerRequestProcessor.java:156)
        ... 10 more
caused by java.nio.file.NoSuchFileException: /home/colin/.cache/deephaven/script-session-classes/7a9e91f0-6e81-42ad-8c69-b3390cf98660/temporaryCompilationDirectory8658035305834051200
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
        at java.base/java.nio.file.Files.createDirectory(Files.java:700)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:134)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:171)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:976)
        at io.deephaven.engine.context.QueryCompilerImpl.maybeCreateClasses(QueryCompilerImpl.java:818)
        ... 13 more
niloc132 commented 2 weeks ago

This regression appears to be caused by #5070 - the commit prior to that change does not have this issue, but that commit causes this.

devinrsmith commented 2 weeks ago

Our main community image does not exhibit this issue, appears to be specific to the embedded python server. For reference, running the same query formula in the community image produces something like:

$ find /cache -type f
/cache/.JsPluginsZipFilesystem13610993811594593474/deephaven-js-plugins.zip
/cache/script-session-classes/f4fcf678-7167-4e3e-b8af-5c4f30495c39/io/deephaven/temp/c_7f3e63ff2190de96d978de3d723129623aaa22226001a02b65dec404ca886cd6v65_0/Formula$FormulaFillContext.class
/cache/script-session-classes/f4fcf678-7167-4e3e-b8af-5c4f30495c39/io/deephaven/temp/c_7f3e63ff2190de96d978de3d723129623aaa22226001a02b65dec404ca886cd6v65_0/Formula.class
jmao-denver commented 2 weeks ago

Here is a somewhat similar/related case and I talked to both @nbauernfeind and @rcaudy about it. https://github.com/deephaven/deephaven-core/blob/c71506e85565658953b966a5cb68014f83fa4794/py/server/tests/test_barrage.py#L23

devinrsmith commented 2 weeks ago

Here is a somewhat similar/related case and I talked to both @nbauernfeind and @rcaudy about it.

https://github.com/deephaven/deephaven-core/blob/c71506e85565658953b966a5cb68014f83fa4794/py/server/tests/test_barrage.py#L23

When you are running two instances of DH on the same physical (or virtual) instance, it's probably better practice to set the application name:

https://deephaven.io/core/docs/how-to-guides/configuration/native-application/#application-name

which will ensure that the data, config, and cache dirs are unique.

rcaudy commented 1 week ago

Closed by https://github.com/deephaven/deephaven-core/pull/5436