marimo-team / marimo

A reactive notebook for Python — run reproducible experiments, execute as a script, deploy as an app, and version with git.
https://marimo.io
Apache License 2.0
6.72k stars 237 forks source link

Dagger usage locally? #2554

Open dmadisetti opened 5 days ago

dmadisetti commented 5 days ago

Describe the bug

Speedy on the mypy fix @mscolnick!

I still get local test failures too with hatch, but much better than before.

Looking at the failures, I think it's because hatch is trying to bring its own binaries- but I don't have globally accessible shared libraries (failure cases below):

```text FAILED tests/_cli/test_cli.py::test_cli_sandbox_edit - assert None is not None FAILED tests/_cli/test_cli.py::test_cli_sandbox_run - assert None is not None FAILED tests/_cli/test_cli.py::test_cli_edit_sandbox_prompt - AssertionError: assert None is not None FAILED tests/_cli/test_cli.py::test_cli_run_sandbox_prompt - AssertionError: assert None is not None FAILED tests/_runtime/test_manage_script_metadata.py::test_manage_script_metadata_uv - assert '"marimo",' in '' FAILED tests/_runtime/test_manage_script_metadata.py::test_manage_script_metadata_uv_deletion - assert '"marimo",' in '' FAILED tests/_server/api/endpoints/test_editing.py::test_format_cell - AssertionError: assert 'cell-123' in {} FAILED tests/_server/api/endpoints/test_files.py::test_rename - AssertionError: assert {'detail': 'F...ready exists'} == {'success': True} FAILED tests/_server/api/endpoints/test_files.py::test_rename_propagates - AssertionError: assert {'detail': 'F...ready exists'} == {'success': True} ```

Nix is weird- so this is fine- but it would be nice to have a standard env. Dagger almost seems to work in nix. It'd be nice to have something reproducible on all systems

Environment

nix run github:dagger/nix#dagger call backend test

Error:

```text cd marimo && typos && cd - && hatch run +py=3.12 test-optional:test 17:26:18 [2/4927] /src ───────────────────────────── test-optional.py3.12 ───────────────────────────── ============================= test session starts ============================== platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0 rootdir: /src configfile: pyproject.toml plugins: asyncio-0.23.8, codecov-0.5.1, anyio-4.6.0, hypothesis-6.102.6, cov-5.0.0 asyncio: mode=Mode.AUTO collected 1767 items / 1 error ==================================== ERRORS ==================================== _________________ ERROR collecting tests/_ast/test_codegen.py __________________ /root/.local/share/hatch/env/virtual/marimo/VsnhxLU2/test-optional.py3.12/lib/python3.12/site-packages/pytest_asyncio/plugin.py:644: in _patched_collect module = collector.obj /root/.local/share/hatch/env/virtual/marimo/VsnhxLU2/test-optional.py3.12/lib/python3.12/site-packages/_pytest/python.py:284: in obj self._obj = obj = self._getobj() /root/.local/share/hatch/env/virtual/marimo/VsnhxLU2/test-optional.py3.12/lib/python3.12/site-packages/_pytest/python.py:546: in _getobj return importtestmodule(self.path, self.config) /root/.local/share/hatch/env/virtual/marimo/VsnhxLU2/test-optional.py3.12/lib/python3.12/site-packages/_pytest/python.py:493: in importtestmodule mod = import_path( /root/.local/share/hatch/env/virtual/marimo/VsnhxLU2/test-optional.py3.12/lib/python3.12/site-packages/_pytest/pathlib.py:582: in import_path importlib.import_module(module_name) /usr/local/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) :1387: in _gcd_import ??? :1360: in _find_and_load ??? :1331: in _find_and_load_unlocked ??? :935: in _load_unlocked ??? /root/.local/share/hatch/env/virtual/marimo/VsnhxLU2/test-optional.py3.12/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:184: in exec_module exec(co, module.__dict__) /home/dylan/src/marimo/tests/_ast/test_codegen.py:12: in ??? :1360: in _find_and_load ??? :1331: in _find_and_load_unlocked ??? :935: in _load_unlocked ??? /root/.local/share/hatch/env/virtual/marimo/VsnhxLU2/test-optional.py3.12/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:184: in exec_module exec(co, module.__dict__) /home/dylan/src/marimo/tests/_ast/codegen_data/test_main.py:9: in ??? marimo/_ast/app.py:225: in cell return self._cell_manager.cell_decorator( marimo/_ast/app.py:472: in cell_decorator return _register(func) marimo/_ast/app.py:454: in _register cell = cell_factory( marimo/_ast/compiler.py:224: in cell_factory code, lnum = inspect.getsourcelines(f) /usr/local/lib/python3.12/inspect.py:1267: in getsourcelines lines, lnum = findsource(object) /usr/local/lib/python3.12/inspect.py:1096: in findsource raise OSError('could not get source code') E OSError: could not get source code =========================== short test summary info ============================ ERROR tests/_ast/test_codegen.py - OSError: could not get source code !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! =============================== 1 error in 4.15s =============================== Stderr: Creating environment: test-optional.py3.12 Creating environment: hatch-uv Checking dependencies Syncing dependencies Installing project in development mode Checking dependencies Syncing dependencies make: *** [Makefile:72: py-test] Error 2 ```

cc: @kpenfound

Code to reproduce

No response

mscolnick commented 5 days ago

Dagger can be run locally. dagger call backend test works for me.

EDIT: Actually I got the same issue eventually

kpenfound commented 4 days ago

I'm not familiar enough with hatch to know what ERROR tests/_ast/test_codegen.py - OSError: could not get source code might be caused by. Does the execution environment need some other packages like git or something possibly?

dmadisetti commented 4 days ago

marimo needs access to the sources files in the library / tests folder. Seems like path info is getting stripped or is maybe inaccessible. Is the whole source directory mounted?

The hatch command works without dagger, so thinking this is might be a volume exposure problem, or bad python paths