mwouts / jupytext

Jupyter Notebooks as Markdown Documents, Julia, Python or R scripts
https://jupytext.readthedocs.io
MIT License
6.6k stars 386 forks source link

Add Xonsh support #1213

Closed jsquaredosquared closed 5 months ago

jsquaredosquared commented 7 months ago

Summary

Hi @mwouts, I have tried following the instructions in the documentation to add support for Xonsh.

File extension: .xsh Comment: "#" Kernel: xonsh-jupyter

However, on my machine, 2 tests are still failing (even though I did not touch the code):

================================================================================= short test summary info ==================================================================================
FAILED tests/integration/cli/test_execute.py::test_execute_r - AttributeError: outputs
FAILED tests/functional/others/test_pytest.py::test_run_pytest_ok - SystemExit: 4
============================================================ 2 failed, 3123 passed, 88 skipped, 6 warnings in 563.72s (0:09:23) ============================================================

Can you provide any guidance on this?

Thanks, Jeff


Full test failures: ```python ______________________________________________________________________________________ test_execute_r ______________________________________________________________________________________ [gw1] linux -- Python 3.11.7 /home/jeffr/miniconda3/envs/jupytext-dev/bin/python tmpdir = local('/tmp/pytest-of-jeffr/pytest-9/popen-gw1/test_execute_r0'), caplog = <_pytest.logging.LogCaptureFixture object at 0x7f7c2e31ef10> capsys = <_pytest.capture.CaptureFixture object at 0x7f7c2e31c3d0> @pytest.mark.requires_nbconvert @pytest.mark.requires_ir_kernel @pytest.mark.skip_on_windows def test_execute_r(tmpdir, caplog, capsys): # pragma: no cover tmp_ipynb = str(tmpdir.join("notebook.ipynb")) tmp_md = str(tmpdir.join("notebook.md")) with open(tmp_md, "w") as fp: fp.write( """```r 1 + 2 + 3 ``` """ ) jupytext(args=[tmp_md, "--to", "ipynb", "--execute"]) nb = read(tmp_ipynb) assert len(nb.cells) == 1 > assert nb.cells[0].outputs[0]["data"]["text/markdown"] == "6" tests/integration/cli/test_execute.py:176: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = {'cell_type': 'markdown', 'id': 'cell-1208', 'metadata': {}, 'source': '```r\n1 + 2 + 3\n```'}, key = 'outputs' def __getattr__(self, key): """Get an attr by calling :meth:`dict.__getitem__`. Like :meth:`__setattr__`, this method converts :exc:`KeyError` to :exc:`AttributeError`. Examples -------- >>> s = Struct(a=10) >>> s.a 10 >>> type(s.get) <... 'builtin_function_or_method'> >>> try: ... s.b ... except AttributeError: ... print("I don't have that key") ... I don't have that key """ try: result = self[key] except KeyError: > raise AttributeError(key) from None E AttributeError: outputs ../miniconda3/envs/jupytext-dev/lib/python3.11/site-packages/nbformat/_struct.py:125: AttributeError ----------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------- [jupytext] Reading /tmp/pytest-of-jeffr/pytest-9/popen-gw1/test_execute_r0/notebook.md in format md [jupytext] Setting kernel python3 [jupytext] Updating notebook metadata with '{"kernelspec": {"name": "python3", "language": "python", "display_name": "Python 3 (ipykernel)"}}' [jupytext] Executing notebook with kernel python3 [jupytext] Writing /tmp/pytest-of-jeffr/pytest-9/popen-gw1/test_execute_r0/notebook.ipynb ----------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------- 0.00s - Debugger warning: It seems that frozen modules are being used, which may 0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off 0.00s - to python to disable frozen modules. 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation. ____________________________________________________________________________________ test_run_pytest_ok ____________________________________________________________________________________ [gw2] linux -- Python 3.11.7 /home/jeffr/miniconda3/envs/jupytext-dev/bin/python tmpdir = local('/tmp/pytest-of-jeffr/pytest-9/popen-gw2/test_run_pytest_ok0') text = 'This is a notebook with a test that is supposed to pass\n```python\ndef test_ok():\n assert True\n```\n' def test_run_pytest_ok( tmpdir, text="""This is a notebook with a test that is supposed to pass ```python def test_ok(): assert True ``` """, ): tmp_md = str(tmpdir.join("notebook.md")) with open(tmp_md, "w") as fp: fp.write(text) > jupytext([tmp_md, "--check", "pytest"]) tests/functional/others/test_pytest.py:18: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ src/jupytext/cli.py:497: in jupytext exit_code += jupytext_single_file(nb_file, args, log) src/jupytext/cli.py:670: in jupytext_single_file pipe_notebook( src/jupytext/cli.py:1182: in pipe_notebook exec_command( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ command = ['pytest', '/tmp/notebook_oo_56at8.py'], input = None, capture = False, warn_only = False def exec_command(command, input=None, capture=False, warn_only=False): """Execute the desired command, and pipe the given input into it""" assert isinstance(command, list) sys.stdout.write("[jupytext] Executing {}\n".format(" ".join(command))) process = subprocess.Popen( command, **( dict(stdout=subprocess.PIPE, stdin=subprocess.PIPE) if input is not None else {} ), ) out, err = process.communicate(input=input) if out and not capture: sys.stdout.write(out.decode("utf-8")) if err: sys.stderr.write(err.decode("utf-8")) if process.returncode: msg = f"The command '{' '.join(command)}' exited with code {process.returncode}" hint = ( "" if warn_only else " (use --warn-only to turn this error into a warning)" ) sys.stderr.write( f"[jupytext] {'Warning' if warn_only else 'Error'}: {msg}{hint}\n" ) if not warn_only: > raise SystemExit(process.returncode) E SystemExit: 4 src/jupytext/cli.py:1140: SystemExit ----------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------- [jupytext] Reading /tmp/pytest-of-jeffr/pytest-9/popen-gw2/test_run_pytest_ok0/notebook.md in format md [jupytext] Executing pytest /tmp/notebook_oo_56at8.py ============================= test session starts ============================== platform linux -- Python 3.11.7, pytest-8.0.0, pluggy-1.4.0 Using --randomly-seed=419236798 rootdir: /tmp plugins: xdist-3.5.0, cov-4.1.0, anyio-4.2.0, randomly-3.15.0, xonsh-0.14.4 collected 0 items / 1 error ==================================== ERRORS ==================================== ______________________________ ERROR collecting . ______________________________ ../miniconda3/envs/jupytext-dev/lib/python3.11/site-packages/pluggy/_hooks.py:501: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ../miniconda3/envs/jupytext-dev/lib/python3.11/site-packages/pluggy/_manager.py:119: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ../miniconda3/envs/jupytext-dev/lib/python3.11/site-packages/_pytest/python.py:210: in pytest_collect_directory if pkginit.is_file(): ../miniconda3/envs/jupytext-dev/lib/python3.11/pathlib.py:1267: in is_file return S_ISREG(self.stat().st_mode) ../miniconda3/envs/jupytext-dev/lib/python3.11/pathlib.py:1013: in stat return os.stat(self, follow_symlinks=follow_symlinks) E PermissionError: [Errno 13] Permission denied: '/tmp/systemd-private-0c4b06cedf23405bad0e7365a61e84d6-systemd-logind.service-bDyMgv/__init__.py' =========================== short test summary info ============================ ERROR ../../../tmp - PermissionError: [Errno 13] Permission denied: '/tmp/sys... =============================== 1 error in 0.74s =============================== ----------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------- ERROR: found no collectors for /tmp/notebook_oo_56at8.py [jupytext] Error: The command 'pytest /tmp/notebook_oo_56at8.py' exited with code 4 (use --warn-only to turn this error into a warning) ===================================================================================== warnings summary ===================================================================================== ../miniconda3/envs/jupytext-dev/lib/python3.11/site-packages/sphinx_gallery/sphinx_compatibility.py:71 ../miniconda3/envs/jupytext-dev/lib/python3.11/site-packages/sphinx_gallery/sphinx_compatibility.py:71 ../miniconda3/envs/jupytext-dev/lib/python3.11/site-packages/sphinx_gallery/sphinx_compatibility.py:71 ../miniconda3/envs/jupytext-dev/lib/python3.11/site-packages/sphinx_gallery/sphinx_compatibility.py:71 ../miniconda3/envs/jupytext-dev/lib/python3.11/site-packages/sphinx_gallery/sphinx_compatibility.py:71 ../miniconda3/envs/jupytext-dev/lib/python3.11/site-packages/sphinx_gallery/sphinx_compatibility.py:71 /home/jeffr/miniconda3/envs/jupytext-dev/lib/python3.11/site-packages/sphinx_gallery/sphinx_compatibility.py:71: RemovedInSphinx80Warning: The alias 'sphinx.util.status_iterator' is deprecated, use 'sphinx.util.display.status_iterator' instead. Check CHANGES for Sphinx API modifications. status_iterator = sphinx.util.status_iterator -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================================= short test summary info ================================================================================== FAILED tests/integration/cli/test_execute.py::test_execute_r - AttributeError: outputs FAILED tests/functional/others/test_pytest.py::test_run_pytest_ok - SystemExit: 4 ============================================================ 2 failed, 3123 passed, 88 skipped, 6 warnings in 563.72s (0:09:23) ============================================================ ```
github-actions[bot] commented 7 months ago

Thank you for making this pull request.

Did you know? You can try it on Binder: Binder:lab or Binder:notebook.

Also, the version of Jupytext developed in this PR can be installed with pip:

pip install git+https://github.com/jsquaredosquared/jupytext.git@main

(this requires nodejs, see more at Developing Jupytext)

mwouts commented 7 months ago

Hey @jsquaredosquared , thank you for this!

Oh I see that the CI is not passing neither, possibly a pytest update. It may take me a few days to fix that, can I come back to you next week? Long story short, this does not seem related at all to your change. Meanwhile feel free to share the command above if anyone wants to test this development version.

Re you PR, can you revert the change on yarn.lock? Thanks!

jsquaredosquared commented 7 months ago

Thanks Marc!

I have reverted the change as requested.

Looking forward to your fix :D Jeff

codecov[bot] commented 5 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 97.75%. Comparing base (fe2ba5e) to head (1997365).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1213 +/- ## ========================================== + Coverage 97.69% 97.75% +0.06% ========================================== Files 29 29 Lines 4464 4464 ========================================== + Hits 4361 4364 +3 + Misses 103 100 -3 ``` | [Flag](https://app.codecov.io/gh/mwouts/jupytext/pull/1213/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Marc+Wouts) | Coverage Δ | | |---|---|---| | [external](https://app.codecov.io/gh/mwouts/jupytext/pull/1213/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Marc+Wouts) | `75.08% <ø> (ø)` | | | [functional](https://app.codecov.io/gh/mwouts/jupytext/pull/1213/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Marc+Wouts) | `88.51% <ø> (ø)` | | | [integration](https://app.codecov.io/gh/mwouts/jupytext/pull/1213/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Marc+Wouts) | `77.28% <ø> (ø)` | | | [unit](https://app.codecov.io/gh/mwouts/jupytext/pull/1213/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Marc+Wouts) | `66.56% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Marc+Wouts#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

mwouts commented 5 months ago

Thank you @jsquaredosquared for your PR! And sorry for the delay in taking it. I will release a new version shortly.

jsquaredosquared commented 5 months ago

Thanks for accepting it! 😀 And thanks for your work on jupytext