search

jupyterlite / xeus

JupyterLite loader for Xeus kernels
https://jupyterlite-xeus.readthedocs.io
BSD 3-Clause "New" or "Revised" License
22 stars 12 forks source link

Build fails with "Object of type PosixPath is not JSON serializable" #113

Open michaelweinold opened 2 months ago

michaelweinold commented 2 months ago

Description

When adding our bw2io package to the environment.yaml as a dependency, I get an error of this kind:

TypeError: Object of type PosixPath is not JSON serializable

Interestingly, the error does not reference the package, nor any of its dependencies. The error trace goes from the micromamba build library to jupyterlite_core/manager > jupyterlite_xeus/add_on > empack/pack and then to the Pyhon JSON decoder.

Any idea what the cause might be?

Reproduce

  1. Run build workflow at michaelweinold/jl_xeus_test

Expected behavior

No error 😇

Context

Build Log 
Transaction finished

To activate this environment, use:

    micromamba activate xeus-python-kernel

Or to execute a single command in this environment, use:

    micromamba run -n xeus-python-kernel mycommand

[LiteBuildApp] ERROR | [lite] [post_build] [jupyterlite-xeus] [ERR] Object of type PosixPath is not JSON serializable
Traceback (most recent call last):
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/doit_cmd.py", line 294, in run
    return command.parse_execute(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/cmd_base.py", line 150, in parse_execute
    return self.execute(params, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/cmd_base.py", line 570, in execute
    return self._execute(**exec_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/cmd_run.py", line 265, in _execute
    return runner.run_all(self.control.task_dispatcher())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/runner.py", line 254, in run_all
    self.run_tasks(task_dispatcher)
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/runner.py", line 213, in run_tasks
    node = task_dispatcher.generator.send(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/control.py", line 629, in _dispatcher_generator
    next_step = node.step()
                ^^^^^^^^^^^
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/control.py", line 336, in step
    return next(self.generator)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/control.py", line 345, in _func
    for value in decorated(*args, **kwargs):
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/control.py", line 473, in _add_task
    new_tasks = generate_tasks(to_load, task_gen, ref.__doc__)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/loader.py", line 390, in generate_tasks
    for task_dict, x_doc in flat_generator(gen_result, gen_doc):
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/doit/loader.py", line 27, in flat_generator
    for item in gen:
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_core/manager.py", line 138, in _delayed_gather
    yield from _gather()
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_core/manager.py", line 131, in _gather
    raise error
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_core/manager.py", line 123, in _gather
    for task in getattr(addon, attr)(self):
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_xeus/add_on.py", line 127, in post_build
    yield from self.copy_kernels_from_prefix()
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_xeus/add_on.py", line 179, in copy_kernels_from_prefix
    yield from self.copy_kernel(kernel_dir, kernel_wasm, kernel_js)
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_xeus/add_on.py", line 261, in copy_kernel
    yield from self.pack_prefix(kernel_dir=kernel_dir)
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/jupyterlite_xeus/add_on.py", line 288, in pack_prefix
    pack_env(
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/empack/pack.py", line 186, in pack_env
    pack_pkg_impl(
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/site-packages/empack/pack.py", line 54, in pack_pkg_impl
    json.dump(pkg_meta, f)
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/encoder.py", line 432, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/encoder.py", line 406, in _iterencode_dict
    yield from chunks
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/encoder.py", line 326, in _iterencode_list
    yield from chunks
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/encoder.py", line 439, in _iterencode
    o = _default(o)
        ^^^^^^^^^^^
  File "/home/runner/micromamba/envs/build-env/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type PosixPath is not JSON serializable
michaelweinold commented 2 months ago

And @martinRenou, I guess you have some bots to block (see above) 🙈

michaelweinold commented 2 months ago

Since the main error seems to originate with this line:

conda_meta_filename = f"{filename_base_from_meta(pkg_meta)}.json"
    with open(filtered_prefix / "conda-meta" / conda_meta_filename, "w") as f:
        json.dump(pkg_meta, f)

Did we (in our Brightway packages) use some non-standard package name?

DerThorsten commented 2 months ago

I am a bit surprised by this error. It seems like a missing str(thePath) somewhere. But I would not expect this to happen for certain packages only.

michaelweinold commented 2 months ago 
from empack import filter_env

path_env = "/opt/homebrew/Caskroom/miniconda/base/envs/xeus-python-kernel-test"

list_pkg_meta = [pkg_meta for pkg_meta in filter_env.iterate_env_pkg_meta(path_env)]

for pkg_meta in list_pkg_meta:
    tempfile = "/tmp/tmp.json"
    with open(tempfile, 'w') as file:
        json.dump(pkg_meta, file)

works fine on my local machine. Weird 🙈

michaelweinold commented 2 months ago

I found the bug - the issue originates from the iterate_pip_pkg_record() function in combination with the better_exceptions_fork package.

Running iterate_pip_pkg_record() while passing it the path to the temporary directory which is generated by the jupyter lite build command:

from empack import filter_env
[pip_pkg for pip_pkg in filter_env.iterate_pip_pkg_record("(...)/tmpfwwgtj1q/env/envs/xeus-python-kernel")]

returns this dictionary:

{'name': 'better_exceptions_fork',
 'version': '0.2.1.post6',
 'files': [PosixPath('lib/python3.11/site-packages/better_exceptions/__init__.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/__init__.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/__main__.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/__main__.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/color.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/color.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/formatter.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/formatter.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/highlighter.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/highlighter.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/log.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/log.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/repl.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions/repl.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/__init__.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/__init__.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/__main__.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/__main__.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/color.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/color.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/formatter.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/formatter.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/highlighter.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/highlighter.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/log.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/log.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/repl.py'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_fork/repl.pyc'),
  PosixPath('lib/python3.11/site-packages/better_exceptions_hook.pth')],
 'fn': 'better_exceptions_fork-0.2.1.post6',
 'build': 'pip',
 'build_number': 0,
 'depends': []}

...and that cannot be serialized to JSON.

I'll see if I can figure out how to mitigate this.

michaelweinold commented 2 months ago

Why is better_exceptions_fork even installed into the environment? It is not a dependency of bw2io or xeus-python - and apparently also not of any of the packages in build-environment.yml. Odd that it would not be there if I just use the standard recipe of the template:

name: xeus-python-kernel
channels:
  - https://repo.mamba.pm/emscripten-forge
  - conda-forge
dependencies:
  - xeus-python
  - numpy
  - matplotlib

...but only appears if I add bw2io.