zanieb
commented
6 months ago Interesting thanks I've never seen this I'll try to reproduce.
Can you packse build the scenarios separately from serve?
Closed notatallshaw closed 6 months ago
zanieb
commented
6 months ago Interesting thanks I've never seen this I'll try to reproduce.
Can you packse build the scenarios separately from serve?
zanieb
commented
6 months ago Sorry I can't reproduce this on Arch Linux or macOS.
notatallshaw
commented
6 months ago Sorry, I've been unable to work on OSS for a few days, as soon as I get a chance I'll create a clean environment and try and reproduce.
notatallshaw
commented
6 months ago Still reproducible in a clean environment. I'm using pyenv because it compiles directly from CPython and, at least with pip, installer tests can be sensitive to changes different distributers make.:
pyenv install 3.12 (And fix any compiler errors if there are any)pyenv virtualenv uv_testing_312pyenv activate uv_testing_312pip install packse[serve]packse fetchpackse build scenarios/Trackeback:
Traceback (most recent call last):
File "/home/damian/.pyenv/versions/uv_testing_312/bin/packse", line 8, in <module>
sys.exit(entrypoint())
^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/cli.py", line 44, in entrypoint
args.call(args)
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/cli.py", line 76, in _call_build
build(
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/build.py", line 88, in build
results = [future.result() for future in futures]
^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/build.py", line 179, in build_scenario
future.result()
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/build.py", line 232, in build_scenario_package
package_destination = create_from_template(
^^^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/template.py", line 64, in create_from_template
chevron_blue.render(file_path.read_text(), scope, no_escape=True)
^^^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/pathlib.py", line 1028, in read_text
return f.read()
^^^^^^^^
File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcb in position 0: invalid continuation byteI put a breakpoint around line 64, in create_from_template, and I found these values:
(Pdb) file_path
PosixPath('/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/templates/package/{{ package-name }}-{{ version }}/src/{{ module-name }}/__pycache__/__init__.cpython-312.pyc')
(Pdb) scope
{'scenario-name': 'local-used-without-sdist', 'package-name': 'local-used-without-sdist-ea57116a', 'module-name': 'local_used_without_sdist_ea57116a', 'version': '0.0.0', 'dependencies': [<Requirement('local-used-without-sdist-a-ea57116a==1.2.3')>], 'optional-dependencies': [], 'requires-python': '>=3.8', 'description': 'Even if there is a 1.2.3 version published, if it is unavailable for some reason (no sdist and no compatible wheels in this case), a 1.2.3 version with a local segment should be usable instead.'}
(Pdb) new_file_path
PosixPath('/home/damian/testing_remote/packse/build/local-simple-5b70bb3c/local-simple-5b70bb3c-0.0.0/src/local_simple_5b70bb3c/__pycache__/__init__.cpython-312.pyc')
(Pdb) file_path.read_bytes()
b'\xcb\r\r\n\x00\x00\x00\x00\xa1L\xf2e\x1e\x00\x00\x00\xe3\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\xf3\x08\x00\x00\x00\x97\x00d\x00Z\x00y\x01)\x02z\r{{ version }}N)\x01\xda\x0b__version__\xa9\x00\xf3\x00\x00\x00\x00\xfa\xb0/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/templates/package/{{ package-name }}-{{ version }}/src/{{ module-name }}/__init__.py\xfa\x08<module>r\x06\x00\x00\x00\x01\x00\x00\x00s\n\x00\x00\x00\xf0\x03\x01\x01\x01\xd8\x0e\x1d\x81\x0br\x04\x00\x00\x00'Thanks for the additional details! If you just git clone this repository and run packse build scenarios do you get the same error?
I'll attempt to reproduce again but I was using pyenv on Linux so I think we're pretty close to the same environment.
notatallshaw
commented
6 months ago Okay, I found a differentiator or it working vs. not working, if I install packse from either pypi or source directory then it fails, regardless of whether I packse fetch the scenarios or git clone the scenarios.
However, it works fine if I git clone packse source and then pip install in editable mode.
git clone https://github.com/astral-sh/packsecd packse/pip install .[serve] --forcepackse build --force scenarios/Output:
Building 'dependency-excludes-non-contiguous-range-of-compatible-versions-5d038b4f' in directory '/home/damian/testing_remote/packse/build/dependency-excludes-non-contiguous-range-of-compatible-versions-5d038b4f'
Traceback (most recent call last):
File "/home/damian/.pyenv/versions/uv_testing_312/bin/packse", line 8, in <module>
sys.exit(entrypoint())
^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/cli.py", line 44, in entrypoint
args.call(args)
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/cli.py", line 76, in _call_build
build(
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/build.py", line 88, in build
results = [future.result() for future in futures]
^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/build.py", line 179, in build_scenario
future.result()
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/build.py", line 232, in build_scenario_package
package_destination = create_from_template(
^^^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/envs/uv_testing_312/lib/python3.12/site-packages/packse/template.py", line 63, in create_from_template
chevron_blue.render(file_path.read_text(), scope, no_escape=True)
^^^^^^^^^^^^^^^^^^^^^
File "/home/damian/.pyenv/versions/3.12.2/lib/python3.12/pathlib.py", line 1028, in read_text
return f.read()
^^^^^^^^
File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcb in position 0: invalid continuation bytepip install -e .packse build --force scenarios/Not looked into why that's happening though.
notatallshaw
commented
6 months ago Stepping back a little bit, it seems that this function is trying to read templated source files and write out concrete source files.
For whatever reason in my environment it ends up reading pyc files, I don't think it should be. I think it would be reasonable to have an exclude (pyc files and __pycache__ directories) filter or an include filter (only py files?).
If I change:
for root, _, files in template_path.walk():To:
for root, dirs, files in template_path.walk():
with contextlib.suppress(ValueError):
dirs.remove("__pycache__")And reinstall, I no longer get an error.
Have you been install with uv pip install rather than pip install? Does perhaps one generate bytecode by default and the other doesn't?
zanieb
commented
6 months ago Ah pip install does compile byte-code on install default whereas uv pip install requires opt-in! Nice catch!
We should definitely filter those files from the walk.
notatallshaw
commented
6 months ago It looks like poetry doesn't compile by default either and you need to pass --compile to the install command.
I could raise a PR to add this to the CI test environment and update the code as above, if you'd like?
Steps to Reproduce:
Output:
Output: