FileNotFoundError while using the "python -m build" command

anusonawane commented 1 year ago

OS version

Ubuntu 22.04.2 LTS

Python version

python=3.8

Pip version

pip= 23.2.1

Guide link

https://packaging.python.org/en/latest/tutorials/packaging-projects/

Problem description

I encountered an error during the build process using the "python -m build" command. This error seems to be related to the absence of an output.json file in the /tmp directory. I'm providing a detailed description of the problem below.

Problem Statement:

I'm attempting to build a package using the python -m build command, as I want to avoid the direct invocation of setup.py using build backend & build frontend
I'm not using a pyproject.toml in the current directory.
The build process involves the Cythonization of code.

Additional Information:

The error message suggests that there might be a problem with temporary file generation or handling.
The error does not occur when using the traditional python setup.py build bdist_wheel --prod command.
I also tried executing the "python -m build" command in final_module dir but encountered the same error.(pyproject.toml is not present )

Package Structure:

my_project/
├── package
│   ├── pyproject.toml
          ( Including build-backend 
              [build-system]
              requires = ["setuptools>=68.0.0", "cython", "toml"]
              build-backend = 'setuptools.build_meta'
           )
│   ├── build.py( Includes cythonization, name of package passed to build()=package)
│   ├── script.py(It runs the "python -m build" command to build the package.)
│   └── module1/
│       └── module2/
│           └── final_module/
│               ├── setup.py
│               └── module_files.py

Error message

Traceback (most recent call last):
  File "/anaconda3/envs/setup/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 19, in read_json
    with open(path, encoding='utf-8') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpe61goaws/output.json'

ERROR Backend operation failed: FileNotFoundError(2, 'No such file or directory')

### Tasks

abravalheri commented 1 year ago

Hi @anusonawane what is your setup.py? Does it have a sys.exit(0) statement on it?

anusonawane commented 1 year ago

Hi @abravalheri thanks for reply,

In setup.py I have setup() containing metadata & the packaging name is final_module. Also I tried to add sys.exit(0) at end of file, it gave same error again.

abravalheri commented 1 year ago

Also I tried to add sys.exit(0) at end of file, it gave same error again.

Usually this kind of errors appear if you have sys.exit(0) in setup.py (because it will interrupt the Python interpreter and prevent setuptools and build to finish running...). But if you don't have sys.exit, it is a different problem and I don't have a quick solution to offer you :P

In setup.py I have setup() containing metadata & the packaging name is final_module.

@anusonawane, I think that the best way for us to help you would be if you provide a complete example (i.e. an example project with explicit code) that results in the error you are descripting. It would be practically impossible (or at least super hard) to investigate more without this kind of reproducer.

Please note that you don't need to share your original code (and actually it is better if you don't). Instead the best is if you could create a minimal reproducer for the error you are seeing, removing everything that is not relevant for the problem.

You can find more details about creating a minimal reproducer in https://stackoverflow.com/help/minimal-reproducible-example.

henryiii commented 1 year ago

Note that there are several ways to spell sys.exit(0), including raise SystemExit(0). Do you see anything in the setup.py that looks like it might exit early, before setuptools is finished with setup()?

Also, you should not depend on toml. It's an old, abandoned library. You should use tomli on Python 3.10 and older, and tomllib (built-in) on Python 3.11+.

anusonawane commented 1 year ago

Hi @abravalheri, @henryiii thanks for the suggestion.

Here I'm attaching the minimal code to reproduce the issue

Command to execute- /Package$ python script.py

* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
* Getting build dependencies for sdist...
running egg_info
creating final_module.egg-info
writing final_module.egg-info/PKG-INFO
writing dependency_links to final_module.egg-info/dependency_links.txt
writing top-level names to final_module.egg-info/top_level.txt
writing manifest file 'final_module.egg-info/SOURCES.txt'
reading manifest file 'final_module.egg-info/SOURCES.txt'
writing manifest file 'final_module.egg-info/SOURCES.txt'

Traceback (most recent call last):
  File "/anaconda3/lib/python3.9/site-packages/pyproject_hooks/_impl.py", line 19, in read_json
    with open(path, encoding='utf-8') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpcpt9w1wj/output.json'

ERROR Backend operation failed: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "Documents/Package/script.py", line 53, in <module>
    build_package_with_setup("module1/module2/final_module")
  File "Documents/Package/script.py", line 30, in build_package_with_setup
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command 'python -m build' returned non-zero exit status 0.

abravalheri commented 1 year ago

Hi @anusonawane, please find the steps I used to debug the example below (as bash comments):

> docker run --rm -it python:3.8-bullseye /bin/bash
wget https://github.com/pypa/packaging-problems/files/12247364/My_Package.zip -P /tmp
unzip /tmp/My_Package.zip -d /tmp
cd /tmp/Package

# Installing tools for building
python -m venv .venv
.venv/bin/python -m pip install 'pip==23.2.1' build

# Now, let's try to build the setuptools project on its own, to see if there is something wrong there
cd module1/module2/final_module

# First we remove the problematic `sys.exit(0)`
sed -i '/sys.exit/d' setup.py
# Then we run `build`
/tmp/Package/.venv/bin/python -m build
# ...
# Successfully built final_module-1.0.0.tar.gz and final_module-1.0.0-py2.py3-none-any.whl
# ^^^^^^^^^^^--- > No problems here, it seems that the setuptools build is OK

# Let's clean this build (and remove all cached files) so we can try again with the user crafted script
rm -rf dist build *.egg-info

# Trying again
cd /tmp/Package
.venv/bin/python script.py
# ...
# File "script.py", line 30, in build_package_with_setup
# ...
# subprocess.CalledProcessError: Command 'python -m build' returned non-zero exit status 0

# Let's fix the custom script to work with virtual environment
sed -i '1i import sys' script.py
sed -i 's/"python"/sys.executable/' script.py

# Now trying to run again
# ...
# Successfully built final_module-1.0.0.tar.gz and final_module-1.0.0-py2.py3-none-any.whl
# ^^^^^^^^^^^--- > No problems here, it seems that running `python script.py` is successful

It seems that the script can run without problem in a virtual environment, if:

I remove the sys.exit(0) that I mentioned earlier to be the usual culprit[^1].
I change the custom script.py to use sys.executable instead of "python" in subprocess.run (this is indicated most of the times and make sure the subprocess uses the same Python executable and site-packages as the ones script.py is using).

Note that in this issue tracker we can help you with packaging problems if they are related to pypa tools. So if you have problems running python -m build or pip install ., this would be the right place to ask. However if you have difficulties running custom build scripts, I would recommend first trying to build the relevant parts directly with python -m build as I did in the example above. If everything goes well but the custom script still fails, it is very likely that the custom script has some implementation problems, and in that case, it might be better/more effective to contact the original author of the script directly.

[^1]: The reason why people should not use sys.exit(0) is that in the light of the new packaging standards setup.py is only a configuration file and no longer a script that you can run directly with python setup.py as CLI. This direct invocation of setup.py is deprecated (you can read more about that in https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html). If someone adds sys.exit(0) to setup.py (or any equivalent as mentioned above by Henry), this means they expect python setup.py to be run directly, which it is no longer happening. sys.exit(0) will halt the execution of setuptools and then the execution of python -m build. A good comparison is pytest's configuration file conftest.py: you don't see people adding sys.exit(0) to conftest.py...

anusonawane commented 1 year ago

@abravalheri thanks for the guidance, by removing sys.exit(0) & egg-info it got resolved.

I have a few questions regarding the direct invocation of setup.py. As I'm in the process of making changes, here's what I've done so far: I've made alterations in pyproject.toml to include a build-backend. The rest of the setup seems to be following the conventional setuptools approach. However, there are some complications when considering the inclusion of Cythonization. As evidenced in the attached files, the process involves Cythonization, and when I execute the python -m build command, it indeed generates a Wheel (whl) file. Unfortunately, it appears that the code hasn't undergone the expected Cythonization.

Here's some additional information: I've added specifications for cythonizing files in include and exclude sections in the pyproject.toml file.

Guidance- https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html https://packaging.python.org/en/latest/tutorials/packaging-projects/

abravalheri commented 1 year ago

Here's some additional information: I've added specifications for cythonizing files in include and exclude sections in the pyproject.toml file.

If you can share a minimal reproducer including the cythonization, we can give you more targeted feedback. Meanwhile the following are my general comments:

As far as I can tell, you seem to be doing some form of subproject building... If you try to glue things together in a custom script, that can go south very easily. It is important to understand how python -m build works:
- Specifically, I noticed that you have a pyproject.toml file in the top-level directory that specifies cython as a build dependency. However, in the directory that you have setup.py you don't have any pyproject.toml. If you call python -m build in a directory without pyproject.toml it will create a brand new isolated virtual environment before asking setuptools to proceed with the build. This new virtual environment will not have cython installed, so if you try to use cython in your setup.py script you might have import errors.
Personally, instead of writing:
```
from setuptools import setup
from Cython.Build import cythonize

setup(
    name='hello',
    ext_modules=cythonize("hello.pyx"),
)
```
I prefer to write:
```
from setuptools import Extension, setup

setup(
    name='hello',
    ext_modules=[Extension(name="hello", sources=["hello.pyx"])],
)
```
because I know that setuptools will compile hello.pyx files automatically as long as Cython is installed (having a pyproject.toml file in the project directory with cython listed as build dependency is important). I prefer this way because it makes my life easier to administrate the contents of the sdist (other people might have different ideas). Optionally, I add setuptools-scm to my pyproject.toml to make sure all files in added to my git repository are added to the sdist.

anusonawane commented 1 year ago

Hi @abravalheri,

I want to ensure that the build backend and build frontend are built correctly, without any direct calls to setup.py. Could you possibly provide me with an example of structuring a project to achieve this goal?

And if I'm using poetry build command, using build-backend = 'setuptools.build_meta' in pyproject.toml is correct? or should i replace it with build-backend = "poetry.core.masonry.api"

Also if I run the python script.py command I'm getting FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpcpt9w1wj/output.json' error again, but if i run python -m build it executes. This output.json file is not getting created in env.

Also if you could take a quick look at my approach to see if I'm missing anything or making any glaring mistakes?

abravalheri commented 1 year ago

I run the python script.py command I'm getting FileNotFoundError ... but if i run python -m build it executes.

That is a clue that the tools seem to be doing their job correctly, but there might be problems in the gluing custom script...

I want to ensure that the build backend and build frontend are built correctly, without any direct calls to setup.py. Could you possibly provide me with an example of structuring a project to achieve this goal?

I am sorry, I will not be able to help with that. I don't have experience of monorepo builds and the usual suspects on the Python ecosystem (setuptools, poetry, pdm, hatch, build, etc...) usually don't cover that use case (AFAIK). You can try to use the APIs that the backends expose directly (e.g. by leveraging build's API on https://pypa-build.readthedocs.io/en/stable/api.html), but if you plan to implement a glue script yourself, I recommend a careful read of:

Alternatively you might try to find a tool that natively supports monorepos (if I am not wrong, pants claims to do that), or split them in standalone projects and define dependence relationships (that is the approach I usually take when I am in that scenario).

if I'm using poetry build command, using build-backend = 'setuptools.build_meta' in pyproject.toml is correct? or should i replace it with build-backend = "poetry.core.masonry.api"

I am not super familiar with poetry workflow. I believe it would be better to ask this question in a Poetry forum directly.

Also if you could take a quick look at my approach to see if I'm missing anything or making any glaring mistakes?

Please note my previous comment. If you can share a minimal reproducer that include all the significant steps you are taking/trying to achieve, we can give you more targeted feedback.

pypa / packaging-problems