ModuleNotFoundError

[kdschlosser]

Description

This has been brought up before and it has been dismissed because of the pyproject.toml file not being used. This is NOT the case with the problem I am having. If you have a pyproject.toml and you have a setup.py and then you add another python source file to keep the code tidy and organized like what should be done when installing using 'pip install .' or if the package is on Pypi doing pip install package_name the build fails with a ModuleNotFoundError when that source file gets imported. I have worked around this problem on Windows by adding the folder the setup.py file resides in to sys.path. This does not work on Linux. The ModuleNotFoundError still occurs.

Expected behavior

To be able to import local source files that get used during the build/install of a library

pip version

23.1.2

Python version

3.8.10, 3.10.2, 3.11.3

OS

Windows, Linux, OSX

How to Reproduce

I have attached a test module so you can see what is happening. extract it go into that folder and install the module using pip install .

pip_failure.zip

Output

Here is the exact error

\pip_failure>pip install .
Processing \pip_failure
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      Traceback (most recent call last):
        File "\python38\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "\python38\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "\python38\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "\Temp\pip-build-env-j604154n\overlay\Lib\site-packages\setuptools\build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "\Temp\pip-build-env-j604154n\overlay\Lib\site-packages\setuptools\build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "\Temp\pip-build-env-j604154n\overlay\Lib\site-packages\setuptools\build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 5, in <module>
      ModuleNotFoundError: No module named 'some_module'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

This error also occurs if the library is hosted on pypi.

Code of Conduct

• [X] I agree to follow the PSF Code of Conduct.

[uranusjr]

This should be reported to setuptools instead.

[kdschlosser]

The thing is I am not sure how setuptools is going to be able to handle it. The way the environment is set up is the responsibility of PIP in this case because it is the one that is calling subprocess.

I did open an issue with setuptools and we will see what they say over there. I don't expect them to respond to the issue. Past experience of them not responding to a single issue I have opened is what makes me think that.

[kdschlosser]

and if you don't mind opening this issue back up because it has not been determined if pip is the cause or if setuptools is. pip has the responsibility of setting up the environment properly. It has not been determined if that is being done right or not.

[pfmoore]

OK, so you do import some_module in your setup.py. The first question is whether setuptools guarantees, when called via the build backend hooks, that the directory containing setup.py is on sys.path. The PEP 517 specification itself does not require that, so it's up to setuptools as a backend to ensure that if they want to support users importing modules in the project directory in the setup.py script.

The alternative is that you can ensure this yourself in your setup.py by adding the project directory to sys.path manually (you can get this directory via __file__). This is what you need to do if setuptools doesn't ensure it for you.

The latter is the fix you should use now, regardless. If setuptools accept that it's a bug that they don't set sys.path, then you may be able to remove that fix when the bug gets fixed, but if setuptools take the view that you can't assume that, then you need the fix permanently to support your usage.

So either way, this still isn't a bug in pip, which is conforming to PEP 517.

[kdschlosser]

again as I have already said. adding the directory to sys.path works but only for Windows builds. The issue is if a user gets the sdist from pypi. Pip sets the working directory of the subprocess to the current working directory. so unless a user is in the same folder as the setup.py (which would be impossible to do if getting the sdist from pypi using pip) the information that is passed to the build backend is incorrect. I was not able to locate the code in pip that sets the cwd to the place where the sdist was extracted into.

I did find the place where pip collects the cwd to be passed to the subprocess call. whether or not this is the correct cwd on Linux I am not able to tell you that.

PIP has the aability to pass information to the build backend using a JSON file. there are no settings that are passed. so pip has the capacity to pass proper directory information to the build backend but it doesn't.

so as I have said before, the problem could be because of pip setting an incorrect cwd or because pip doesn't sent any config data concerning paths. All I can tell you is when I run python setup.py bdist_wheel everything works fine. This indicates to me that there is not an issue with setuptools because all setuptools does is run the setup.py file when being used as the build backend.

IDK why there is a separation between pip and setuptools at all. This adds a mess of finger pointing as to where a problem is and nothing gets fixed. both setuptools and pip alike both are under the pypa umbrella and as such this problem only happens when using pip which makes it the responsibility of the people working on pip to get it fixed which means the people working on pip need to contact the people that are working on setuptools to get the problem resolved. Telling a user to go and report the problem elsewhere is bad support, really bad support.

I don't care about PEP 517 whatever that is. All I know is when I use setuptools to perform the build it works without an issue. when I use pip it doesn't what goes on between pip and setuptools is not something I care about and nor do I want to spend the time to fix someones coding mistakes.

I have a question. How do you know that the problem is not pip? If you are 100% sure it is not pip then you most know where the problem is in setuptools that is causing the issue. If you know that information why has it not been given to setuptools so they can correct the problem? My guess is because you don't know where in setuptools the problem is and if you don't know that then you don't know if the problem is in the setuptools code. Which means you do not know if pip is the cause of the issue or not. You are basing your finger pointing at setuptools because that is where the error is being generated from. Just because setuptools is throwing the error doesn't mean the problem isn't upstream of it.

I am sure you have heard of many people complaining about packaging in Python. This is a perfect example of why those complaints exist. I am hoping that someone will bring this specific conversation up in some kind of a discussion about why there are so many issues with packaging in Python, Finger pointing is NOT a solution.

If someone posts an issue for one of my repositories and the issue is because of a library that is used I go and open an issue for that library. I find what the exact issue is and I tell the people that work on that library where the issue is coming from.

Could you imagine if every person that used pip for packaging turned around and redirected their users to open issues here when pip throws an error?? Holy cow you guys would have millions of open issues. millions and millions and millions of issues. You would not even know what to do about it..

You admit there is a problem. so do what needs to be done and get it fixed.

[kdschlosser]

Oh and just to let you know.. This works.

python -m pip install .

Why is the question of the day,.

[pfmoore]

You admit there is a problem. so do what needs to be done and get it fixed.

I can't. The change needs to be in your code. You add the lines

import os, sys

sys.path.append(os.path.dirname(os.path.abspart(__file__))

to the top of your setup.py, before the import.

Or you discuss this with the setuptools team. They may tell you you've hit a bug with their software and be willing to fix it. This is not a pip issue, there's nothing in the standards that requires pip to set a particular CWD when invoking a build backend, so there simply is no pip bug here.

How do you know that the problem is not pip? If you are 100% sure it is not pip then you most know where the problem is in setuptools that is causing the issue.

I do. But you're wrong to say "in setuptools". The problem is because your code is making an assumption (the cwd) that setuptools doesn't guarantee for you. What I cannot do, as I'm not a setuptools developer, is decide whether that's your mistake (because the guarantee was never intended) or theirs (because they did intend to provide that guarantee, but failed to do so).

You are basing your finger pointing at setuptools because that is where the error is being generated from.

Quite frankly, that's just insulting. And at this point I'd remind you that we have a code of conduct and you're getting terribly close to violating it. I have many years of expertise, both with pip and with setuptools, and I am trying to help you understand what the problem is that you're hitting. If you don't want the help, then fine, I'll leave you to it. If you do want help, I'd ask that you respect the experience of the people trying to help you.

You keep demanding "support" here. You're not paying anything for these tools, and the maintainers are all volunteers. We owe you nothing. This issue is closed, the problem is not with pip and you should look elsewhere for help at this point. I've spent as much of my free time trying to help you as I'm willing to.

[kdschlosser]

I get it now why people hate dealing with python packaging. My point is that pip sets the cwd and setuptools uses what has been passed to it. You have done ZERO to look into if the cwd is being properly set. the fact that this doesn't work pip install . even when setting sys.path before setuptools is even imported and this does work python -m pip install . doesn't make you question where the problem might be coming from? setuptools has ZERO knowledge of the above so why does the behavior change? something with pip is why it changes. it could be something as simple as the entry point and something being done differently one way that is not being done in the other. lord knows pip is far from perfect code and is riddled with issues. That makes me confused as to why you believe that the code cannot possibly have a single flaw in it that could cause this problem. Whether it be pip or setuptools it is going to force users to lump all of their build code into a single file? That is just plain crazy!!!!

I am going to instruct every user that has this error (which is all of them that run on Linux) to come here and and create issues. Doing exactly what you have just done because after all it's not my software that has the problem which is your standpoint. the problem is stemming from running pip so this is where they need to create issues and then you can tell each and every single one of the users to open an issue with setuptools. I can pass the buck too.

[notatallshaw]

I am not a Pip or Setuptools maintainer so I have no skin in the game so I'm going to provide some criticism to your response trying to be objective and factual.

I get it now why people hate dealing with python packaging.

This is an emotional attack on a project where so far participants have only tried to help you resolve your issue.

You will find in general people are going to be less helpful not more helpful when you attack a project they are providing volunteer support for.

My point is that pip sets the cwd and setuptools uses what has been passed to it.

Have you considered that your approach does not work with how I stallers and build systems are defined to work with each other?

If this technical issue is the crux of your problem perhaps try objectively asking if this is in fact required or an implemention detail by Pip before emotionally attacking the project.

That isn't rhetorical either, I honestly don't know the answer, I have never looked at the spec or the code.

My point is that pip sets the cwd and setuptools uses what has been passed to it. You have done ZERO to look into if the cwd is being properly set. the fact that this doesn't work pip install . even when setting sys.path before setuptools is even imported and this does work python -m pip install .

This is extremely difficult to follow your thought process. The other participant in this thread has provided you a workaround and given you their analysis. As best as I can tell you are wilfully ignoring it because you think that Pip should support you for your use case when there is no such agreement or contract to do so.

I am going to instruct every user that has this error (which is all of them that run on Linux) to come here and and create issues.

This called brigading, this is a form of online harassment. It is clearly a violation of both GitHubs terms of service and this projects code of conduct. I am stating this factually as I have no power to enact any response, I'm just letting you know that you have broken the agreement you acknowledged to participate in this space.

The social impact of making threats like this is to create an unconstructive and often toxic environment where at best everyone but yourself wants to stop participating and at worst contributes to causing a breakdown in any support of the product you're trying to use to anyone.

Doing exactly what you have just done because after all it's not my software that has the problem which is your standpoint

The person who has responded to you has given you a workaround to get your code working and a clear explanation of the issue.

What they have done is assist you and been willing to have a dialogue even though in their opinion this project is not what is causing what you think is an issue but instead it's the behavior of the downstream project in the way it's been agreed to interact with that downstream project.

the problem is stemming from running pip so this is where they need to create issues and then you can tell each and every single one of the users to open an issue with setuptools. I can pass the buck too.

You can either choose to be purposefully ignorant and actively harmful to the open source ecosystem or you can choose to interact with people in a constructive way and help out the open source ecosystem.

No one is paying anyone to provide software here, there is no company maintaining all these projects, if it does not meet your expectations there is no service contract you can invoke.