search

LLNL / ATS

ATS - Automated Testing System - is an open-source, Python-based tool for automating the running of tests of an application across a broad range of high performance computers.
BSD 3-Clause "New" or "Revised" License
7 stars 5 forks source link

Project install option broken? #93

Closed jwhite242 closed 1 year ago

jwhite242 commented 2 years ago

Not sure what the root of the issue is here given i can import the module in question manually in the interpreter, but using the project install option is currently not very happy. I used the python 3.8.2 module on the LLNL toss3 machines so you can hopefully reproduce it? Anyway, cloning, then attempting to install into an alternate directory gives this enormous stack pointing to something that's definitely built in to the python install. Following the steps on the readme leads to:

$ git clone git@github.com:LLNL/ATS.git ats3_source

$ python3 -m pip install ats3_source/ --target=ats3/

Processing ./ats3_source
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 381, in run
    resolver.resolve(requirement_set)
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 201, in resolve
    self._resolve_one(requirement_set, req)
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 365, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 312, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 223, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 49, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_internal/distributions/source/legacy.py", line 37, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_internal/distributions/source/legacy.py", line 90, in _setup_isolation
    reqs = backend.get_requires_for_build_wheel()
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 151, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 255, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/subprocess.py", line 64, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 63, in _build_backend
    obj = import_module(mod_path)
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/white242/pip-build-env-uf448wdc/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
  File "/tmp/white242/pip-build-env-uf448wdc/overlay/lib/python3.8/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/tmp/white242/pip-build-env-uf448wdc/overlay/lib/python3.8/site-packages/_distutils_hack/__init__.py", line 77, in do_override
    ensure_local_distutils()
  File "/tmp/white242/pip-build-env-uf448wdc/overlay/lib/python3.8/site-packages/_distutils_hack/__init__.py", line 63, in ensure_local_distutils
    core = importlib.import_module('distutils.core')
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/tmp/white242/pip-build-env-uf448wdc/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 17, in <module>
    from distutils.dist import Distribution
  File "/tmp/white242/pip-build-env-uf448wdc/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 19, in <module>
    from distutils.util import check_environ, strtobool, rfc822_escape
  File "/tmp/white242/pip-build-env-uf448wdc/overlay/lib/python3.8/site-packages/setuptools/_distutils/util.py", line 11, in <module>
    import subprocess
  File "/usr/tce/packages/python/python-3.8.2/lib/python3.8/subprocess.py", line 69, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

pip installing into a virtualenv seems to work ok still, but wanted to try out this option to compare the workflow differences for everyone (unless that's not supported anymore, which is fine too).

Much appreciate any help sorting this (and/or me) out!

dawson6 commented 2 years ago

Hi @jwhite242 and @davidbloss

David, can you help Jeremy here. That installation method is not the one I normally use for a public install of ATS. I believe Jeremy is working off of your install notes.

Jeremy, python 3.8.2 is the correct python to use, so that is good.

However, the 'main' branch of ATS is in a state of flux. Can you use the 'python3' branch which is the latest python3 stable release. IE, can you add a

git checkout python3

To your method after you do the clone.

This doesn't solve the issue you are seeing (I confirm I saw the something similar when I attempted your commands)

Jeremy, here are commands I would do for a local install, based on what you posted, But would like @davidbloss to check in here with his input on the best way to do this.

git clone git@github.com:LLNL/ATS.git ats3_source cd ats3_source git checkout python3 cd .. python3 -m virtualenv --system-site-packages --python=python3.8 'pwd'/ats3_install ./ats3_install/bin/python -m pip install 'pwd'/ats3_source

In the above there are backticks around the pwd. The above may get you going, I am not sure of the impact of my pip command versus yours with the 'pip install' option.

dawson6 commented 2 years ago

Also, if you need help installing the ATSLLNL extensions, chat me up please.

dawson6 commented 2 years ago

@davidbloss Can you review the install instructions, particularly on RZVernal. Left message in teams. But my prior 'virtualenv' and 'pip install' methods are not working on rzvernal.

jwhite242 commented 1 year ago

Hey Shawn, finally got back into this and got to a point where those missing extensions are the holdup instead of python2-3 porting of our other bits. So where do I get those LLNL extensions to start exercising the schedulers/machines?

dawson6 commented 1 year ago

Hi Jeremy

https://lc.llnl.gov/gitlab/dawson/atsllnl ssh://git@czgitlab.llnl.gov:7999/dawson/atsllnl.git

And hang on, I am preparing a presentation for SWING and am currently documeting how to install the python3 version of ATS in a private space.

I have verified it works for the core ATS in this repo. I am now verifying it works with the extensions. Will get back to you in a bit with how I do this.

MishaZakharchanka commented 1 year ago

@jwhite242 Hey Jeremy, I began looking at this issue and was able to recreate it. Just wanted to double check that this is something that needs attention and isn't potentially fixed in one of your incoming PRs.

jwhite242 commented 1 year ago

So, it wasn't really urgent or anything, I was just trying that option because it was there in the docs and wanted to see what the workflow difference might be. If that's not an option you all want to support in ATS that's totally fine; I've been successfully using it just as a normal package in a virtual environment for some time now and can certainly continue on that way.

Edit: also, no, I've not tackled this issue in that set of PR's.

dawson6 commented 1 year ago

If this is not a necessary install method, and a user can install and test in other ways, we can just update the documentation and close this out. Energy better spent elsewhere.