Closed edwardchalstrey1 closed 1 year ago
This section of the SRD image building instructions says Please check that everything has built correctly before proceeding.
That wasn't done here, so we'll need to build a new image. Please can you do this and check the build logs before converting the candidate VM into an image? If there's a problem in the image building scripts we can debug that separately.
If there's a problem in the image building scripts we can debug that separately.
I've built the image gone through the whole process of creating a candidate VM and building an image from it twice now and had the same problem, so that's where the issue lies. Do you have some idea as to why Python might fail a safety check? my thoughts are this is the relevant line of the readout above:
[2023-06-23 09:48:33: WARNING] ... transformers [4.19.2] is affected by issue 51450 (for versions ['<4.23.0'])
transformers is on our tier3 allowlist - but I couldn't see a GitHub issue 51450 on the transformers GH if that's what this means?
but I guess you're suggesting I create another candidate VM and consult the /var/log/cloud-init-output.log
@jemrobinson ?
Yes, that's what I'm suggesting. That contains the full error log and you should be able to see where the error is. As you've already created and deployed a broken image, you can also look at /var/log/cloud-init-output.log
in the VM you've just deployed.
Seems like the error is related to Python2Supports
(see full error above), I'll look into this
cannot import name 'Python2Supports' from 'virtualenv.create.describe' (/opt/pyenv/versions/3.10.12/lib/python3.10/site-packages/virtualenv/create/describe.py)
Can you put a longer excerpt from the output log (or a link to the full log) here?
^ Looks like it may be related to transformers after all:
Vulnerability found in transformers version 4.19.2
It's not that. It's the exception that's thrown that the top of your log file extract
This is related to torch
. Recently torch
separated its GPU-enabled package from the CPU-enabled one. As we want one build to work on both CPU and GPU VMs the best solution is to drop torch from the list of required Python packages. Do you know where to make this change @edwardchalstrey1 ?
Ok, so remove torch from here: https://github.com/alan-turing-institute/data-safe-haven/blob/develop/deployment/secure_research_desktop/packages/packages-python.yaml
Is torchvision totally separate or should that be dropped too?
Yes, you'll need to drop both
^ this issue suggests we should downgrade poetry:
poetry>=1.3.0,<1.4.0
In our packages-pyton.yaml
we currently have:
poetry: # also used by installation scripts
"all": [">1.0.0"] # increase solver flexibility
Newest error - looks like we need to set the version of pystan
(crashtest
isn't something in our packages-pyton.yaml
)
You could try "all": [">=3.0"]
as the pystan
requirement. There was a major rewrite between v2 and v3 so we'd still like to exclude v2 if possible.
Further version/dependency problems
OK, so if poetry and pystan are incompatible then feel free to drop pystan from any of the Python versions where it isn't working. This is only the pre-installed list of packages - users can still install their own packages in anything less than a tier 4.
For example
pystan:
"3.9": ["uninstallable"] # incompatible with poetry < 1.4
That seems to have solved all Python issues, but there seems to be a problem with "Azure Data Studio" - not sure whether this is something we need in a SRD?
Can you see the first error in that log extract? What does it say? What do you think the solution might be?
Side question: Why do we support 3 different Ubuntu versions for the SRD?
How about this: https://github.com/alan-turing-institute/data-safe-haven/pull/1494/files#diff-36be5faed69d88cab6e29c016dcbc944921f617db1bd0c1605499715276d8041
It's a good idea but it probably won't work. What's happening here is that during the package_install
step (which happens before run_commands
) the VM couldn't update one or more of the Ubuntu repositories. This means that it didn't see a load of packages that it needed to install and so skipped them. Running apt-get update --fix-missing
will install missing dependencies of already-installed packages but it won't install the ones that we wanted which were skipped.
The best solution is just to rerun the deployment script and hope that the connection error was temporary (it usually is).
Side question: Why do we support 3 different Ubuntu versions for the SRD?
We started with 18.04
and switched to 20.04
when it became available. We've basically dropped support for 18.04
but the scripts are still there in case anyone needs them for some reason. We'd like to switch to 22.04
but several of the Azure features (like automatic updates) don't support 22.04
yet.
after removing pystan from all Python versions, I'm now back to the original 'Python2Supports' error:
Oh this is weird, it's trying to install torch
again
I'm going to try removing keras
and tsfresh
too, since these both have torch
as a dependency, which may be why it's still being installed
Latest error:
:white_check_mark: Checklist
:computer: System information
:cactus: Powershell module versions
:no_entry_sign: Describe the problem
When building an SRE from the
release-v4.0.4
branch, the smoke tests fail on all Python versions - inspecting the SRD build log reveals there was a problem with installing Python (see logs below).:deciduous_tree: Log messages
Failed Python 3.10 installation on candidate VM:
``` • Installing xgboost (1.7.6) ImportError cannot import name 'Python2Supports' from 'virtualenv.create.describe' (/opt/pyenv/versions/3.10.12/lib/python3.10/site-packages/virtualenv/create/describe.py) at /opt/pyenv/versions/3.10.12/lib/python3.10/site-packages/virtualenv/create/via_global_ref/builtin/python2/python2.py:6 inPython safety check failure during SRD build
```none Code : ProvisioningState/succeeded Level : Info DisplayStatus : Provisioning succeeded Message : Enable succeeded: [stdout] [2023-06-23 08:47:56: SUCCESS] Build started: [2023-06-23 08:48:54: SUCCESS] Running cloud-init modules: 57s [2023-06-23 09:17:00: SUCCESS] Installing/updating Ubuntu packages: 28m5s [2023-06-23 09:17:33: SUCCESS] Preparing system for installation: 32s [2023-06-23 09:19:51: SUCCESS] Installing additional deb/source packages: 2m18s [2023-06-23 09:24:33: SUCCESS] Updating ClamAV database: 4m42s [2023-06-23 09:25:05: SUCCESS] Installing Spark/Hadoop: 32s [2023-06-23 09:36:18: SUCCESS] Installing Julia and packages: 11m13s [2023-06-23 09:44:07: SUCCESS] Installing Python (3.9.17) and packages: 7m49s [2023-06-23 09:48:33: FAIL ] Installing Python (3.10.12) and packages: 4m26s [2023-06-23 09:48:34: FAIL ] Finished build: 1h0m37s [2023-06-23 09:48:33: INFO ] Memory available: 17 GB [2023-06-23 09:48:33: INFO ] ..... mean usage: 10.39% => 1.7 GB [2023-06-23 09:48:33: INFO ] ...... min usage: 5.38% => 0.9 GB [2023-06-23 09:48:33: INFO ] ...... max usage: 28.45% => 4.8 GB [2023-06-23 09:48:33: INFO ] CPU available: 8 cores [2023-06-23 09:48:33: INFO ] ..... mean usage: 26.19% => 2.1 cores [2023-06-23 09:48:33: INFO ] ...... min usage: 0.10% => 0.0 cores [2023-06-23 09:48:33: INFO ] ...... max usage: 100.00% => 8.0 cores [2023-06-23 09:48:33: WARNING] Safety check found problems with Python 3.9.17 [2023-06-23 09:48:33: WARNING] ... transformers [4.19.2] is affected by issue 51450 (for versions ['<4.23.0']) ```Failed smoke tests on SRE
```none bats run_all_tests.bats run_all_tests.bats ✓ Julia packages ✓ Julia functionality ✗ Python packages (3.9) (from function `setup_python' in file run_all_tests.bats, line 15, from function `test_python_packages' in file run_all_tests.bats, line 28, in test file run_all_tests.bats, line 67) `test_python_packages '3.9'' failed with status 127 /opt/tests/run_all_tests.bats: line 14: pyenv: command not found /opt/tests/run_all_tests.bats: line 10: pyenv: command not found /opt/tests/run_all_tests.bats: line 15: pyenv: command not found ✗ Python functionality (3.9) (from function `setup_python' in file run_all_tests.bats, line 15, from function `test_python_functionality' in file run_all_tests.bats, line 21, in test file run_all_tests.bats, line 70) `test_python_functionality '3.9'' failed with status 127 /opt/tests/run_all_tests.bats: line 14: pyenv: command not found /opt/tests/run_all_tests.bats: line 10: pyenv: command not found /opt/tests/run_all_tests.bats: line 15: pyenv: command not found ✓ Python virtual environments (3.9) ✗ Python package repository (3.9) (from function `setup_python' in file run_all_tests.bats, line 15, from function `test_python_repository' in file run_all_tests.bats, line 35, in test file run_all_tests.bats, line 76) `test_python_repository '3.9'' failed with status 127 /opt/tests/run_all_tests.bats: line 14: pyenv: command not found /opt/tests/run_all_tests.bats: line 10: pyenv: command not found /opt/tests/run_all_tests.bats: line 15: pyenv: command not found ✗ Python packages (3.10) (from function `setup_python' in file run_all_tests.bats, line 15, from function `test_python_packages' in file run_all_tests.bats, line 28, in test file run_all_tests.bats, line 81) `test_python_packages '3.10'' failed with status 127 /opt/tests/run_all_tests.bats: line 14: pyenv: command not found /opt/tests/run_all_tests.bats: line 10: pyenv: command not found /opt/tests/run_all_tests.bats: line 15: pyenv: command not found ✗ Python functionality (3.10) (from function `setup_python' in file run_all_tests.bats, line 15, from function `test_python_functionality' in file run_all_tests.bats, line 21, in test file run_all_tests.bats, line 85) `test_python_functionality '3.10'' failed with status 127 /opt/tests/run_all_tests.bats: line 14: pyenv: command not found /opt/tests/run_all_tests.bats: line 10: pyenv: command not found /opt/tests/run_all_tests.bats: line 15: pyenv: command not found ✓ Python virtual environments (3.10) ✗ Python package repository (3.10) (from function `setup_python' in file run_all_tests.bats, line 15, from function `test_python_repository' in file run_all_tests.bats, line 35, in test file run_all_tests.bats, line 91) `test_python_repository '3.10'' failed with status 127 /opt/tests/run_all_tests.bats: line 14: pyenv: command not found /opt/tests/run_all_tests.bats: line 10: pyenv: command not found /opt/tests/run_all_tests.bats: line 15: pyenv: command not found ✗ Python packages (3.11) (from function `setup_python' in file run_all_tests.bats, line 15, from function `test_python_packages' in file run_all_tests.bats, line 28, in test file run_all_tests.bats, line 96) `test_python_packages '3.11'' failed with status 127 /opt/tests/run_all_tests.bats: line 14: pyenv: command not found /opt/tests/run_all_tests.bats: line 10: pyenv: command not found /opt/tests/run_all_tests.bats: line 15: pyenv: command not found ✗ Python functionality (3.11) (from function `setup_python' in file run_all_tests.bats, line 15, from function `test_python_functionality' in file run_all_tests.bats, line 21, in test file run_all_tests.bats, line 99) `test_python_functionality '3.11'' failed with status 127 /opt/tests/run_all_tests.bats: line 14: pyenv: command not found /opt/tests/run_all_tests.bats: line 10: pyenv: command not found /opt/tests/run_all_tests.bats: line 15: pyenv: command not found ✓ Python virtual environments (3.11) ✗ Python package repository (3.11) (from function `setup_python' in file run_all_tests.bats, line 15, from function `test_python_repository' in file run_all_tests.bats, line 35, in test file run_all_tests.bats, line 105) `test_python_repository '3.11'' failed with status 127 /opt/tests/run_all_tests.bats: line 14: pyenv: command not found /opt/tests/run_all_tests.bats: line 10: pyenv: command not found /opt/tests/run_all_tests.bats: line 15: pyenv: command not found ✗ R packages (from function `assert_output' in file ../bats/bats-assert/src/assert_output.bash, line 178, in test file run_all_tests.bats, line 114) `assert_output --regexp 'All [0-9]+ packages are installed'' failed -- regular expression does not match output -- regexp (1 lines): All [0-9]+ packages are installed output (2 lines): Error in library(stringr) : there is no package called ‘stringr’ Execution halted -- ✓ R functionality ✓ R package repository ✓ Mounted drives (/data) ✓ Mounted drives (/home) ✓ Mounted drives (/output) ✓ Mounted drives (/shared) ✓ Mounted drives (/scratch) 22 tests, 10 failures ```:recycle: To reproduce