Smoke tests failing on Tier 3 equivalent SRE

craddm commented 1 month ago

:white_check_mark: Checklist

[x] I have searched open and closed issues for duplicates.
[x] This is a problem observed when deploying a Data Safe Haven.
[x] I can reproduce this with the latest version.
[x] I have read through the documentation.
[x] This isn't an open-ended question (open a discussion if it is).

:computer: System information

Operating System: macOs/Linux
Data Safe Haven version: release-v5.0.0rc3

:package: Packages

List of packages

```none Paste list of packages here ```

:no_entry_sign: Describe the problem

Smoke tests fail on Tier 3 SREs because necessary packages are not on the current allowlists. The allowlists need to be brought up-to-date.

:deciduous_tree: Log messages

Relevant log messages

```none root@shm-turtles-sre-leonardo-vm-workspace-01:/usr/local/smoke_tests# bats run_all_tests.bats ✓ Mounted drives (/data) ✓ Mounted drives (/home) ✓ Mounted drives (/output) ✓ Mounted drives (/shared) ✗ Python package repository (from function `initialise_python_environment' in file run_all_tests.bats, line 11, in test file run_all_tests.bats, line 66) `initialise_python_environment' failed ERROR: HTTP error 403 while getting http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 (from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/simple/pip/) (requires-python:>=3.8) ERROR: Could not install requirement pip from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 because of HTTP error 403 Client Error: Forbidden for url: http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl for URL http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 (from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/simple/pip/) (requires-python:>=3.8) ✗ R package repository (in test file run_all_tests.bats, line 73) `[ "$status" -eq 0 ]' failed with status 2 ✗ Python functionality (from function `initialise_python_environment' in file run_all_tests.bats, line 11, in test file run_all_tests.bats, line 80) `initialise_python_environment' failed ERROR: HTTP error 403 while getting http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 (from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/simple/pip/) (requires-python:>=3.8) ERROR: Could not install requirement pip from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 because of HTTP error 403 Client Error: Forbidden for url: http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl for URL http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 (from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/simple/pip/) (requires-python:>=3.8) ✗ R functionality (from function `install_r_package_version' in file run_all_tests.bats, line 31, in test file run_all_tests.bats, line 87) `install_r_package_version "MASS" "7.3-52"' failed trying URL 'http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/cran-proxy/src/contrib/remotes_2.5.0.tar.gz' Error in download.file(url, destfile, method, mode = "wb", ...) : cannot open URL 'http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/cran-proxy/src/contrib/remotes_2.5.0.tar.gz' In addition: Warning message: In download.file(url, destfile, method, mode = "wb", ...) : cannot open URL 'http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/cran-proxy/src/contrib/remotes_2.5.0.tar.gz': HTTP status was '403 Forbidden' Warning in download.packages(pkgs, destdir = tmpd, available = available, : download of package ‘remotes’ failed Error in library("remotes", lib = "/root/.local/bats-r-environment") : there is no package called ‘remotes’ Execution halted ✗ MS SQL database (Python) (from function `initialise_python_environment' in file run_all_tests.bats, line 11, in test file run_all_tests.bats, line 98) `initialise_python_environment' failed ERROR: HTTP error 403 while getting http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 (from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/simple/pip/) (requires-python:>=3.8) ERROR: Could not install requirement pip from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 because of HTTP error 403 Client Error: Forbidden for url: http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl for URL http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 (from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/simple/pip/) (requires-python:>=3.8) ✓ MS SQL database (R) ✗ Postgres database (Python) (from function `initialise_python_environment' in file run_all_tests.bats, line 11, in test file run_all_tests.bats, line 115) `initialise_python_environment' failed ERROR: HTTP error 403 while getting http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 (from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/simple/pip/) (requires-python:>=3.8) ERROR: Could not install requirement pip from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 because of HTTP error 403 Client Error: Forbidden for url: http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl for URL http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/packages/pip/24.2/pip-24.2-py3-none-any.whl#sha256=2cd581cf58ab7fcfca4ce8efa6dcacd0de5bf8d0a3eb9ec927e07405f4d9e2a2 (from http://nexus.leonardo.blue.develop.turingsafehaven.ac.uk/repository/pypi-proxy/simple/pip/) (requires-python:>=3.8) ✓ Postgres database (R) 12 tests, 6 failures ```

:recycle: To reproduce

Run smoke tests on a Tier 3 SRE

JimMadge commented 1 month ago

@craddm How sure are you it is a problem with the lists themselves?

It looks like pip fails to install. I suppose that isn't on the list though :eyes:

craddm commented 1 month ago

Exactly! pip isn't on the list so it gets denied. To the extent that things are on the list, things install (at least for R)

craddm commented 1 month ago

So this partially due to things missing from the existing allowlists. Adding additional packages to the allowlists is allowing some things to progress.

The situation for R is a little more complicated than for Python. I think Nexus is only keeping the most recent version of specific packages, and also isn't keeping version info. the existing tests try to install a specific version of the MASS package, which doesn't work

Error in download_version_url(package, version, repos, type) : 
  couldn't find package 'MASS'

I'm not sure if that's the actual problem with MASS, as the error is different for, say, rlang:

> remotes::install_version("rlang", version = "1.1.3")
Error in download_version_url(package, version, repos, type) : 
  version '1.1.3' is invalid for package 'rlang'

I'd also note that the version of R installed is quite old - 4.1.2 as opposed to the most recent 4.4.0, so some packages won't be installable if we can only get at the latest CRAN versions.

JimMadge commented 1 month ago

@craddm Anything you need to work on this one?

craddm commented 1 month ago

It looks like Nexus doesn't cache packages downloaded using the remotes::install_version function at all. At Tier 2, when all packages are allowed, you can install old versions using that function, but they still don't get cached, even though the request is still going through the proxy.

This was wrong, they're there, it's just the url is structured slightly differently so they're under a subfolder

craddm commented 1 month ago

@craddm Anything you need to work on this one?

An up-to-date allowlist! Do we want to replicate the lists from the powershell release, or start afresh (i.e. a minimal allowlist that we'll add to over time on request)?

Feels like it might be a good chance to prune the list and start again.

jemrobinson commented 1 month ago

I think the v5 lists here are/were @JimMadge's attempt to minimise. Do you have any thoughts on obviously-missing R packages @craddm ?

craddm commented 1 month ago

I think the v5 lists here are/were @JimMadge's attempt to minimise. Do you have any thoughts on obviously-missing R packages @craddm ?

It's currently very, very limited, missing things like ggplot2 and other tidyverse packages, stats things like lme4. First off, need to figure out why Nexus can't handle remotes::install_version at Tier 3, and how to install an up-to-date R instead of the one from the apt repositories

jemrobinson commented 1 month ago

Assume for the moment that we can't install a more up-to-date R version. The smoke tests are supposed to check the following things:

our version of R/Python/Julia/whatever works as expected (ideally just using things from the standard library)
we can install packages from the allowed list
we can't install packages that are not on the allowed list

If the current smoke tests aren't doing this well, let's just update the tests.

craddm commented 1 month ago

Assume for the moment that we can't install a more up-to-date R version. The smoke tests are supposed to check the following things:

our version of R/Python/Julia/whatever works as expected (ideally just using things from the standard library)

we can install packages from the allowed list

we can't install packages that are not on the allowed list

If the current smoke tests aren't doing this well, let's just update the tests.

The problem with that is, you are using an older version of R and Nexus is only allowing you to update the latest version of packages, some of which don't work for the old version of R (including MASS). (and the smoke tests were trying to use the remotes package, which wasn't on the allowlist)

So we can find a way to make the smoke tests pass, but that won't mean it's really functioning

jemrobinson commented 1 month ago

OK, so would adding the remotes package to the allowlist make this possible? If so, let's do that. If not, what other alternatives to updating R are there?

craddm commented 1 month ago

OK, so would adding the remotes package to the allowlist make this possible? If so, let's do that. If not, what other alternatives to updating R are there?

Yes, adding remotes allows you to try installing older versions, but then you hit the problem that Nexus won't allow it to install older versions.

I'm trying to figure out how to get Nexus to allow remotes to install older versions.

JimMadge commented 1 month ago

I think the v5 lists here are/were @JimMadge's attempt to minimise. Do you have any thoughts on obviously-missing R packages @craddm ?

I honestly can't remember.

I would like to remove the burden of maintaining a complete list from the dev team. At least, I don't think we should maintain that in this repo (https://github.com/uk-tre/packages could be a route).

I think we do need some kind of minimal list that works with our tests though. To ensure the system works.

JimMadge commented 1 month ago

OK, so would adding the remotes package to the allowlist make this possible? If so, let's do that. If not, what other alternatives to updating R are there?

Yes, adding remotes allows you to try installing older versions, but then you hit the problem that Nexus won't allow it to install older versions.

I'm trying to figure out how to get Nexus to allow remotes to install older versions.

@craddm You might need to add an extra content selector(s) here

alan-turing-institute / data-safe-haven