Open krassowski opened 5 months ago
Even after defining the test user and passing KEYCLOAK_USERNAME
/KEYCLOAK_PASSWORD
I get errors because some tests are using hard-coded github-actions.nebari.dev
namespace rather than my local deployment:
FAILED tests/tests_deployment/test_grafana_api.py::test_grafana_api_not_accessible_with_default_credentials - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /monitoring/api/datasources (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd1f90>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_dask_gateway.py::test_dask_gateway - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd2140>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_dask_gateway.py::test_dask_gateway_cluster_options - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd3ac0>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_jupyterhub_ssh.py::test_simple_jupyterhub_ssh - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd21a0>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_jupyterhub_ssh.py::test_print_jupyterhub_ssh - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd21a0>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_jupyterhub_ssh.py::test_exact_jupyterhub_ssh - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd21a0>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_jupyterhub_ssh.py::test_contains_jupyterhub_ssh - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd21a0>: Failed to establish a new connection: [Errno 113] No route to host'))
It looks like that can be changed by undocumented NEBARI_HOSTNAME
variable
Still the tests are failing for me on setup when getting the JupyterHub token (ERROR at setup of test_contains_jupyterhub_ssh
):
tests/tests_deployment/test_dask_gateway.py::test_dask_gateway ERROR
tests/tests_deployment/test_dask_gateway.py::test_dask_gateway_cluster_options ERROR
tests/tests_deployment/test_jupyterhub_ssh.py::test_simple_jupyterhub_ssh ERROR
tests/tests_deployment/test_jupyterhub_ssh.py::test_print_jupyterhub_ssh ERROR
tests/tests_deployment/test_jupyterhub_ssh.py::test_exact_jupyterhub_ssh ERROR
tests/tests_deployment/test_jupyterhub_ssh.py::test_contains_jupyterhub_ssh ERROR
File "/home/krassowski/nebari/tests/tests_deployment/test_jupyterhub_ssh.py", line 19, in api_token
return get_jupyterhub_token("jupyterhub-ssh")
File "/home/krassowski/nebari/tests/tests_deployment/utils.py", line 44, in get_jupyterhub_token
return r.json()["token"]
KeyError: 'token'
The response itself is:
{'status': 403, 'message': 'User test-user not found or no permissions to generate tokens'}
Ok, the token issue was a genuine problem on my branch. But the, running the ssh tests results in a prompt to unlock ssh agent, which I think should not be the case:
Declining leads to:
raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
--------------- Captured log setup ------------------
2024-05-06 11:56:00 INFO 1909 transport: Connected (version 2.0, client AsyncSSH_2.13.2)
2024-05-06 11:56:32 ERROR 1909 transport: Exception (client): key cannot be used for signing
2024-05-06 11:56:32 ERROR 1907 transport: Traceback (most recent call last):
2024-05-06 11:56:32 ERROR 1907 transport: File "envs/nebari/lib/python3.10/site-packages/paramiko/transport.py", line 2220, in run
2024-05-06 11:56:32 ERROR 1907 transport: handler(m)
2024-05-06 11:56:32 ERROR 1907 transport: File "site-packages/paramiko/auth_handler.py", line 404, in _parse_service_accept
2024-05-06 11:56:32 ERROR 1907 transport: sig = self.private_key.sign_ssh_data(blob, algorithm)
2024-05-06 11:56:32 ERROR 1907 transport: File "envs/nebari/lib/python3.10/site-packages/paramiko/agent.py", line 496, in sign_ssh_data
2024-05-06 11:56:32 ERROR 1907 transport: raise SSHException("key cannot be used for signing")
2024-05-06 11:56:32 ERROR 1907 transport: paramiko.ssh_exception.SSHException: key cannot be used for signing
2024-05-06 11:56:32 ERROR 1907 transport:
Just for visibility, I opened https://github.com/nebari-dev/nebari-docs/pull/449 to update documentation on running tests which solves some of the issues mentioned above, but many remain.
Based on what was discussed last week, this is not expected behavior, and as you just suggested, the docs for those tests env variables are lacking, I would recommend to not only addressing those in our docs, but also having an extra check for the current available credentials to at least give the user more visibility on whats happening
Describe the bug
The developer experience for adding tests is not great.
KEYCLOAK_USERNAME
is undefined. It is not mentioned in the documentationand one for deployment tests
Both are incorrect. The first command will invoke not only CLI tests but also the deployment tests because these are in
tests/tests_deployment
. The CLI tests are in fact intests/tests_unit
. The second command will just error out withERROR: file or directory not found: tests_deployment/
Expected behavior
Tests work not only on main branch but also on forks and can be run locally.
OS and architecture in which you are running Nebari
Yes
How to Reproduce the problem?
As per https://www.nebari.dev/docs/community/nebari-tests/#test-cli-with-pytest run:
See
Command output
No response
Versions and dependencies used.
No response
Compute environment
None
Integrations
No response
Anything else?
No response