[WIP] Initial tests

[jcrist]

Theoretically this should be simple to test, but my initial attempt so far hasn't been great.

Here I mimic the tests used in dockerspawner to do an integrated test with a mocked jupyterhub. This unfortunately hangs somewhere, and its not in my code. I suspect I'm using the mocked hub incorrectly.

I can confirm that everything works fine in a production setting, the issue here is just getting a test environment running.

[jcrist]

Logs from the current run:

$ py.test yarnspawner --pdb -x -vv -s
============================================================================================================================= test session starts ==============================================================================================================================
platform linux -- Python 3.7.2, pytest-4.0.2, py-1.7.0, pluggy-0.8.0 -- /home/testuser/miniconda/bin/python
cachedir: .pytest_cache
rootdir: /home/testuser/yarnspawner, inifile:
plugins: asyncio-0.9.0
collected 1 item

yarnspawner/tests/test_spawner.py::test_basic 19/01/08 00:07:31 INFO client.RMProxy: Connecting to ResourceManager at master.example.com/172.27.0.4:8032
19/01/08 00:07:32 INFO skein.Driver: Driver started, listening on 41611
Collecting packages...
Packing environment at '/home/testuser/miniconda' to 'yarnspawner-test-env.tar.gz'
[########################################] | 100% Completed | 28.0s
[D 08:04.960 MockHub app:1630] Writing PID 4775 to /tmp/tmpdzbw09en
[I 08:04.961 MockHub app:1673] Using Authenticator: jupyterhub.tests.mocking.MockPAMAuthenticator-0.9.4
[I 08:04.962 MockHub app:1673] Using Spawner: jupyterhub.tests.mocking.MockSpawner-0.9.4
[D 08:04.972 MockHub app:1050] Generating new cookie_secret
[I 08:04.972 MockHub app:1055] Writing cookie_secret to /home/testuser/yarnspawner/jupyterhub_cookie_secret
[D 08:04.978 MockHub app:1071] Connecting to db: sqlite:////tmp/tmpj470f3w1
[D 08:04.992 MockHub orm:656] Stamping empty database with alembic revision 896818069c98
[I 08:04.998 alembic.runtime.migration migration:117] Context impl SQLiteImpl.
[I 08:04.998 alembic.runtime.migration migration:122] Will assume non-transactional DDL.
[I 08:05.060 alembic.runtime.migration migration:327] Running stamp_revision  -> 896818069c98
[D 08:05.061 alembic.runtime.migration migration:516] new branch insert 896818069c98
[I 08:05.500 MockHub proxy:431] Generating new CONFIGPROXY_AUTH_TOKEN
[I 08:05.523 MockHub app:1201] Not using whitelist. Any authenticated user will be allowed.
[D 08:05.551 MockHub app:1473] Loading state for admin from db
[D 08:05.554 MockHub app:1489] Loaded users:
       admin admin
[I 08:05.586 MockHub app:1855] Hub API listening on http://edge.example.com:8081/@/space%20word/hub/
[W 08:05.589 MockHub proxy:565] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 08:05.589 MockHub proxy:567] Starting proxy @ http://127.0.0.1:36839/@/space%20word/
[D 08:05.590 MockHub proxy:568] Proxy cmd: ['configurable-http-proxy', '--ip', '127.0.0.1', '--port', '36839', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://edge.example.com:8081/@/space%20word/hub/error']
[D 08:05.597 MockHub proxy:517] Writing proxy pid file: jupyterhub-proxy.pid
00:08:05.806 [ConfigProxy] info: Proxying http://127.0.0.1:36839 to (no default)
00:08:05.810 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 08:06.240 MockHub proxy:603] Proxy started and appears to be up
[D 08:06.241 MockHub proxy:296] Fetching routes to check
[D 08:06.242 MockHub proxy:686] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 08:06.258 MockHub proxy:301] Checking routes
[I 08:06.259 MockHub proxy:370] Adding default route for Hub: /@/space%20word/ => http://edge.example.com:8081
00:08:06.259 [ConfigProxy] info: 200 GET /api/routes
[D 08:06.260 MockHub proxy:686] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/@/space%20word
00:08:06.263 [ConfigProxy] info: Adding route /@/space word -> http://edge.example.com:8081
00:08:06.265 [ConfigProxy] info: 201 POST /api/routes/@/space%20word
[I 08:06.265 MockHub app:1912] JupyterHub is now running at http://127.0.0.1:36839/@/space%20word/
[D 08:06.576 MockHub user:190] Creating <class 'yarnspawner.spawner.YarnSpawner'> for alice:

[jcrist]

cc @yuvipanda, do you have any advice for testing a spawner implementation against jupyterhub?

[minrk]

Ah. Maybe it's the fact that you don't have pytest-tornado and the jupyterhub fixtures use pytest-tornado. jupyterhub master has switched to pytest-asyncio, but I'm not sure if you can use the app fixture with pytest-asyncio or not. Try installing pytest-tornado and marking the test with pytest.mark.gen_test instead to see if it works.

[minrk]

An alternative is to do what kubespawner does and instantiate your Spawner with the necessary attributes defined/mocked out. In this case, there is less code that you don't control relevant to your tests, so while it may be similarly brittle from one release to the next of JupyterHub, the brittleness should at least be focused on a smaller set of simple missing attributes, rather than the potentially complex test scaffolding in the app fixture.

[jcrist]

Try installing pytest-tornado and marking the test with pytest.mark.gen_test instead to see if it works.

That was it, thanks for the tip.

An alternative is to do what kubespawner does and instantiate your Spawner with the necessary attributes defined/mocked out.

I tried this originally, but ran into issues because I'd have to mock the api token, and the singleuser instance that's started expects to communicate with the hub on startup (yarnspawner also needs an api endpoint to report its address to). Setting up a full mock hub seemed more robust in comparison.

Publicly available mocking utilities (as described in https://github.com/jupyterhub/jupyterhub/issues/2376) would be much appreciated.