Running tests in CI gets "stuck"

[batpad]

Everything seems to work fine running locally, but the tests in CI seem to get stuck. This time I waited 45+ minutes and it was stuck on:

 Found notebooks, using pytest-notebook to run them...
  ============================= test session starts ==============================
  platform linux -- Python 3.11.8, pytest-8.1.1, pluggy-1.4.0
  rootdir: /srv/repo
  plugins: anyio-3.7.1, pytest_notebook-0.10.0
  collected 1 item

  image-tests/test-notebook.ipynb 

~Locally, the tests seem to run really quickly and pass, so not sure what's going on here.~

Let's discuss when we chat @sunu

[batpad]

Sorry, I mis-spoke here - the test does not, in fact, work locally for me - I was testing from the wrong branch 🤦 .

It does seem like the visualization tests might be tripping up pytest-notebook that we are using to run the tests. Thanks @sunu for digging into this.

We'll discuss a bit and come up with options. @jsignell unfortunately, for now, we may want to stick to String outputs in the tests, but let me and @sunu dig a bit and discuss and be able to lay out some options for the shorter and longer term.

I think it's nice to get a clear picture of what the limits of the current pytest-notebook might be, so I think it was good to include more complex tests to try, and I think we can definitely add the ability to test more complex viz to our backlog and figure out whether this is something we can fix upstream or use a slightly different tool for.

Again, sorry about lying about it working locally ✌️ - this all seems to make sense.

[jsignell]

Yeah I thought it was working locally as well, but in reality I probably forgot to rebuild the image 🤦🏻

[batpad]

This was resolved by removing the more complex viz tests and sticking to string comparisons to test outputs.

Thanks much @jsignell !