computationalmodelling / nbval

A py.test plugin to validate Jupyter notebooks
Other
438 stars 50 forks source link

`--sanitize-with` option seems to be behaving weirdly with "newly computed (test) output" #204

Open tlvu opened 6 months ago

tlvu commented 6 months ago

We have this failure below and we could not understand why our output-sanitize.cfg regex file is unable to cover the diff.

Why is the "newly computed (test) output" has the "Text(0.5, 91.20243008191655, 'Longitude')" part in it.

  _ pavics-sdi-fix_nbs_jupyter_alpha/docs/source/notebooks/regridding.ipynb::Cell 27 _
  Notebook cell execution failed
  Cell 27: Cell outputs differ

  Input:
  # Now we can plot easily the results as a choropleth map!
  ax = shapes_data.plot(
      "tasmin", legend=True, legend_kwds={"label": "Minimal temperature 1993-05-20 [K]"}
  )
  ax.set_ylabel("Latitude")
  ax.set_xlabel("Longitude");

  Traceback:
  dissimilar number of outputs for key "text/plain"<<<<<<<<<<<< Reference outputs from ipynb file:
  <Figure size LENGTHxWIDTH with N Axes>
  ============ disagrees with newly computed (test) output:
  Text(0.5, 91.20243008191655, 'Longitude')
  <Figure size LENGTHxWIDTH with N Axes>
  >>>>>>>>>>>>

output-sanitize.cfg:

[finch-figure-size]
regex: <Figure size \d+x\d+\swith\s\d\sAxes>
replace: <Figure size LENGTHxWIDTH with N Axes>

Full output-sanitize.cfg file: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/a4592eab55ad177b00cba77f126be56ff6566287/notebooks/output-sanitize.cfg#L92-L94

Notebook: https://github.com/Ouranosinc/pavics-sdi/blob/4ffec2df463b413c78991e9481fbe182537b3a65/docs/source/notebooks/regridding.ipynb

command:

py.test --nbval pavics-sdi-fix_nbs_jupyter_alpha_refresh_output/docs/source/notebooks/regridding.ipynb --sanitize-with notebooks/output-sanitize.cfg --dist=loadscope --numprocesses=0

============================= test session starts ============================== platform linux -- Python 3.10.13, pytest-8.1.1, pluggy-1.4.0 rootdir: /home/jenkins/agent/workspace/_workflow-tests_new-docker-build plugins: anyio-4.3.0, dash-2.16.1, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.5.0