search

ibm-granite-community / granite-snack-cookbook

Granite Snack Cookbook -- easily consumable recipes (python notebooks) that showcase the capabilities of the Granite models
Creative Commons Attribution 4.0 International
72 stars 17 forks source link

automate testing of PDL recipe #37

Closed adampingel closed 1 month ago

adampingel commented 1 month ago

The github action should execute this notebook:

https://github.com/ibm-granite-community/granite-snack-cookbook/blob/main/recipes/PDL/Prompt_Declaration_Language_python.ipynb

Please update the two files in .github

deanwampler commented 1 month ago

See PR: #54

deanwampler commented 1 month ago

A few notes about how the test failures where debugged, for posterity...

Python 3.10 vs 3.11

At first, Prompt_Declaration_Language_python.ipynb was included in the .github/notebook_lists/vanilla_notebooks.txt and .github/workflows/vanilla_workflow.yaml, but the tests failed with a mystifying error that pdl couldn't be found while executing from pdl.pdl import exec_str. In fact, the pip install actually failed, but the CI log contained now information about this. The import then failed. The problem was Python 3.10, which is what the "vanilla" tests. Use. I created new .github/notebook_lists/pdl_notebooks.txt and .github/workflows/pdl_workflow.yaml for just this notebook, with Python 3.11.

Can't Expect User Input During CI Tests

The example of a chat would fail, because user input was expected. I wrapped the invocation of this "program" thusly:

if os.environ.get('GRANITE_TESTING') is None:
  print(exec_str(pdl))
else:
  print(\"Since this is the test environment, we skip running the chat example.\")

The user still sees the example and it runs normally in an interactive notebook, but not in the CI test.

deanwampler commented 1 month ago

Closing, since #54 is now ready to merge.