Fix CI run on `ubuntu-latest-py3.12`

glatterf42 commented 1 month ago

At the moment, this PR mainly runs the CI checks to debug the mysterious error message observed even after multiple re-runs and cache deletions on ubuntu-latest-py3.12:

 Error in IRkernel::installspec() : 
  jupyter-client has to be installed but “jupyter kernelspec --version” exited with code 1.

This error is not flaky and does not seem to originate from outdated cached tool versions (at least after removing the ubuntu-latest-R-4.4.1 cache, the error persisted). It might be related to ubuntu-latest slowly changing to 24.04 over the course of this month, but the ...-py3.11 test uses the exact same runner image and workflow file without problems.

How to review

Required: describe specific things that reviewer(s) must do, in order to ensure that the PR achieves its goal. If no review is required, write “No review: …” and describe why.

PR checklist

[ ] Continuous integration checks all ✅
[ ] Add or expand tests; coverage checks both ✅
[ ] Add, expand, or update documentation.
[ ] Update release notes.

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 98.8%. Comparing base (c49a115) to head (8bebe09).

:exclamation: Current head 8bebe09 differs from pull request most recent head d156586

Please upload reports for the commit d156586 to get more accurate results.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #547 +/- ## ======================================= - Coverage 98.9% 98.8% -0.2% ======================================= Files 44 44 Lines 4819 4819 ======================================= - Hits 4770 4765 -5 - Misses 49 54 +5 ``` [see 2 files with indirect coverage changes](https://app.codecov.io/gh/iiasa/ixmp/pull/547/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iiasa)

khaeru commented 1 month ago

I am not sure why this should be different suddenly for one particular job, but one thing to consider is that our pip install … step happens within the GitHub Actions runner work directory:

https://github.com/iiasa/ixmp/blob/c49a1152305b4aedd349048d2c6d4f1dcbb4e82b/.github/workflows/pytest.yaml#L104-L107

In the subsequent step: https://github.com/iiasa/ixmp/blob/c49a1152305b4aedd349048d2c6d4f1dcbb4e82b/.github/workflows/pytest.yaml#L119-L133

The GHA runner invokes RScript.
Rscript runs that script content, including IRkernel::installspec() (or your added debug calls).
Those invoke certain Python executables.
These will need to determine a Python /site-packages/ from which to perform imports.

It's possible that the overall PYTHONPATH is not making it through this call stack, and thus while jupyter-kernelspec runs, it doesn't know where to get its imports.

A second thing to look into is that the install of reticulate appears to be failing. I don't believe this should cause the other error, but it's a possibility:

gcc -I"/opt/R/4.4.1/lib/R/include" -DNDEBUG   -I/usr/local/include    `libpng-config --cflags` -fpic  -g -O2  -c dummy.c -o dummy.o
/bin/bash: line 1: libpng-config: command not found
gcc -I"/opt/R/4.4.1/lib/R/include" -DNDEBUG   -I/usr/local/include    `libpng-config --cflags` -fpic  -g -O2  -c read.c -o read.o
/bin/bash: line 1: libpng-config: command not found
read.c:3:10: fatal error: png.h: No such file or directory
    3 | #include <png.h>
      |          ^~~~~~~
compilation terminated.
make: *** [/opt/R/4.4.1/lib/R/etc/Makeconf:195: read.o] Error 1
ERROR: compilation failed for package ‘png’
* removing ‘/home/runner/work/_temp/Library/png’
ERROR: dependency ‘png’ is not available for package ‘reticulate’
* removing ‘/home/runner/work/_temp/Library/reticulate’

The downloaded source packages are in
    ‘/tmp/Rtmpl4Ujah/downloaded_packages’
Warning messages:
1: In i.p(...) : installation of package ‘png’ had non-zero exit status
2: In i.p(...) :
  installation of package ‘reticulate’ had non-zero exit status

glatterf42 commented 1 month ago

It's not about reticulate, because that fails even on the successful ubuntu-latest-py3.11 runs. That also seems odd, I agree, especially because I thought that was needed to run the R tutorials or so, but the test suite at large doesn't seem to mind.

glatterf42 commented 1 month ago

It's not specifically about PYTHONPATH, either. That variable is an empty string both within Rscript as well as outside, both on Python 3.11 and 3.12. However, the new debug output shows that within Rscript, even calling pip list doesn't work because no module named pip can be found.

khaeru commented 1 month ago

It may help to try something like python -c "import sys; print(sys.executable, sys.path)". PYTHONPATH is only used to initialize sys.path, which also gets (or may fail to get) entries from other locations.

glatterf42 commented 1 month ago

Indeed: '/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages' is present (printed from your suggested query) outside of Rscript, but missing inside of it.

glatterf42 commented 1 month ago

GitHub rolled back their shift from Ubuntu 22.04 to 24.04 for ubuntu-latest, so this PR is no longer needed -- for now. However, since I don't trust the eventual migration will go more smoothly, I'm going to keep the branch around.

iiasa / ixmp