Pinning R 4.0 with runtime.txt on mybinder.org not working as expected

fkohrt commented 3 years ago

Bug description

Pinning the R version to 4.0 using runtime.txt on mybinder.org does not work – I get R 4.1 instead.

Expected behaviour

I expect R version 4.0.5 ("Shake and Throw"):

> R.version.string
[1] "R version 4.0.5 (2021-03-31)"

Actual behaviour

I get R version 4.1.0 ("Camp Pontanezen"):

> R.version.string
[1] "R version 4.1.0 (2021-05-18)"

How to reproduce

Set up a repository with the following runtime.txt:

r-4.0-2021-05-19

Your personal set up

I am using the mybinder.org service.

Versions:

jupyterhub --version: 1.3.0
python --version: Python 3.7.10

Full environment

``` aiohttp==3.7.4.post0 alembic @ file:///home/conda/feedstock_root/build_artifacts/alembic_1613901514078/work anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1612745536224/work/dist argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1602546578258/work async-generator==1.10 async-timeout==3.0.1 attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1605083924122/work Babel @ file:///home/conda/feedstock_root/build_artifacts/babel_1605182336601/work backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work backports.functools-lru-cache==1.6.1 bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1612213472466/work blinker==1.4 brotlipy==0.7.0 certifi==2020.12.5 certipy==0.1.3 cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1613413867554/work chardet @ file:///home/conda/feedstock_root/build_artifacts/chardet_1610093487176/work cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1612993734132/work decorator==4.4.2 defusedxml==0.6.0 entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1602701733603/work/dist/entrypoints-0.3-py2.py3-none-any.whl idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1593328102638/work importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1610355166636/work ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1613852948592/work/dist/ipykernel-5.5.0-py3-none-any.whl ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1612487002309/work ipython-genutils==0.2.0 ipywidgets @ file:///home/conda/feedstock_root/build_artifacts/ipywidgets_1609995587151/work jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1610146791823/work Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1612119311452/work json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1600692310011/work jsonschema==3.2.0 jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1610375432619/work jupyter-core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1612125257337/work jupyter-offlinenotebook @ file:///home/conda/feedstock_root/build_artifacts/jupyter-offlinenotebook_1610833110941/work jupyter-resource-usage @ file:///home/conda/feedstock_root/build_artifacts/jupyter-resource-usage_1608550684149/work jupyter-rsession-proxy @ https://github.com/jupyterhub/jupyter-rsession-proxy/archive/d5efed5455870556fc414f30871d0feca675a4b4.zip jupyter-server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1613709403212/work jupyter-server-proxy==1.4.0 jupyter-shiny-proxy @ https://github.com/ryanlovett/jupyter-shiny-proxy/archive/47557dc47e2aeeab490eb5f3eeae414cdde4a6a9.zip jupyter-telemetry @ file:///home/conda/feedstock_root/build_artifacts/jupyter_telemetry_1605173804246/work jupyterhub @ file:///home/conda/feedstock_root/build_artifacts/jupyterhub-feedstock_1607688225833/work jupyterlab @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_1612296988614/work jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1601375948261/work jupyterlab-server @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_server_1613760084674/work jupyterlab-widgets @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_widgets_1609173350931/work Mako @ file:///home/conda/feedstock_root/build_artifacts/mako_1610659158978/work MarkupSafe @ file:///home/conda/feedstock_root/build_artifacts/markupsafe_1610127564475/work mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1610112870915/work multidict==5.1.0 nbclassic @ file:///home/conda/feedstock_root/build_artifacts/nbclassic_1610352513187/work nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1612903843315/work nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert_1605401831777/work nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1611005694671/work nbgitpuller==0.9.0 nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1605195931949/work notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1610575319689/work nteract-on-jupyter==2.1.3 oauthlib==3.0.1 packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1612459636436/work pamela==1.0.0 pandocfilters==1.4.2 parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1607618318316/work pexpect==4.8.0 pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602535628301/work prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1605543085815/work prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1613068548426/work psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1610127094245/work ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1593275161868/work pycurl==7.43.0.6 Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1613412098605/work PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1610910308735/work pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1608055815057/work pyparsing==2.4.7 pyrsistent @ file:///home/conda/feedstock_root/build_artifacts/pyrsistent_1610146801408/work PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1610291444829/work python-dateutil==2.8.1 python-editor==1.0.4 python-json-logger @ file:///home/conda/feedstock_root/build_artifacts/python-json-logger_1602545356084/work pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1612179539967/work pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1613478743724/work requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1608156231189/work ruamel.yaml @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml_1610291373612/work ruamel.yaml.clib @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml.clib_1610146839777/work Send2Trash==1.5.0 simpervisor==0.4 six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1610318316464/work SQLAlchemy @ file:///home/conda/feedstock_root/build_artifacts/sqlalchemy_1612225082967/work terminado @ file:///home/conda/feedstock_root/build_artifacts/terminado_1609794176029/work testpath==0.4.4 tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1610094704591/work traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1602771532708/work typing-extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1602702424206/work urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1611695416663/work wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work webencodings==0.5.1 widgetsnbextension @ file:///home/conda/feedstock_root/build_artifacts/widgetsnbextension_1603129364725/work yarl==1.6.3 zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1603668650351/work ```

Related information

Explicitly demanding R 4.1 using the following runtime.txt…

r-4.1-2021-05-20

…does not work (which makes sense):

Error during build: Version '4.1' of R is not supported.

Try here:

welcome[bot] commented 3 years ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

betatim commented 3 years ago

Hi,

unfortunately using runtime.txt doesn't work very well :-/ I think right now using the approach from https://github.com/binder-examples/r-conda (if possible) is the easier way to get a specific version of R installed.

Overall our support for R is a bit basic. This has a few reasons, but the most important one is that we lack people who really use and know R. It would be great to find someone who has more experience in that area and wants to help out.

If I recall correctly the runtime.txt approach uses APT packages to install things. And as we learnt over the last few years there is no good way to pin those to specific versions. We use a separate PPA that provides different versions but it isn't designed to allow you to install arbitrary version, instead they target the latest only (I think). Which is what has lead us to recommending conda as the package manager to install R.

sje30 commented 3 years ago

I get the same problem as reported above.

Another issue with the R 4.0.2 support (or 4.1.0) is that Rstudio is too old (from 2019) compared to R 4.1.0 and so plots are not visible.

For an example, see: https://github.com/sje30/rescience-hor and hit the binder button.

(it takes a few mins to spin up)

after running system("make") in the R session, you will get a warning:

Warning message:
R graphics engine version 14 is not supported by this version of RStudio. The Plots tab will be disabled until a newer version of RStudio is installed.

CC: @nuest

fkohrt commented 3 years ago

Another issue with the R 4.0.2 support (or 4.1.0) is that Rstudio is too old (from 2019) compared to R 4.1.0 and so plots are not visible.

@sje30: This is tracked as jupyterhub/repo2docker#1041

aplamada commented 3 years ago

@betatim Hi, Using conda the specified versions in the environment.yml are not really guarantied. I tried to ask for r-base>=4.1 but in the end I got R version 4.0.5 (2021-03-31) - see the forked binder-examples/r-conda.

My guess is that the step RUN mamba install -p ${NB_PYTHON_PREFIX} r-base r-irkernel=1.1 r-devtools -y (done after RUN TIMEFORMAT='time: %3R' bash -c 'time /tmp/install-miniforge.bash' && rm -rf /tmp/install-miniforge.bash /tmp/env) is causing the issue...

manics commented 3 years ago

I've transferred this issue to the repo2docker repo which is the tool that handles the builds behind BinderHub.

This is an excerpt of the internal generated Dockerfile:

...
COPY --chown=1000:1000 src/environment.yml ${REPO_DIR}/environment.yml
USER ${NB_USER}
RUN TIMEFORMAT='time: %3R' \
bash -c 'time mamba env update -p ${NB_PYTHON_PREFIX} -f "environment.yml" && \
time mamba clean --all -f -y && \
mamba list -p ${NB_PYTHON_PREFIX} \
'

RUN mamba install -p ${NB_PYTHON_PREFIX} r-base r-irkernel=1.1 r-devtools -y && \
mamba clean --all -f -y && \
mamba list -p ${NB_PYTHON_PREFIX}
...

r-irkernel=1.1 is installed after environment.yml, but that brings in r-base-4.0.5 (you can check with conda create -ntmp r-irkernel=1.1) https://github.com/jupyterhub/repo2docker/blob/6e2a6af959a366ea9cd5e268450122d0f7064afd/repo2docker/buildpacks/_r_base.py#L19-L20 https://github.com/jupyterhub/repo2docker/blob/6e2a6af959a366ea9cd5e268450122d0f7064afd/repo2docker/buildpacks/conda/__init__.py#L347-L353

aplamada commented 3 years ago

A first step can be to improve the tests: to check also the minor version and add more tests.

To check the minor version, e.g. 4.0 we can change version$major != "4" into

! startsWith(paste(version$major,version$minor,sep="."),"4.0")

Is this something that would add value and that I can suggest in a pull request?

manics commented 3 years ago

@aplamada More tests are always welcome!

I think a long term fix for this bug may require one of:

removing the pinned version in https://github.com/jupyterhub/repo2docker/blob/6e2a6af959a366ea9cd5e268450122d0f7064afd/repo2docker/buildpacks/_r_base.py#L19-L20
making the version conditional on something else
changing the order so environment.yml is installed after r so that it can override it.

I'm not sure what's best though.

aplamada commented 3 years ago

@manics Thanks for your feedback! I will think about the options that you mentioned. I have also the gut feeling that R with conda might follow closer the Python with Conda approach, that is the irkernel version to be locked for each r version.

Also it is not very clear to me why R with install.R is not following Python with requirements.txt . So either install.R can happen always in the conda env of R, or Python can be installed with apt via ppa:deadsnakes/ppa .

meeseeksmachine commented 3 years ago

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/psa-latest-rstudio-will-not-work-with-jupyterhub-or-binder-use-r-version-4-0/10957/1

yuvipanda commented 2 years ago

Using R from runtime.txt gets it from apt, and https://github.com/jupyterhub/repo2docker/pull/1104 (particularly https://github.com/jupyterhub/repo2docker/pull/1104/commits/5246a0e7c242d98bac44fa5620c42f6477d185bc) fixes this bug.

aplamada commented 2 years ago

@yuvipanda This is great news! I will ask for your help regarding #1086.

fkohrt commented 2 years ago

Unfortunately, the repository does not build at all anymore:

Build logs

``` Waiting for build to start... ---> b74115f6dbbb Step 26/67 : COPY --chown=1000:1000 build_script_files/-2fusr-2flib-2fpython3-2e8-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-391af5 /etc/profile.d/activate-conda.sh ---> bf3034943bab Step 27/67 : COPY --chown=1000:1000 build_script_files/-2fusr-2flib-2fpython3-2e8-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2fenvironment-2epy-2d3-2e7-2elock-4f1154 /tmp/env/environment.lock ---> bc4596abf8c6 Step 28/67 : COPY --chown=1000:1000 build_script_files/-2fusr-2flib-2fpython3-2e8-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2finstall-2dminiforge-2ebash-514214 /tmp/install-miniforge.bash ---> 00e4175b61aa Step 29/67 : RUN TIMEFORMAT='time: %3R' bash -c 'time /tmp/install-miniforge.bash' && rm -rf /tmp/install-miniforge.bash /tmp/env ---> [Warning] Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. ---> Running in d37b939f410e [91m++ dirname /tmp/install-miniforge.bash [0m[91m+ cd /tmp [0m[91m+ MINIFORGE_VERSION=4.9.2-2 + MAMBA_VERSION=0.7.4 + SHA256SUM=7a7bfaff87680298304a97ba69bcf92f66c810995a7155a2918b99fafb8ca1dc + URL=https://github.com/conda-forge/miniforge/releases/download/4.9.2-2/Mambaforge-4.9.2-2-Linux-x86_64.sh + INSTALLER_PATH=/tmp/miniforge-installer.sh [0m[91m+ unset HOME + wget --quiet https://github.com/conda-forge/miniforge/releases/download/4.9.2-2/Mambaforge-4.9.2-2-Linux-x86_64.sh -O /tmp/miniforge-installer.sh [0m[91mtime: 3.393 + chmod +x /tmp/miniforge-installer.sh [0m[91m+ echo '7a7bfaff87680298304a97ba69bcf92f66c810995a7155a2918b99fafb8ca1dc /tmp/miniforge-installer.sh' [0m[91m+ sha256sum --quiet -c - [0m[91m+ bash /tmp/miniforge-installer.sh -b -p /srv/conda [0mPREFIX=/srv/conda Unpacking payload ... [91m 0%| | 0/57 [00:00 13235cb3bb56 Step 30/67 : RUN mkdir -p ${NPM_DIR} && chown -R ${NB_USER}:${NB_USER} ${NPM_DIR} ---> [Warning] Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. ---> Running in b2354f64d1e8 Removing intermediate container b2354f64d1e8 ---> 451d1704e07e Step 31/67 : RUN echo "deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/" > /etc/apt/sources.list.d/r-ubuntu.list ---> [Warning] Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. ---> Running in 1732ade2282a Removing intermediate container 1732ade2282a ---> 77554c45e7d5 Step 32/67 : RUN wget --quiet -O - 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xe298a3a825c0d65dfd57cbb651716619e084dab9' | apt-key add - ---> [Warning] Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. ---> Running in 66165d89813f [91mWarning: apt-key output should not be parsed (stdout is not a terminal) [0mOK Removing intermediate container 66165d89813f ---> b6296b37fe86 Step 33/67 : RUN apt-get update > /dev/null && apt-get install --yes r-base=4.0.5-1.1804.0 r-base-core=4.0.5-1.1804.0 r-base-dev=4.0.5-1.1804.0 r-recommended=4.0.5-1.1804.0 libclang-dev libzmq3-dev > /dev/null && apt-get -qq purge && apt-get -qq clean && rm -rf /var/lib/apt/lists/* ---> [Warning] Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. ---> Running in 9ede65c4848d [91mE: Unable to correct problems, you have held broken packages. [0mRemoving intermediate container 9ede65c4848d The command '/bin/sh -c apt-get update > /dev/null && apt-get install --yes r-base=4.0.5-1.1804.0 r-base-core=4.0.5-1.1804.0 r-base-dev=4.0.5-1.1804.0 r-recommended=4.0.5-1.1804.0 libclang-dev libzmq3-dev > /dev/null && apt-get -qq purge && apt-get -qq clean && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100Built image, launching... Failed to connect to event stream ```

Maybe related: #1116

fkohrt commented 2 years ago

It works now, maybe because of #1114. Not sure though.

jupyterhub / repo2docker