Repository structure overhaul

[eitsupi]

I will be working over the next few weeks on the repository overhaul that was discussed in several issues.

The current repository structure was originally designed to be updated manually, and I forced it to be changed to be updated automatically several years ago (#181), so it is not optimized for automatic updates. (Files that should be updated manually and files that are automatically generated are not completely separated.) And, because Dockerfile is generated from json and R scripts, it is not possible to generate complex Dockerfiles that incorporate multi-stage builds.

Therefore, rewrite the structure assuming automation as follows:

• Dockerfile and bake files are generated by pouring variables into manually written templates with a template engine (The glue R pakcage).
• Variables are stored in TSV and JSON files and are automatically generated daily by CI.
• Completely rewrite the Dockerfiles to share layers instead of hierarchy. Now:

  FROM rocker/r-ver:4.3.3
  RUN ...

  Future:

  FROM ubuntu:jammy
  COPY rocker_scripts/install_R.sh /rocker_scripts/install_R.sh
  RUN /rocker_scripts/install_R.sh  # <- same as `rocker/r-ver`
  RUN ...
  COPY rocker_scripts/ /rocker_scripts/

PoC can be found here. https://github.com/eitsupi/rocker-versioned-next

[eitsupi]

Sorry for the slow work. I have not given up on completing this this week and here is the work in progress. eitsupi/rocker-versioned-next#1

I am thinking of making the following changes:

• rocker/binder and cuda images are excluded from the central bakefile.
• latest-daily images are being removed as they have not been able to build recently.

[cboettig]

All for removing latest-daily. why exclude rocker/binder though?

[eitsupi]

why exclude rocker/binder though?

As I wrote here, I think the current build strategy is not ideal for rocker/binder. https://github.com/rocker-org/rocker-versioned2/issues/614#issuecomment-2028107573

[cboettig]

:100: right, sorry I got confused between the two threads. I've wondered if we should be pinning python versions in rocker/binder, but the rationale there given the recent security patch etc makes sense, and easier to do this with the build strategy you suggest in https://github.com/rocker-org/rocker-versioned2/issues/614#issuecomment-2028107573 !