Runtime manager for Tebako Ruby builds

[ronaldtse]

We want to pre-compile the Ruby runtimes on GitHub Actions, in the tebako-runtimes repository.

We need the notion of a "Tebako runtime repository" which will compile Ruby runtimes of:

• "version" (3.1.3, 3.2.4, 3.3.3, etc) x "architecture" (x86_64, arm64, aarch64) x "os" (linux glibc, linux musl, windows, macOS).

Each Ruby version needs to be patched to be useable by Tebako.

We need code to manage this, so we can compile Ruby runtimes into a local repository, and load Ruby runtimes from a remote repository.

When the local repository is uploaded to a GitHub repository, we can use it as a remote repository, allowing users to build/customize their own Ruby runtimes (with the necessary Tebako patches, of course).

---

Let's create a system for managing Tebako runtime repositories, both local and remote. We'll create a TebakoRuntimeRepository class that can handle local and remote repositories, and a TebakoRuntimeBuilder class for compiling Ruby runtimes with Tebako patches.

First, let's define the structure:

require 'fileutils'
require 'open-uri'
require 'json'
require 'digest'
require 'yaml'

module Tebako
  class TebakoRuntimeRepository
    METADATA_FILE = 'tebako_runtime_metadata.yml'

    def initialize(path)
      @path = path
      FileUtils.mkdir_p(@path)
      load_metadata
    end

    def add_runtime(version, os, arch, file_path)
      runtime_key = "#{version}-#{os}-#{arch}"
      sha256 = Digest::SHA256.file(file_path).hexdigest
      @metadata[runtime_key] = {
        'version' => version,
        'os' => os,
        'arch' => arch,
        'sha256' => sha256,
        'filename' => File.basename(file_path)
      }
      FileUtils.cp(file_path, File.join(@path, File.basename(file_path)))
      save_metadata
    end

    def get_runtime(version, os, arch)
      runtime_key = "#{version}-#{os}-#{arch}"
      runtime_info = @metadata[runtime_key]
      return nil unless runtime_info

      file_path = File.join(@path, runtime_info['filename'])
      return nil unless File.exist?(file_path)

      file_path
    end

    def list_runtimes
      @metadata.values
    end

    private

    def load_metadata
      metadata_path = File.join(@path, METADATA_FILE)
      @metadata = File.exist?(metadata_path) ? YAML.load_file(metadata_path) : {}
    end

    def save_metadata
      File.write(File.join(@path, METADATA_FILE), @metadata.to_yaml)
    end
  end

  class TebakoRuntimeBuilder
    def initialize(repository)
      @repository = repository
    end

    def build_runtime(version, os, arch)
      # Placeholder for actual build process
      puts "Building Ruby #{version} for #{os} (#{arch})..."

      # Clone Ruby source
      system("git clone https://github.com/ruby/ruby.git -b v#{version} ruby-#{version}")

      # Apply Tebako patches
      apply_tebako_patches(version)

      # Configure and build Ruby
      Dir.chdir("ruby-#{version}") do
        system("./configure --prefix=/usr/local")
        system("make -j#{Etc.nprocessors}")
        system("make install DESTDIR=#{Dir.pwd}/install")
      end

      # Package the built Ruby
      output_file = "ruby-#{version}-#{os}-#{arch}.tar.gz"
      system("tar -czf #{output_file} -C ruby-#{version}/install .")

      # Add to repository
      @repository.add_runtime(version, os, arch, output_file)

      puts "Ruby #{version} for #{os} (#{arch}) built and added to repository."
    end

    private

    def apply_tebako_patches(version)
      # Apply necessary Tebako patches
      # This is a placeholder and should be implemented based on specific Tebako requirements
      puts "Applying Tebako patches for Ruby #{version}..."
    end
  end

  class TebakoRuntimeManager
    def initialize(local_repo_path, remote_repo_url = nil)
      @local_repo = TebakoRuntimeRepository.new(local_repo_path)
      @remote_repo_url = remote_repo_url
    end

    def ensure_runtime(version, os, arch)
      runtime_path = @local_repo.get_runtime(version, os, arch)
      return runtime_path if runtime_path

      if @remote_repo_url
        download_runtime(version, os, arch)
      else
        build_runtime(version, os, arch)
      end
    end

    def build_runtime(version, os, arch)
      builder = TebakoRuntimeBuilder.new(@local_repo)
      builder.build_runtime(version, os, arch)
    end

    def download_runtime(version, os, arch)
      runtime_key = "#{version}-#{os}-#{arch}"
      metadata_url = "#{@remote_repo_url}/#{TebakoRuntimeRepository::METADATA_FILE}"
      metadata = YAML.load(URI.open(metadata_url).read)

      runtime_info = metadata[runtime_key]
      raise "Runtime not found in remote repository" unless runtime_info

      filename = runtime_info['filename']
      download_url = "#{@remote_repo_url}/#{filename}"
      local_path = File.join(@local_repo.instance_variable_get(:@path), filename)

      URI.open(download_url) do |remote_file|
        File.open(local_path, 'wb') do |local_file|
          local_file.write(remote_file.read)
        end
      end

      actual_sha256 = Digest::SHA256.file(local_path).hexdigest
      if actual_sha256 != runtime_info['sha256']
        FileUtils.rm(local_path)
        raise "Integrity check failed for downloaded runtime"
      end

      @local_repo.add_runtime(version, os, arch, local_path)
      local_path
    end

    def list_runtimes
      @local_repo.list_runtimes
    end
  end
end

This implementation provides:

1. TebakoRuntimeRepository: Manages a local repository of Ruby runtimes.
2. TebakoRuntimeBuilder: Builds Ruby runtimes with Tebako patches.
3. TebakoRuntimeManager: Manages both local and remote repositories, ensuring runtimes are available.

To use this system:

1. Set up a local repository:

local_repo = Tebako::TebakoRuntimeRepository.new('/path/to/local/repo')

2. Build a runtime:

builder = Tebako::TebakoRuntimeBuilder.new(local_repo)
builder.build_runtime('3.2.4', 'linux', 'x86_64')

3. Use the runtime manager to ensure a runtime is available:

manager = Tebako::TebakoRuntimeManager.new('/path/to/local/repo', 'https://github.com/tamatebako/tebako-runtimes/releases/latest/download')
runtime_path = manager.ensure_runtime('3.2.4', 'linux', 'x86_64')

To set up the GitHub Actions workflow for building runtimes in the tebako-runtimes repository, create a .github/workflows/build_runtimes.yml file:

name: Build Tebako Runtimes

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        ruby-version: ['3.1.3', '3.2.4', '3.3.3']
        include:
          - os: ubuntu-latest
            arch: x86_64
          - os: macos-latest
            arch: x86_64
          - os: windows-latest
            arch: x86_64

    runs-on: ${{ matrix.os }}

    steps:
    - uses: actions/checkout@v2

    - name: Set up Ruby
      uses: ruby/setup-ruby@v1
      with:
        ruby-version: ${{ matrix.ruby-version }}

    - name: Build Tebako Runtime
      run: |
        ruby -r ./lib/tebako/runtime_manager -e "
          repo = Tebako::TebakoRuntimeRepository.new('runtimes')
          builder = Tebako::TebakoRuntimeBuilder.new(repo)
          builder.build_runtime('${{ matrix.ruby-version }}', '${{ runner.os }}', '${{ matrix.arch }}')
        "

    - name: Upload Runtime
      uses: actions/upload-artifact@v2
      with:
        name: tebako-runtime-${{ matrix.ruby-version }}-${{ runner.os }}-${{ matrix.arch }}
        path: runtimes/ruby-${{ matrix.ruby-version }}-${{ runner.os }}-${{ matrix.arch }}.tar.gz

    - name: Upload Metadata
      uses: actions/upload-artifact@v2
      with:
        name: tebako-runtime-metadata
        path: runtimes/tebako_runtime_metadata.yml

This workflow will build Ruby runtimes for different versions, operating systems, and architectures. The built runtimes and metadata will be uploaded as artifacts.

To make these runtimes available as a remote repository, you can create a release in the tebako-runtimes repository and upload the artifacts to the release. Then, users can use the release URL as the remote repository URL when initializing the TebakoRuntimeManager.

This system allows for building custom Ruby runtimes with Tebako patches, managing them in local repositories, and distributing them via remote repositories (like GitHub releases). Users can either build their own runtimes or download pre-built ones from a remote repository.

[ronaldtse]

To automate the upload of built runtimes to GitHub releases, you can extend your GitHub Actions workflow. Here's how you can modify the build_runtimes.yml file to automatically create a release and upload the built runtimes:

name: Build and Release Tebako Runtimes

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
  workflow_dispatch:

jobs:
  build:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        ruby-version: ['3.1.3', '3.2.4', '3.3.3']
        include:
          - os: ubuntu-latest
            arch: x86_64
          - os: macos-latest
            arch: x86_64
          - os: windows-latest
            arch: x86_64

    runs-on: ${{ matrix.os }}

    steps:
    - uses: actions/checkout@v2

    - name: Set up Ruby
      uses: ruby/setup-ruby@v1
      with:
        ruby-version: ${{ matrix.ruby-version }}

    - name: Build Tebako Runtime
      run: |
        ruby -r ./lib/tebako/runtime_manager -e "
          repo = Tebako::TebakoRuntimeRepository.new('runtimes')
          builder = Tebako::TebakoRuntimeBuilder.new(repo)
          builder.build_runtime('${{ matrix.ruby-version }}', '${{ runner.os }}', '${{ matrix.arch }}')
        "

    - name: Upload Runtime Artifact
      uses: actions/upload-artifact@v2
      with:
        name: tebako-runtime-${{ matrix.ruby-version }}-${{ runner.os }}-${{ matrix.arch }}
        path: runtimes/ruby-${{ matrix.ruby-version }}-${{ runner.os }}-${{ matrix.arch }}.tar.gz

    - name: Upload Metadata Artifact
      uses: actions/upload-artifact@v2
      with:
        name: tebako-runtime-metadata
        path: runtimes/tebako_runtime_metadata.yml

  release:
    needs: build
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'

    steps:
    - uses: actions/checkout@v2

    - name: Download all workflow run artifacts
      uses: actions/download-artifact@v2

    - name: Prepare release assets
      run: |
        mkdir release_assets
        cp -r tebako-runtime-*/* release_assets/
        cp tebako-runtime-metadata/tebako_runtime_metadata.yml release_assets/

    - name: Create Release
      id: create_release
      uses: actions/create-release@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        tag_name: v${{ github.run_number }}
        release_name: Release ${{ github.run_number }}
        draft: false
        prerelease: false

    - name: Upload Release Assets
      uses: actions/upload-release-asset@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        upload_url: ${{ steps.create_release.outputs.upload_url }}
        asset_path: ./release_assets
        asset_name: tebako-runtimes.zip
        asset_content_type: application/zip

This updated workflow does the following:

1. The build job remains largely the same, building runtimes for each combination of OS, Ruby version, and architecture.

2. A new release job is added, which runs after all build jobs complete.

3. The release job:

   • Downloads all artifacts from the build jobs.
   • Prepares the release assets by copying all runtime files and the metadata file into a single directory.
   • Creates a new GitHub release using the actions/create-release action.
   • Uploads the release assets using the actions/upload-release-asset action.

4. The release is created only when there's a push to the main branch, ensuring that releases are only created for merged changes.

5. The release is tagged and named using the GitHub run number, which increments automatically with each workflow run.

To use this workflow:

1. Make sure your repository has the GITHUB_TOKEN secret available (this is typically provided automatically by GitHub Actions).

2. Commit this workflow file to your repository in the .github/workflows/ directory.

3. Push changes to the main branch or manually trigger the workflow using the "workflow_dispatch" event.

4. After the workflow runs successfully, a new release will be created with all the built runtimes and the metadata file.

This setup automates the entire process of building, packaging, and releasing Tebako runtimes. Users can then download these runtimes directly from the GitHub releases page, or use the release URL in their TebakoRuntimeManager to automatically download and use the pre-built runtimes.

Remember to update your TebakoRuntimeManager to handle downloading from a zip file if you decide to package all assets into a single zip file for the release.