broken vl2png install - Githubissues

janrito commented 1 year ago

It looks like the versions of node and vega utils are getting messed up

train:
    needs: deploy-runner
    runs-on: [self-hosted, cml]
    ...
    container:
      # CML 0 DVC 3 UBUNTU 20.04
      image: docker://ghcr.io/iterative/cml:0-dvc3-base1
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11.5"

      - uses: iterative/setup-dvc@v1
      - uses: iterative/setup-cml@v2
    run: |
      ...
      dvc plots show --show-vega plots/roc.yaml >vega.json
      vl2png vega.json >ROC.png


+ vl2png vega.json
/usr/lib/node_modules/vega-lite/build/vega-lite.js:219
  const duplicate = structuredClone;
                    ^

ReferenceError: structuredClone is not defined
    at /usr/lib/node_modules/vega-lite/build/vega-lite.js:219:21
    at /usr/lib/node_modules/vega-lite/build/vega-lite.js:2:66
    at Object.<anonymous> (/usr/lib/node_modules/vega-lite/build/vega-lite.js:5:3)
    at Module._compile (node:internal/modules/cjs/loader:1198:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1252:10)
    at Module.load (node:internal/modules/cjs/loader:1076:32)
    at Function.Module._load (node:internal/modules/cjs/loader:911:12)
    at Module.require (node:internal/modules/cjs/loader:1100:19)
    at require (node:internal/modules/cjs/helpers:119:18)
    at Object.<anonymous> (/usr/lib/node_modules/vega-lite/bin/render.js:7:18)
Error: Process completed with exit code 1.```

shcheklein commented 1 year ago

I think you don't need to use both setup-cml and the custom image:

container:
      # CML 0 DVC 3 UBUNTU 20.04
      image: docker://ghcr.io/iterative/cml:0-dvc3-base1

(I'm not sure if that image is updated @0x2b3bfa0 @dacbd ?)

Try to replicated this example:

https://github.com/iterative/example-get-started/blob/main/.github/workflows/cml.yaml

dacbd commented 1 year ago

@janrito as @shcheklein mentioned since you are using all of the setup-* type actions I would not use the container, it has node16 and the latest version of vega-lite requires node18.

janrito commented 1 year ago

Thanks! is the container no longer supported/recommended?

janrito commented 1 year ago

I'm still struggling with this - with a self-hosted runner, if an image is not specified, it will use ubuntu 18, which won't work for any recent python for example. IS there a way of specifying the ubuntu-latest image used in github actions?

dacbd commented 1 year ago

@janrito can you share your whole workflow?

You can use container images, which is totally valid, I would just recommend that you create that container yourself with all the dependencies that your project/pipeline needs.

janrito commented 1 year ago

Sure, I don't think I'm doing anything particularly special.

Here is how I thought it should work:


jobs:
  deploy-runner:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: iterative/setup-cml@v2
      - name: Deploy runner on EC2
        run: |
          cml runner \
            ...
          --labels=cml

  train-model:
    needs: deploy-runner
    runs-on: [self-hosted, cml"]
    ...
    steps:
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11.5"

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 18

      - uses: iterative/setup-cml@v2

      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Install Python dependencies
        ...

      - name: Train model
        ...

However, this won't work because the default image is on ubuntu 18, which does not even have a python 3.11 version available to it. It probably won't get far with node either.

So, we can use your prebuilt image instead, which includes dvc and cml already


jobs:
  deploy-runner:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: iterative/setup-cml@v2
      - name: Deploy runner on EC2
        run: |
          cml runner \
            ...
          --labels=cml

  train-model:
    needs: deploy-runner
    runs-on: [self-hosted, cml"]
    ...
    container:
      # CML 0 DVC 3 UBUNTU 20.04
      image: docker://ghcr.io/iterative/cml:0-dvc3-base1
    steps:
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11.5"

      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Install Python dependencies
        ...

      - name: Train model
        ...

This is ubuntu 20 and works. But you are telling me that this is an unsupported image that uses an old version of node.

I don't want this to break immediately, so the alternative is to use a generic image:


jobs:
  deploy-runner:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: iterative/setup-cml@v2
      - name: Deploy runner on EC2
        run: |
          cml runner \
            ...
          --labels=cml

  train-model:
    needs: deploy-runner
    runs-on: [self-hosted, cml"]
    ...
    container:
      image: ubuntu:latest
    steps:
      - name: Setup build requirements
        run: apt update && apt install -y build-essential git python3-dev python3-pip lz4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11.5"

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 18

      - uses: iterative/setup-cml@v2

      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Install Python dependencies
        ...

      - name: Train model
        ...

This is ubuntu 22 and works fine. But I have to set up git, install a compiler and setup python (twice). It would be nice if I just had a simple image to use, even the default GitHub runner one.

dacbd commented 1 year ago

@janrito Yeah, I'm sorry for the inconvenience here. There is a somewhat hidden option in cml runner for you to select a different ec2 image, (the provisioning of the instance might take slightly longer) like so: --cloud-image="ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20221206" \ see here as an example:

I would do the following:

name: Build CI Workflow Environment
on:
  workflow_dispatch:
  push:
    paths:
      - .github/env/training/Dockerfile

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Setup docker buildx
        uses: docker/setup-buildx-action@v3
      - name: Build and Push
        uses: docker/build-push-action@v5
        with:
          context: .github/env/training
          file: .github/env/training/Dockerfile
          push: true
          tags: 'ghcr.io/${{ github.repository }}:environment-training'


name: use container built in repo
on: 
  push:
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    container: 
      image: ghcr.io/dacbd/err:environment-training
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v3
        with:
          python-version: '3.11.5'
      - uses: iterative/setup-cml@v2
      - name: test
        run: cml --version
      - name: train
        run: python --version

links:

janrito commented 1 year ago

This + cml would be really useful to just have available

iterative / setup-cml

broken vl2png install #96