Autograding action not able to read GitHub Secrets (Actions)

[unaihuete93]

Hello,

I was testing running some pytest to test some Python files and it fails when getting the GitHub Secrets from environment variables.

autograding workflow

name: GitHub Classroom Workflow

on: [push]

permissions:
  checks: write
  actions: read
  contents: read

env:
  COG_SERVICE_ENDPOINT: ${{ secrets.COG_SERVICE_ENDPOINT }}
  COG_SERVICE_KEY: ${{ secrets.COG_SERVICE_KEY }}

jobs:
  build:
    name: Autograding
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: education/autograding@v1

autograding.json

{
  "tests": [
    {
      "name": "Test 1.1",
      "setup": "sh .devcontainer/post-create.sh",
      "run": "pytest 1-rest-client.py",
      "input": "",
      "output": "",
      "comparison": "included",
      "timeout": 10,
      "points": null
    }
  ]
}

post-create.sh installs all neccesary python libraries.

This is the error I get, it does not read the env variable.

Running another custom workflow (one below) executing my pytest file works. What am I missing?

name: Python execution
on: [push]

permissions:
  checks: write
  actions: read
  contents: read

env:
  COG_SERVICE_ENDPOINT: ${{ secrets.COG_SERVICE_ENDPOINT }}
  COG_SERVICE_KEY: ${{ secrets.COG_SERVICE_KEY }}

jobs:
  build:
    name: Python test execution
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: sh .devcontainer/post-create.sh
      - run: pytest 1-rest-client.py

[unaihuete93]

the example repo can be found here: https://github.com/unhueteb-org/Tecnun2023-Azure-VisionAI

[markpatterson27]

When autograding runs grading tests, it runs them in a child process with very few environment variables brought in.

https://github.com/education/autograding/blob/1d058ce58864938499105ab5cd1c941651ce7e27/src/runner.ts#L128-L135

You could try running the pytest tests as seperate steps. If the test passes, drop a file. Then in autograding test if the file exists. Something like:

...
      # delete and recreate result dir
      - name: Reset results dir
        run: |
          rm -rf .github/results
          mkdir -p .github/results

      # Test 1.1
      - run: pytest 1-rest-client.py && touch .github/results/test1-1-pass
      # Test 1.2
      - run: pytest 1-sdk-client.py && touch .github/results/test1-2-pass
...
      - uses: education/autograding@v1

Then in .github/classroom/autograding.json:

{
  "tests": [
    {
      "name": "Activity 1 - Accept assignment",
      "setup": "",
      "run": "[ -e .github/results/test1-1-pass ] && exit 0 || exit 1",
      "input": "",
      "output": "",
      "comparison": "included",
      "timeout": 10,
      "points": 1
    },
{
      "name": "Activity 1 - Accept assignment",
      "setup": "",
      "run": "[ -e .github/results/test1-2-pass ] && exit 0 || exit 1",
      "input": "",
      "output": "",
      "comparison": "included",
      "timeout": 10,
      "points": 1
    },
  ]
}

etc

[unaihuete93]

Thanks for the reply @markpatterson27 , I will try it! :)

[gpincheiraa]

Hi! using a file for save the required environment variables and then read it from autograding tasks load using "source" command do the trick:

classroom.yml

jobs:
  build:
    name: Autograding
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: |
          echo GITHUB_ACTOR="$GITHUB_ACTOR" >> owner-environment
          echo GITHUB_REPOSITORY_OWNER="$GITHUB_REPOSITORY_OWNER" >> owner-environment
      - uses: education/autograding@v1

autograding.json

{
  "tests": [
    {
      "name": "Verificacion Links a otras paginas",
      "setup": "",
      "run": "source owner-environment && cat owner-environment",
      "input": "",
      "output": "",
      "comparison": "included",
      "timeout": 10,
      "points": 100
    }
  ]
}

The important part is source owner-environment for load the environment variables in your custom script.

[booleanchile]

it is possible another method to read variables?

[sgbaird]

@markpatterson27

Does using the file-detection based approach vs. an environment variable-setting script approach pose any major differences in terms of "security concerns" (xref: https://github.com/education/autograding/issues/19#issuecomment-627435284 and https://github.com/education/autograding/issues/19#issuecomment-1351330956)? My intuition is that either way, the robust method for preventing "circumvention" is for the teacher to run the tests in a private instance (e.g., locally) and use those grades directly or check for mismatches between GitHub Classroom grades via the CLI.

@booleanchile I tried saving a sitecustomize.py file, but the env vars are still not recognized. I guess children processes in this context don't run sitecustomize.py. I've considered using something like python-dotenv, but that still requires you to create a .env file, install the package, and use from dotenv import loadenv(); loadenv() inside of any applicable Python scripts. The previous suggestions (https://github.com/education/autograding/issues/69#issuecomment-1384557654 and https://github.com/education/autograding/issues/69#issuecomment-1497674655) seem like the most reasonable options without modifications to your actual assignment code.

"Using secrets in a workflow" gh docs are very relevant.

@gpincheiraa

using a file for save the required environment variables and then read it from autograding tasks load using "source" command do the trick:

I kept getting source command not found or similar. After a lot of troubleshooting, I found something that works for me using . ./<script_name> instead of source <script_name>. For example:

autograding.json

{
    "tests": [
        {
            "name": "GitHub secrets test (env vars)",
            "setup": "sudo -H pip3 install -r requirements.txt",
            "run": ". ./setenv.sh && pytest github_secrets_test.py::test_env_vars_exist",
            "input": "",
            "output": "",
            "comparison": "exact",
            "timeout": 5,
            "points": 2
        },
        {
            "name": "Orchestrator client test",
            "setup": "sudo -H pip3 install -r requirements.txt",
            "run": ". ./setenv.sh && pytest orchestrator_client_test.py",
            "input": "",
            "output": "",
            "comparison": "exact",
            "timeout": 5,
            "points": 3
        },
        {
            "name": "Microcontroller client test",
            "setup": "sudo -H pip3 install -r requirements.txt",
            "run": ". ./setenv.sh && pytest microcontroller_client_test.py",
            "input": "",
            "output": "",
            "comparison": "exact",
            "timeout": 5,
            "points": 3
        },
        {
            "name": "GitHub secrets test (basic comms)",
            "setup": "sudo -H pip3 install -r requirements.txt",
            "run": ". ./setenv.sh && pytest github_secrets_test.py::test_basic_hivemq_communication",
            "input": "",
            "output": "",
            "comparison": "exact",
            "timeout": 5,
            "points": 2
        }
    ]
}

classroom.yml

name: GitHub Classroom Workflow

on:
  - push
  - workflow_dispatch

permissions:
  checks: write
  actions: read
  contents: read

jobs:
  build:
    name: Autograding
    runs-on: ubuntu-latest
    if: github.actor != 'github-classroom[bot]'
    steps:
      - uses: actions/checkout@v4
      - name: Create file for setting env vars
        # https://github.com/education/autograding/issues/69#issuecomment-1497674655
        # https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#using-secrets-in-a-workflow
        env:
          HIVEMQ_HOST: ${{ secrets.HIVEMQ_HOST }}
          HIVEMQ_USERNAME: ${{ secrets.HIVEMQ_USERNAME }}
          HIVEMQ_PASSWORD: ${{ secrets.HIVEMQ_PASSWORD }}
          COURSE_ID: ${{ secrets.COURSE_ID }}
        run: |
          echo "#!/bin/sh" > setenv.sh
          echo "export HIVEMQ_HOST=\"$HIVEMQ_HOST\"" >> setenv.sh
          echo "export HIVEMQ_PASSWORD=\"$HIVEMQ_PASSWORD\"" >> setenv.sh
          echo "export HIVEMQ_USERNAME=\"$HIVEMQ_USERNAME\"" >> setenv.sh
          echo "export COURSE_ID=\"$COURSE_ID\"" >> setenv.sh
          chmod +x setenv.sh
      - uses: education/autograding@v1

Note: I'm not sure if chmod +x setenv.sh was necessary. I know that echo "#!/bin/sh" > setenv.sh is not strictly necessary, but I added it in anyway. It might be better to avoid #!/bin/bash (as opposed to #!/bin/sh) since the shell doesn't seem to be bash.