green-coding-solutions / eco-ci-energy-estimation

Eco CI Energy estimation for Github Actions Runner VMs
MIT License
48 stars 10 forks source link

Github enterprise support: Install go if not present #16

Closed willi-mueller closed 11 months ago

willi-mueller commented 1 year ago

I think this workflow is currently crashing if the runner does not have go installed. Could you thus change the workflow to install go if not present? This would support machines configured on github enterprise which do not have go installed.

I tried running this workflow on our github enterprise setup but got the following error. Below follows the raw log. I replaced the name of our organization and repo with my-org and my-repo respectively:

2023-04-11T18:03:15.9643409Z Requested labels: size/small
2023-04-11T18:03:15.9643786Z Job defined at: my-org/my-repo/.github/workflows/sql_lint_and_docker_build.yml@refs/heads/trx-7941_lint-all-code
2023-04-11T18:03:15.9643932Z Waiting for a runner to pick up this job...
2023-04-11T18:03:16.5599357Z Job is about to start running on the runner: my-org-small-runner-deployment-hqhsn-x9zvw (organization)
2023-04-11T18:03:20.7330234Z Current runner version: '2.299.1'
2023-04-11T18:03:20.7339132Z Runner name: 'my-org-small-runner-deployment-hqhsn-x9zvw'
2023-04-11T18:03:20.7340036Z Runner group name: 'Default'
2023-04-11T18:03:20.7341265Z Machine name: 'my-org-small-runner-deployment-hqhsn-x9zvw'
2023-04-11T18:03:20.7344169Z ##[group]GITHUB_TOKEN Permissions
2023-04-11T18:03:20.7345023Z Actions: write
2023-04-11T18:03:20.7345351Z Checks: write
2023-04-11T18:03:20.7345739Z Contents: write
2023-04-11T18:03:20.7346077Z Deployments: write
2023-04-11T18:03:20.7346454Z Discussions: write
2023-04-11T18:03:20.7346820Z Issues: write
2023-04-11T18:03:20.7347175Z Metadata: read
2023-04-11T18:03:20.7347518Z Packages: write
2023-04-11T18:03:20.7347839Z Pages: write
2023-04-11T18:03:20.7348234Z PullRequests: write
2023-04-11T18:03:20.7348686Z RepositoryProjects: write
2023-04-11T18:03:20.7349051Z SecurityEvents: write
2023-04-11T18:03:20.7349360Z Statuses: write
2023-04-11T18:03:20.7349675Z ##[endgroup]
2023-04-11T18:03:20.7354916Z Secret source: Actions
2023-04-11T18:03:20.7355513Z Prepare workflow directory
2023-04-11T18:03:20.8453144Z Prepare all required actions
2023-04-11T18:03:20.8715818Z Getting action download info
2023-04-11T18:03:21.7182247Z Download action repository 'green-coding-berlin/eco-ci-energy-estimation@53fc78a7eed614c9c8d21a98fe992e790e91d8af' (SHA:53fc78a7eed614c9c8d21a98fe992e790e91d8af)
2023-04-11T18:03:22.4494459Z Download action repository 'actions/checkout@v3' (SHA:8f4b7f84864484a7bf31766abe9204da3cbe65b3)
2023-04-11T18:03:22.8057423Z Download action repository 'green-coding-berlin/eco-ci-energy-estimation@main' (SHA:08c0aa1f16bfc2b69092f508d8dd95a6bbbfe447)
2023-04-11T18:03:23.3250224Z Download action repository 'actions-internal/google-github-actions_auth@v0.8.0' (SHA:ceee102ec2387dd9e844e01b530ccd4ec87ce955)
2023-04-11T18:03:23.5949293Z Download action repository 'actions-internal/google-github-actions_setup-gcloud@v0' (SHA:ee9693ff89cdf73862b8a13988f6a71070e8fc58)
2023-04-11T18:03:24.0835533Z Getting action download info
2023-04-11T18:03:24.2992281Z Download action repository 'actions/cache@v3' (SHA:88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8)
2023-04-11T18:03:25.0645578Z Getting action download info
2023-04-11T18:03:25.2640365Z Getting action download info
2023-04-11T18:03:25.4973641Z Getting action download info
2023-04-11T18:03:25.7571340Z Uses: my-org/my-repo/.github/workflows/sql_lint_and_docker_build.yml
2023-04-11T18:03:25.7575394Z ##[group] Inputs
2023-04-11T18:03:25.7575991Z   ENVIRONMENT: pr
2023-04-11T18:03:25.7576416Z ##[endgroup]
2023-04-11T18:03:25.7577869Z Complete job name: Development code linting & Docker build / Development code linting & Docker build
2023-04-11T18:03:25.8283969Z A job started hook has been configured by the self-hosted runner administrator
2023-04-11T18:03:25.8494745Z ##[group]Run '/etc/arc/hooks/job-started.sh'
2023-04-11T18:03:25.8518409Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-04-11T18:03:25.8518784Z ##[endgroup]
2023-04-11T18:03:25.8917432Z 2023-04-11 18:03:25.881  DEBUG --- Running ARC Job Started Hooks
2023-04-11T18:03:25.8918827Z 2023-04-11 18:03:25.884  DEBUG --- Running hook: /etc/arc/hooks/job-started.d/update-status
2023-04-11T18:03:25.9500292Z ##[group]Run green-coding-berlin/eco-ci-energy-estimation@53fc78a7eed614c9c8d21a98fe992e790e91d8af
2023-04-11T18:03:25.9500783Z with:
2023-04-11T18:03:25.9501049Z   task: start-measurement
2023-04-11T18:03:25.9501515Z   branch: trx-7941_lint-all-code
2023-04-11T18:03:25.9501962Z   send-data: true
2023-04-11T18:03:25.9502242Z ##[endgroup]
2023-04-11T18:03:25.9826940Z ##[group]Run if [[ -d /tmp/eco-ci ]]; then
2023-04-11T18:03:25.9827377Z if [[ -d /tmp/eco-ci ]]; then
2023-04-11T18:03:25.9827738Z   rm -rf /tmp/eco-ci
2023-04-11T18:03:25.9828087Z fi
2023-04-11T18:03:25.9828382Z mkdir /tmp/eco-ci
2023-04-11T18:03:25.9829066Z git clone --depth 1 --single-branch --branch main https://github.com/green-coding-berlin/spec-power-model /tmp/eco-ci/spec-power-model
2023-04-11T18:03:25.9856258Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-04-11T18:03:25.9856638Z ##[endgroup]
2023-04-11T18:03:25.9992788Z Cloning into '/tmp/eco-ci/spec-power-model'...
2023-04-11T18:03:28.1742651Z ##[group]Run go install github.com/guptarohit/asciigraph/cmd/asciigraph@latest
2023-04-11T18:03:28.1743293Z go install github.com/guptarohit/asciigraph/cmd/asciigraph@latest
2023-04-11T18:03:28.1777038Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-04-11T18:03:28.1777433Z ##[endgroup]
2023-04-11T18:03:28.1847618Z /runner/_work/_temp/7d25ecc4-57f3-45ec-a407-e38ed7532a13.sh: line 1: go: command not found
2023-04-11T18:03:28.1884038Z ##[error]Process completed with exit code 127.
2023-04-11T18:03:28.2398168Z Post job cleanup.
2023-04-11T18:03:28.2413363Z A job completed hook has been configured by the self-hosted runner administrator
2023-04-11T18:03:28.2465461Z ##[group]Run '/etc/arc/hooks/job-completed.sh'
2023-04-11T18:03:28.2495540Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-04-11T18:03:28.2496133Z ##[endgroup]
2023-04-11T18:03:28.2621552Z 2023-04-11 18:03:28.260  DEBUG --- Running ARC Job Completed Hooks
2023-04-11T18:03:28.2659848Z 2023-04-11 18:03:28.264  DEBUG --- Running hook: /etc/arc/hooks/job-completed.d/update-status
2023-04-11T18:03:28.3044016Z Cleaning up orphan processes

If someone could pick that up it would be great. I'm working for a large multinational organization with dozens of development teams and some of us would love to introduce your tool to get awareness of and improve our energy consumption! Thank you!

ArneTR commented 1 year ago

Well thats an easy addon, sure we can do that.

But to better understand the issue: We are using the standard Github Image. Does Github provide a different image for Enterprise users and could you point us to it?

Or are you running a custom image?

willi-mueller commented 1 year ago

Excellent! I don't know the details of the runner, I suppose it's a custom image. I can ask our ops team. What would you need to know? Probably a list of what is installed in our image to compare that with what eco-ci needs?

ArneTR commented 1 year ago

Thanks for the update.

With a fresh head and now reading it again I have to say if you actually use a custom image I see the installation as a bit more tricky.

Since we are just an action we want to "dirty" the actual machine as minimal as we can. So installing go on the box could conflict with current versions and / or leave an unintended binary.

However to actually comfort your request I have following idea: What would you think if we just make the graph optional, than you can still use the action without any modification and just will not see the graph. If you really want to see it, than I think it is best if you install go on your custom image ... so we are not forcing people to get go installed through the action.

What you think about that?

willi-mueller commented 1 year ago

Excellent idea, I appreciate that you revisited your first proposal and, in the end, did not confused by my initial thoughts :D

dan-mm commented 1 year ago

Hi @jups23

I have added an optional flag called display-graph, which you can use to circumnavigate your issues

If you call the start-measurement task with that flag set to false, it will skip the go-installation. You must then also call the display-results task with that flag set to false as well, otherwise it will still try to display the graph. The README has also been updated to reflect this information.

Hope this solution works for you, if not, please let us know!

ArneTR commented 1 year ago

@jups23 Is the feature working for you? can this be closed?

willi-mueller commented 1 year ago

Thank you for the follow-up! Yes, this flag is great!

I needed to install python 3.10, that's not available on our custom runner using

- uses: actions/setup-python@v4
  with:
    python-version: '3.10' 

However, I got stuck with the following error now. Can you help us?

2023-05-08T12:42:11.1660948Z Cache hit failed! ❌
2023-05-08T12:42:11.1682575Z ##[group]Run # first activate our venv
2023-05-08T12:42:11.1682925Z # first activate our venv
2023-05-08T12:42:11.1683273Z source /tmp/eco-ci/venv/bin/activate
2023-05-08T12:42:11.1683702Z python3.10 -m pip install -r /tmp/eco-ci/spec-power-model/requirements.txt
2023-05-08T12:42:11.1684094Z # now reset to old venv
2023-05-08T12:42:11.1684695Z deactivate our venv
2023-05-08T12:42:11.1685094Z # reactivate the old one, if it was present
2023-05-08T12:42:11.1685469Z if [[ $ECO_CI_PREVIOUS_VENV != '' ]]; then
2023-05-08T12:42:11.1686316Z   source $ECO_CI_PREVIOUS_VENV/bin/activate
2023-05-08T12:42:11.1686650Z fi
2023-05-08T12:42:11.1703768Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-05-08T12:42:11.1704113Z env:
2023-05-08T12:42:11.1704570Z   pythonLocation: /opt/hostedtoolcache/Python/3.10.11/x64
2023-05-08T12:42:11.1705017Z   PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.10.11/x64/lib/pkgconfig
2023-05-08T12:42:11.1705430Z   Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.11/x64
2023-05-08T12:42:11.1705830Z   Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.11/x64
2023-05-08T12:42:11.1706230Z   Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.11/x64
2023-05-08T12:42:11.1706637Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.11/x64/lib
2023-05-08T12:42:11.1706982Z ##[endgroup]
2023-05-08T12:42:11.7445826Z Collecting contourpy==1.0.7
2023-05-08T12:42:11.8690587Z   Downloading contourpy-1.0.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (300 kB)
2023-05-08T12:42:11.8969688Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 300.3/300.3 kB 11.9 MB/s eta 0:00:00

[... installing many dependencies ...]

2023-05-08T12:42:42.8656939Z echo "ECO_CI_WORKFLOW_ID=$workflow_id" >> $GITHUB_ENV
2023-05-08T12:42:42.8657263Z 
2023-05-08T12:42:42.8657686Z /runner/_work/_actions/green-coding-berlin/eco-ci-energy-estimation/v1/scripts/init_variables.sh
2023-05-08T12:42:42.8658209Z echo "ECO_CI_API_BASE=https://api.green-coding.berlin" >> $GITHUB_ENV
2023-05-08T12:42:42.8658810Z echo "ECO_CI_INIT=DONE" >> $GITHUB_ENV;
2023-05-08T12:42:42.8683981Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-05-08T12:42:42.8684801Z env:
2023-05-08T12:42:42.8685292Z   pythonLocation: /opt/hostedtoolcache/Python/3.10.11/x64
2023-05-08T12:42:42.8686194Z   PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.10.11/x64/lib/pkgconfig
2023-05-08T12:42:42.8686656Z   Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.11/x64
2023-05-08T12:42:42.8687054Z   Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.11/x64
2023-05-08T12:42:42.8687457Z   Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.11/x64
2023-05-08T12:42:42.8687861Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.11/x64/lib
2023-05-08T12:42:42.8688211Z ##[endgroup]
2023-05-08T12:42:43.0489531Z ##[error]Unable to process file command 'env' successfully.
2023-05-08T12:42:43.0501636Z ##[error]Invalid format '4959'
2023-05-08T12:42:43.0782497Z Post job cleanup.
2023-05-08T12:42:43.0802161Z A job completed hook has been configured by the self-hosted runner administrator

I suppose that this error has nothing to do with go, so let me know if you'd like me to open a new issue and we close this one.

ArneTR commented 1 year ago

This error I have never seen before. If you are not using the provided Github default image it might be that some features that should be available in Github Actions are not working as expected.

What happens for instance if in your jobs you write to GITHUB_STEP_SUMMARY or you access the env context (https://docs.github.com/en/actions/learn-github-actions/contexts#env-context)

I have a suspicion that particular the latter might not be available ...

dan-mm commented 11 months ago

@willi-mueller : the eco-ci has undergone a rewrite since we've last discussed this, especially in how the setup/step-summary/env are accessed, so your issue might be fixed now. Are you still experiencing this issue?

willi-mueller commented 11 months ago

Thank you! I left the team where I observed the problem, but I'll forward the good news with an invitation to try it out again at the next opportunity. Thank you!

I'd therefore close the issue.