search

catchpoint / workflow-telemetry-action

Github action to collect metrics (CPU, memory, I/O, etc ...) from your workflows to help you debug and optimize your CI/CD pipeline
Apache License 2.0
283 stars 37 forks source link

workflow-telemetry-action: `Error: EACCES: permission denied` #57

Open bschilder opened 8 months ago

bschilder commented 8 months ago

Hi there, thanks for the awesome action! I'm currently using it as part of the rworkfows action and one of my users reported an error.

Any insights you might be able to provide would be greatly appreciated! @rwxdash @suleymanbarman

1. Bug description

An error arises during the telemetry step in the rworkflows action.

Console output

https://github.com/tempbioc/parody/actions/runs/7700556648/job/20984649706

Run runforesight/workflow-telemetry-action@v1
  with:
    github_token: ***
    comment_on_pr: false
    metric_frequency: 5
    proc_trace_min_duration: -1
    proc_trace_sys_enable: false
    proc_trace_chart_show: true
    proc_trace_chart_max_count: 100
    proc_trace_table_show: false
    job_summary: true
    theme: light
/usr/bin/docker exec  6c2e97cc4a283c72fe7bf139a67a7da38acadf3432c0c132475639d0cd[27](https://github.com/tempbioc/parody/actions/runs/7700556648/job/20984649706#step:4:28)aae2 sh -c "cat /etc/*release | grep ^ID"
[Workflow Telemetry] Initializing ...
[Workflow Telemetry] Starting step tracer ...
[Workflow Telemetry] Started step tracer
[Workflow Telemetry] Starting stat collector ...
[Workflow Telemetry] Started stat collector
[Workflow Telemetry] Starting process tracer ...
[Workflow Telemetry] Using proc_tracer_ubuntu-22
Error: [Workflow Telemetry] Unable to start process tracer
Error: [Workflow Telemetry] Error
Error: Error: EACCES: permission denied, open '/__w/_temp/_runner_file_commands/save_state_ac4bd7c5-921e-48b3-b0[29](https://github.com/tempbioc/parody/actions/runs/7700556648/job/20984649706#step:4:30)-d0cb99c12220'
[Workflow Telemetry] Initialization completed
Run echo "RGL_USE_NULL=TRUE" >> $GITHUB_ENV
/__w/_temp/06874ccf-cdc5-4e06-aed4-f0e66a4aa052.sh: line 1: /__w/_temp/_runner_file_commands/set_env_1542[31](https://github.com/tempbioc/parody/actions/runs/7700556648/job/20984649706#step:4:32)dd-e084-4f24-9ded-f0a82f647c04: Permission denied
/__w/_temp/06874ccf-cdc5-4e06-aed4-f0e66a4aa052.sh: line 2: /__w/_temp/_runner_file_commands/set_env_15[42](https://github.com/tempbioc/parody/actions/runs/7700556648/job/20984649706#step:4:44)31dd-e084-4f24-9ded-f0a82f647c04: Permission denied
/__w/_temp/06874ccf-cdc5-4e06-aed4-f0e66a4aa052.sh: line 3: /__w/_temp/_runner_file_commands/set_env_154231dd-e084-4f24-9ded-f0a82f647c04: Permission denied
/__w/_temp/06874ccf-cdc5-4e06-aed4-f0e66a4aa052.sh: line 4: /__w/_temp/_runner_file_commands/set_env_154231dd-e084-4f24-9ded-f0a82f647c04: Permission denied
/__w/_temp/06874ccf-cdc5-4e06-aed4-f0e66a4aa052.sh: line 5: /__w/_temp/_runner_file_commands/set_env_154231dd-e084-4f24-9ded-f0a82f647c04: Permission denied
Error: Process completed with exit code 1.

Expected behaviour

Telemetry actions runs all the way through and produces reports.

2. Reproducible example

https://github.com/tempbioc/parody/actions/runs/7700556648/workflow

Potentially related issues

rwxdash commented 8 months ago

Hey @bschilder, thank you for raising an issue.

At a glance, the workflow with the problem seems to be using a container to run the job. Since the workflow-telemetry-action uses an eBPF program (called proc_tracer_ubuntu-22 in this case) and it would have to be run with sudo, it would need security privileges to run in a container.

As you pointed out (#44), the solution would be to run the container with privileged: true and mount the debugfs syscalls, since it probably won't be available within a container by default.

It does look like GitHub Action allows these to be set in the jobs.<job_id>.container.options. See: https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container#setting-container-resource-options

Hope this clarifies some things and helps you resolve the issue.

Bests.

bschilder commented 8 months ago

Thanks so much for the rapid and helpful reply! I'll try this out and let you know how it goes!

serkan-ozal commented 7 months ago

Hi @bschilder,

Had you a chance to try solution suggested by @rwxdash?