autowarefoundation / autoware-github-actions

Apache License 2.0
17 stars 24 forks source link

feat(colcon-build): use nice command to not overuse resources #308

Closed xmfcx closed 3 months ago

xmfcx commented 3 months ago

Description

nice is part of GNU coreutils, let's you run a program with modified scheduling priority.

Comes bundled in the our Ubuntu containers.

This PR aims to utilize it to prevent colcon build from overusing the runner resources.

man page

Run COMMAND with an adjusted niceness, which affects process scheduling. With no COMMAND, print the current niceness. Niceness values range from -20 (most favorable to the process) to 19 (least favorable to the process).

I set the default to be 19 in this PR.

Original default was 10.

Related PR:

Failure example for a runner due to over-usage of a machine during colcon build:

Tests performed

Effects on system behavior

Not applicable.

Interface changes

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

After all checkboxes are checked, anyone who has write access can merge the PR.

xmfcx commented 3 months ago

Ok works after removing the quotes too:

xmfcx commented 3 months ago

Even with the nice command, https://github.com/autowarefoundation/autoware.universe/actions/runs/10454375310/job/28946832098?pr=8498

this one has lost connection due to resource overuse.

Machine had 15GB RAM and 8vCPU and 0 swap.

Added a specs step for these: https://github.com/autowarefoundation/autoware.universe/actions/runs/10454375310/job/28946832098?pr=8498#step:5:1

xmfcx commented 3 months ago

We can still merge this but I will create another PR that enables user to set parallel workers or makeflags