Set up CI/CD pipeline

[dylan-gonzalez]

Setting up the foundation of our CI/CD pipeline

• [ ] research into CI/CD for autonomous vehicle systems

  • [ ] look into automated unit testing methods for logic in ROS nodes
  • [ ] look into frameworks for testing modules with sample inputs (sensor data, control commands etc)
  • [ ] create test cases (build tests and autoware scenarios),

• [ ] Pick out a style guide for the team to use, describe documentation and naming conventions (PEP8, Header and docstring comments, maintainers and editors); We have some old documentation guides on Notion

• [ ] setup CI/CD pipeline

  • [x] look into using the existing Autoware ci/cd pipeline
  • [ ] take a look at this: https://github.com/Monash-Connected-Autonomous-Vehicle/mcav-autoware/tree/main/.github/workflows
  • [ ] set up linters to check conventions; Create a ruff.toml for the python Ruff linter(https://github.com/astral-sh/ruff) and integrate into CI/CD and guide people to setting up locally

• [ ] Consider setting up dependency management guidelines; see python's poetry or pipenv dependency managers to freeze dependencies for stable builds

• ROS2 requires python 3.5 at least, we can target a specific python version up to 3.11 (3.12 was released a month ago so dependencies might not be available)

x2 people

[sxppro]

i'll need to look into how they're running the ci/cd pipeline as well as how we build autoware

[AnthonyZhOon]

We could start this off by meeting and sharing what we know about this, I'm on-campus weekdays but only have 1-2 hours until after 5pm

[npnquang]

I'm currently in Vietnam, so I cannot join the meeting in person, and it is 4 hours behind Australia. Other than that, I'm all good for a meeting

[dylan-gonzalez]

Btw can you guys prioritise figuring out how the rest of the team can write test cases for their different components?

[npnquang]

I found the folder storing the YAML files to run the CI/CD pipelines of Autoware. These are helpful to setup our own pipelines. In the build-main.yaml file, they run the colcon build command with some args.

[dylan-gonzalez]

i've assigned @Avocado-hash (Nguyen Nguyen) to this task

edit: he's working on sensing now

[dylan-gonzalez]

Next steps:

• organise a working session @npnquang @AnthonyZhOon @sxppro
• understand the different workflows for autoware, autoware.universe, autoware_launch, etc.
• test doing a push to a test branch --> try forcing an error/success

After that:

• see if we can launch AWSIM/CARLA from a workflow inside a docker container?

[sxppro]

Okay right now we're looking at the autoware.universe pipeline — specifically the build-and-test workflow. We are currently running into an error:

jsk_recognition_msgs: Cannot locate rosdep definition for [rostest]

that is causing the rosdeps to not be successfully installed.

Comparing our workflow run to Autoware's autoware.universe repo, they can install all the rosdeps successfully but I don't know why ours is different. We'll have to look more into this.

[sxppro]

Some more info we've looked at:

• the colcon build action that runs in the pipeline: https://github.com/autowarefoundation/autoware-github-actions/tree/main/colcon-build
• the workflow also references a container image by Autoware, which also contains a nice overview of their repositories: https://github.com/autowarefoundation/autoware/pkgs/container/autoware-universe

[sxppro]

We have identified the issue as jsk_recognition_msgs is a new package added by us that does not exist in the original repository.

We likely just need to install rostest as a rosdep in the pipeline (which is not currently being done by the workflow definition).

[npnquang]

I tried to find a way to install the rostest package that we are missing and then found this. It says that there is no version of rostest for ROS Humble. I think we need to look into why we have rostest in the first place.

[sxppro]

@dylan-gonzalez rostest is added as a testdepend here: https://github.com/Monash-Connected-Autonomous-Vehicle/autoware.universe/blob/cbb62d06a84e40a91036effc458408e3b55364fe/sensing/jsk_recognition_msgs/package.xml

[sxppro]

There's been some tests written with it: https://github.com/Monash-Connected-Autonomous-Vehicle/autoware.universe/tree/cbb62d06a84e40a91036effc458408e3b55364fe/sensing/jsk_recognition_msgs/test

[dylan-gonzalez]

Can you see if anyone has made an issue about it on the autoware.universe repo?

Or alternatively check this out and see if you can use g_tests instead??

[dylan-gonzalez]

Hey @AnthonyZhOon I'm thinking that we some repository that stores all relevant dotfiles for the MCAV PCs (and for people doing mcav stuff on their own laptops) so that we can just automatically clone that repo onto the MCAV PCs and have all the settings like bashrc, vscode extensions, etc. configured to be synced? Can we set something up like that?

[dylan-gonzalez]

also @AnthonyZhOon there's an autoware_lint_common package? Can someone look into that to see if we can use the linting that is already used in autoware?

[sxppro]

@Robin-Chea can u remember where you added these tests from in this commit or did you write them yourself? 😮 https://github.com/Monash-Connected-Autonomous-Vehicle/autoware.universe/commit/92f7bfa0e12a1cd6bfa9c2fcdd9ae111e9664f40

[Robin-Chea]

I dont remember :)

[sxppro]

I dont remember :)

surely u know whats the commit for right 😬

[Robin-Chea]

😬

[AnthonyZhOon]

Local install https://docs.astral.sh/ruff/tutorial/ Git page with docs https://github.com/astral-sh/ruff Place a ruff.toml in the top-closest relevant directory, can do multiple in the file structure

[dylan-gonzalez]

tomorrow night 19-12-23

@sxppro @AnthonyZhOon we'll go through some research on running headless simulations through the CI/CD pipeline