Generate Debian packages for the ROS packages in Autoware.core and Autoware.universe

esteve commented 1 year ago

Checklist

[X] I've read the contribution guidelines.
[X] I've searched other issues and no duplicate issues were found.
[X] I've agreed with the maintainers that I can plan this task.

Description

Given that Ubuntu is the platform of choice for developing and running Autoware, providing binary Debian packages will help users deploying Autoware

Purpose

Provide Debian packages for our users and developers

Possible approaches

Integrate https://github.com/jspricke/ros-deb-builder-action into our GitHub actions workflow

Definition of done

Debian packages are generated for all ROS packages and can be downloaded from our repositories

Related issues:

isamu-takagi commented 1 year ago

I made some changes to actions to support a differential build. I would like to decide a release operation of Autoware and propose features to upstream to support it.

https://github.com/tier4/autoware-apt-testing/tree/main https://github.com/isamu-takagi/ros-deb-builder-action/tree/develop (contains a lot of temporary code)

The features I think we need are:

support differential build
support multiple suite/components/architectures
use version in package.xml
keep the old versions

So first, we need to decide how to release Autoware and how to manage the package version.

isamu-takagi commented 1 year ago

My idea is that each package will only build when the version tag in package.xml is changed, and the repository keeps all previously built versions.

esteve commented 1 year ago

@isamu-takagi thanks for listing all the features, really nice.

I prefer to follow what MoveIt is doing:

https://github.com/moveit/moveit2_packages/blob/main/.github/workflows/build.yaml#L6

it runs once every day (in MoveIt's case, at 1:17am), so there's no need for differential builds, because it doesn't runs when a new branch is merged. But I think using the version from package.xml would be a useful alternative to using tags, though it's strongly encouraged for ROS packages/repositories to use tags for every new release.

esteve commented 1 year ago

@isamu-takagi I tried adding you as an assignee to this issue, but for some reason your username does not show up on the list. @kenji-miyake do you know why I can't add @isamu-takagi ?

isamu-takagi commented 1 year ago

it runs once every day (in MoveIt's case, at 1:17am), so there's no need for differential builds, because it doesn't runs when a new branch is merged.

My concern is that the build takes a long time. If the dependent packages have not changed, it is efficient to use the released ones.

But I think using the version from package.xml would be a useful alternative to using tags, though it's strongly encouraged for ROS packages/repositories to use tags for every new release.

I think there is a request to use the old package version. in this case, If the package has not changed, the same binary will be generated as a different version.

kenji-miyake commented 1 year ago

@kenji-miyake do you know why I can't add @isamu-takagi ?

@esteve Hmm, sorry I'm not sure. I believe you have Maintain access to this repository. For now, I'll add him.

jspricke commented 1 year ago

My concern is that the build takes a long time. If the dependent packages have not changed, it is efficient to use the released ones.

I measured about a minute for a package that is already in ccache on the free Github VMs. We can speed it up by using a ram disk if we have more powerful VMs. Also we think about parallelizing the build. Note that it is not enough to compare the diff in the repo itself as upstream dependencies could have changed and require a rebuild (ABI changes, message changes..).

But I think using the version from package.xml would be a useful alternative to using tags, though it's strongly encouraged for ROS packages/repositories to use tags for every new release.

I think there is a request to use the old package version. in this case, If the package has not changed, the same binary will be generated as a different version.

Currently the package includes the date as a Debian revision to make sure new version are recognized as an update by apt. This is the same for the OSRF buildfarm. Generating a Debian revision only in case of a change is not trivial but we can discuss that.

Can you please send relevant patches upstream so we can discuss them?

esteve commented 1 year ago

Can you please send relevant patches upstream so we can discuss them?

We will when we feel like we have something that can be upstreamed, until then we'll continue with our experiments here.

isamu-takagi commented 1 year ago

Can you please send relevant patches upstream so we can discuss them?

https://github.com/isamu-takagi/ros-deb-builder-action/pull/1 This is the current diff, but not enough discussion. I would like to continue the discussion here.

Currently the package includes the date as a Debian revision to make sure new version are recognized as an update by apt. This is the same for the OSRF buildfarm. Generating a Debian revision only in case of a change is not trivial but we can discuss that.

The autoware.universe repository contains a lot of packages, so versioning with tags is not suitable. Multiple versions of deb packages for each tag is generated even if the code of stable package hasn't changed. So in current repository structure, we need versioning for each package.

If versioning by tag, the autoware.universe repository should be divided into multiple small repositories. This allows the repository/package maintainers to set the version at their release timing.

~If using this method, It's need to deploy deb packages to another repository for distribution.~ I noticed sources.repos resolve this.

jspricke commented 1 year ago

isamu-takagi/ros-deb-builder-action#1 This is the current diff, but not enough discussion. I would like to continue the discussion here.

Fine with me. I'm also open to a video call if you are interested.

The autoware.universe repository contains a lot of packages, so versioning with tags is not suitable. Multiple versions of deb packages for each tag is generated even if the code of stable package hasn't changed. So in current repository structure, we need versioning for each package.

If versioning by tag, the autoware.universe repository should be divided into multiple small repositories. This allows the repository/package maintainers to set the version at their release timing.

I think that is a good idea in general (though unrelated to packaging) as monorepos do not work well with git.

~If using this method, It's need to deploy deb packages to another repository for distribution.~ I noticed sources.repos resolve this.

Not sure I got you here but it is also possible to pass extra apt repositories to sbuild via SBUILD_CONF.

isamu-takagi commented 1 year ago

If versioning by tag, the autoware.universe repository should be divided into multiple small repositories. This allows the repository/package maintainers to set the version at their release timing.

What do you think about this? @mitsudome-r @yukkysaito @kenji-miyake @xmfcx

esteve commented 1 year ago

Fine with me. I'm also open to a video call if you are interested.

We can do that, @isamu-takagi is based in Japan and I'm in Spain, I'll send you an email.

yukkysaito commented 1 year ago

What do you think about this? @mitsudome-r @yukkysaito @kenji-miyake @xmfcx

@isamu-takagi I am thinking now... On the negative side, some packages have dependencies between packages, and it is a pain to manage PR across packages when modifying functionality.

Are you considering a repository split for each package? Or per component?

Or instead of splitting the repository and releasing them separately, how about deb-package and release each one at the time of semantic version release on autoware?

kenji-miyake commented 1 year ago

If versioning by tag, the autoware.universe repository should be divided into multiple small repositories. This allows the repository/package maintainers to set the version at their release timing.

What do you think about this? @mitsudome-r @yukkysaito @kenji-miyake @xmfcx

@isamu-takagi I think flexible ways are better ways. I mean, reading the version in package.xml is suitable at least for Autoware as of today. If someone want to use Git tags, then they can update the version in package.xml when they create a tag.

jspricke commented 1 year ago

@isamu-takagi I think flexible ways are better ways. I mean, reading the version in package.xml is suitable at least for Autoware as of today. If someone want to use Git tags, then they can update the version in package.xml when they create a tag.

Note that the package.xml version does not accommodate changes in git so new commits will have the same version number. That's why we use git describe --tags.

isamu-takagi commented 1 year ago

Note that the package.xml version does not accommodate changes in git so new commits will have the same version number. That's why we use git describe --tags.

What is the difference between tag and package.xml? New commits will have the same version unless a new tag is created. I think it can be replaced by just updating the package.xml instead of creating the tag. Did I miss something?

jspricke commented 1 year ago

Note that the package.xml version does not accommodate changes in git so new commits will have the same version number. That's why we use git describe --tags.

What is the difference between tag and package.xml? New commits will have the same version unless a new tag is created. I think it can be replaced by just updating the package.xml instead of creating the tag. Did I miss something?

Using git describe --tags will generate a version number specific to the used commit. It has the form <tag>-<number of commits on top>-g<sha1 of commit>. This makes sure that new commits have a higher version number (due to the counter) and that one can identify the commit belonging to the package (by the sha1).

jspricke commented 1 year ago

Note that this is mostly relevant when creating packages for non tagged versions such as nightly builds. For tagged release builds git describe --tags will resolve to the tag so it should be the same as what is in the package.xml

isamu-takagi commented 1 year ago

I understood. At least, if it has a build time suffix as debian version, either tag or package.xml is fine. But using git describe --tags will give a commit hash.

isamu-takagi commented 1 year ago

So my understanding is that there is not much difference there. If using the package.xml version, It will be replaced like this: $(package.xml version)-$(time)-$(rev-parse HEAD).

jspricke commented 1 year ago

So my understanding is that there is not much difference there. If using the package.xml version, It will be replaced like this: $(package.xml version)-$(time)-$(rev-parse HEAD).

That could result in updating to older commits if they are rebuild as it is missing the number of commits on top. Also we currently don't add the hash at the end.

isamu-takagi commented 1 year ago

It might be better to consider nightly build and release build separately. I'll think it again.

isamu-takagi commented 1 year ago

That could result in updating to older commits if they are rebuild as it is missing the number of commits on top. Also we currently don't add the hash at the end.

That’s right. It is the advantage of git describe.

isamu-takagi commented 1 year ago

In conclusion, I found that there is no problem with version management by tags for a while.

After thinking about it, using tags is fine with for latest releases and nightly builds. For these, simply build the latest main branch or a specific tag. It's a bit unnatural for the version to be updated even if the package hasn't changed, but it doesn't matter.

My concern is for stable release. The reason I want to avoid using tags is that the Autoware repository is too large. Since the version of the repository is reflected in all packages, it is not possible to control the release timing for each package. This will the package is released with the latest code even if the package is not stable and maintainer want to provide the previous version as stable.

However, there is no problem with tag version at this stage. This is because the current Autoware is under development and it is difficult to provide a stable version. So as a first step, creating a tag for the entire autoware.universe and providing all packages with the same version of that tag is a reasonable choice.

In the future we need to support individual releases of packages. At this time, the version in package.xml is required, or split the repository. As @yukkysaito said, splitting the repository has the disadvantage of managing PR, but I hope that the code will be stable and easy to handle at this time.

kenji-miyake commented 1 year ago

@jspricke @isamu-takagi Thank you for the additional information and summaries.

There was one thing I misunderstood. When we say "use tags", I think there are two meanings:

Update packages only when a new tag is released.
Use tag-based information (such as git describe --tags) to determine the package versions, but they are built regularly.

I imagined the No. 1, but it seems the important point is mainly the No. 2, right? In that case, I think it'd be okay for now.

In the future, perhaps we can consider other options like "updating packages only when package.xml is updated, and keeping all the versions (not only the latest) of deb packages.", as I mentioned above. But I know there might be a problem then; the repository size would be extremely large.

Please correct me if I'm saying something wrong.

kenji-miyake commented 1 year ago

Regarding stable releases, I think we can resolve it by preparing multiple repositories (channels). For example, prepare the following 3 channels/repositories:

Stable/autoware-deb-packages-stable
Beta (Latest release)/autoware-deb-packages-beta
Nightly/autoware-deb-packages-nightly

It's very similar to Rust's way. https://doc.rust-lang.org/book/appendix-07-nightly-rust.html

And set the different timings of updating the packages.

Stable is updated when a maintainer manually updated the tag to use. It's assumed to be done after passing some tests such as scenario tests.
Beta is updated when a new tag is pushed.
Nightly is updated daily.

Then, users select the repository to use by changing the URL of add-apt-repository. (Or we can provide some commands to switch the channels.)

What do you think about it?

kenji-miyake commented 1 year ago

That could result in updating to older commits if they are rebuild

@jspricke By the way, regarding the above part in https://github.com/autowarefoundation/autoware/issues/3222#issuecomment-1404688816, when will they be rebuilt with older commits? I'm not sure when it happens. Is that when the maintainer reruns the workflow? The question is just out of curiosity.

jspricke commented 1 year ago

@jspricke By the way, regarding the above part in #3222 (comment), when will they be rebuilt with older commits? I'm not sure when it happens. Is that when the maintainer reruns the workflow? The question is just out of curiosity.

For reruns, yes. For example when you rebuild everything do to an ABI change in an upstream library.

isamu-takagi commented 1 year ago

@esteve @jspricke @xmfcx @kenji-miyake @yukkysaito @mitsudome-r Thank you for the meeting. From what we talked about, it would be nice to provide a nightly build as a first step. So how about creating a new repository something like autoware-deb-packages-nightly or autoware-apt-repository-nightly?

esteve commented 1 year ago

@isamu-takagi

Thank you for the meeting. From what we talked about, it would be nice to provide a nightly build as a first step. So how about creating a new repository something like autoware-deb-packages-nightly or autoware-apt-repository-nightly?

That's one option, the other option is to keep them all in one repository and use branches for nightly, stable, etc. That's what the moveit2_packages repository does for different Ubuntu/ROS distros. We could store packages in separate nightly branches named after the date they were built (e.g. jammy-humble-nightly-20230201) and always have a tag that points to the latest nightly (e.g. jammy-humble-nightly-latest)

jspricke commented 1 year ago

Or just not set SQUASH_HISTORY and have a branch with all the version and the HEAD pointing to the latest ;).

esteve commented 1 year ago

Or just not set SQUASH_HISTORY and have a branch with all the version and the HEAD pointing to the latest ;).

I believe it's easier to address each set of nightly builds by branch or tag, otherwise it can get messy.

jspricke commented 1 year ago

I'm pretty sure it is the opposite but feel free.

esteve commented 1 year ago

A tag or a branch is more descriptive that a commit id, and each set of nightly builds will be separate from the others

jspricke commented 1 year ago

That's already the case with the current implementation.

esteve commented 1 year ago

In that case, that's good, is that with SQUASH_HISTORY set to false?

jspricke commented 1 year ago

which is the default, yes

isamu-takagi commented 1 year ago

That's one option, the other option is to keep them all in one repository and use branches for nightly, stable, etc.

This seems to be good because the settings can be collectively managed in one repository. So let's create autoware-deb-packages or autoware-apt-repository. What name is intuitive?

SQUASH_HISTORY

Depends on package size. It is useful to set it to false to create branches or tags at old commit. But if the data is large, it have to be set to true.

jspricke commented 1 year ago

SQUASH_HISTORY

Depends on package size. It is useful to set it to false to create branches or tags at old commit. But if the data is large, it have to be set to true.

From our experience it does not matter.

esteve commented 1 year ago

This seems to be good because the settings can be collectively managed in one repository. So let's create autoware-deb-packages or autoware-apt-repository. What name is intuitive?

I had autoware_packages in my repo, named after the moveit2_packages repo, but I like your options more. autoware-deb-packages seems slightly more descriptive to me, but both are fine.

isamu-takagi commented 1 year ago

@mitsudome-r Could you create a repository named autoware-deb-packages?

mitsudome-r commented 1 year ago

@esteve @isamu-takagi I created the repository. https://github.com/autowarefoundation/autoware-deb-packages

isamu-takagi commented 1 year ago

@mitsudome-r @esteve Thank you. I'm going to port my test code. It builds autoware_common.

esteve commented 1 year ago

@isamu-takagi thanks, I'm building all the packages locally with the latest changes merged in, it should hopefully build, will report with the results

esteve commented 1 year ago

@isamu-takagi build failed on GitHub (see https://github.com/esteve/autoware_packages/actions/runs/4056342022/jobs/6980712591), but it does not seem to be related to Autoware or ros-deb-builder-action. ndt_omp needs its dependencies fixed (https://github.com/tier4/ndt_omp/pull/20 is waiting to be reviewed and merged).

I left the build.sh script running on my laptop last night, but there was an unrelated error. I triggered the script again and hopefully it'll build this time.

esteve commented 1 year ago

@isamu-takagi I got to https://github.com/tier4/tamagawa_imu_driver locally, but the build failed because the latest tag available does not follow the Debian versioning convention. I've filed a ticket at https://github.com/jspricke/ros-deb-builder-action/issues/5

esteve commented 1 year ago

@knzo25 @yukke42 I've managed to build all the packages up to lidar_centerpoint, however one of its dependencies is python3-open3d-pip, which is only available via PIP. The current version on PyPI for open3d is 0.16, whereas the version available via a Debian package on Ubuntu Jammy is 0.14 Would the Debian package work as well? Otherwise we'd have to backport it from Ubuntu Lunar (which has 0.16)

yukke42 commented 1 year ago

@knzo25 @yukke42 I've managed to build all the packages up to lidar_centerpoint, however one of its dependencies is python3-open3d-pip, which is only available via PIP. The current version on PyPI for open3d is 0.16, whereas the version available via a Debian package on Ubuntu Jammy is 0.14 Would the Debian package work as well? Otherwise we'd have to backport it from Ubuntu Lunar (which has 0.16)

@esteve Sorry for the late reply. I have checked it works with both open3d ==0.16 and 0.14.1.

knzo25 commented 1 year ago

@esteve Could you paste the PR in question (when available) to avoid duplicate work? :pray:

https://github.com/autowarefoundation/autoware.universe/pull/2869

esteve commented 1 year ago

@esteve Could you paste the PR in question (when available) to avoid duplicate work? pray

autowarefoundation/autoware.universe#2869

I added a link to this PR to the description of this issue so that all the linked PRs are in the same place, otherwise they're linked randomly from the comments which makes it less readable.

autowarefoundation / autoware