Unite efforts?

[awesomebytes]

Hello. I've just discovered this repo. Thanks for linking to my projects and mentioning me as the author or the CI for Gentoo Prefix and for ros-overlay on Gentoo Prefix.

I see a lot of work duplicated in this repo and it makes me sad. Maintaining the pipeline of Gentoo Prefix > ROS base > ROS desktop (in 32b and 64b, and for Kinetic and Melodic, that's a lot of combinations!) takes continuous effort. It's usually not much work, but things change or break upstream all the time (such is Gentoo, a rolling release).

I would propose to create a version of this repo that starts from my work and then adds your extras as an additional layer. This would avoid that we both need to fix the same errors, or that you need to re-implemented the fixes I implement.

By what I see in your Dockerfile (just checked the Kinetic one). You are mainly building on top of ros_base.

You could just start your image doing docker pull from: awesomebytes/roogp_ros_kinetic_ros_base awesomebytes/roogp_ros_melodic_ros_base

Or follow from from one of the binary releases, if you prefer, as I do in https://github.com/awesomebytes/pepper_os (mainly to avoid the 127 max layers of Docker).

I would find great if you would provide ready-to-use, as I do, ROS Kinetic/Melodic on Nao & Pepper. It's a matter of setting up a job to package it. You can copy my approach of uploading it to GitHub releases of the repo for free storage on nightly builds. Or, if you have some server to use as a file host, just upload it without splitting (I just need to do it to keep it under 2GB release files). I say this because of experience. During the development months of RoboCup plenty of times I didn't have time to fix the upstream problems, and having pre-compiled versions being built every night allowed for the development to continue. Eventually I'd go back, report bugs upstream, fix stuff... and we are back on the latest SW, yay.

This way, you can focus on maintaining less things (probably just naoqi-* stuff). And when you have time to dedicate to fixes of the general pipeline, we all benefit from it.

There may be discrepancies on how to structure the Dockerfiles, the patches... in general how to work on all of this. I'm open to conversation. Up until now I've been working mostly alone in it, so I just did what seemed quicker and alright.

[awesomebytes]

I thought maybe the best would be to lead by example and I made an implementation on ros_kinetic_32 in a fork:

• Dockerfile to build over my ros_base: https://github.com/awesomebytes/sbre_robot_ros_gentoo_prefix/blob/ros_kinetic_32/ros_base/Dockerfile.from_roogp
• Separated Dockerfile for the packaging step: https://github.com/awesomebytes/sbre_robot_ros_gentoo_prefix/blob/ros_kinetic_32/ros_base/Dockerfile.package_from_roogp
• Example Azure Pipelines setup to build binary releases: https://github.com/awesomebytes/sbre_robot_ros_gentoo_prefix/blob/ros_kinetic_32/azure-pipelines.yml

Edit: maybe easier to visualize in PR form: https://github.com/softbankrobotics-research/sbre_robot_ros_gentoo_prefix/pull/5

In order to adopt this for yourselves you'd just need to change the DockerHub account to yours and register in Azure Pipelines and integrate your repo in it (adding your secrets of DockerHub password and GitHub Token) as I've written in comments in the PR.

Ping me if you need any help!

[mcaniot]

Agreed ! I will take a look on the PR and certainly based on your ros_base to get rid of the Gentoo dependency.

[awesomebytes]

In between writing my PhD thesis breaks I kept running builds until they worked.

Find the following PRs:

• ROS Kinetic: https://github.com/softbankrobotics-research/sbre_robot_ros_gentoo_prefix/pull/6
• ROS Kinetic 32b: https://github.com/softbankrobotics-research/sbre_robot_ros_gentoo_prefix/pull/7
• ROS Melodic: https://github.com/softbankrobotics-research/sbre_robot_ros_gentoo_prefix/pull/8
• ROS Melodic 32b: If you create a new branch called 'ros_melodic_32' I'll PR my branch that supports that too. Maybe someone wants to use that.

As a proof of working, you can find at the releases tab of my fork ready-to-use built images: https://github.com/awesomebytes/sbre_robot_ros_gentoo_prefix/releases They take 1h30-2h~ to build: builds page

Important bits we may want to discuss about:

• All these branches share in common that they build from my 'desktop' image instead of 'ros_base'. That is, as explained in the PRs, because in that image big (and slow to compile) dependencies like PyQt are already built minimizing the time to build this image and the amount of maintenance to do.
• naoqi_libqi, naoqi_libqicore and naoqi_driver all have workarounds to build in every version. I think the best that could be done is that you make a PR to ros-overlay so these are not needed. I'm sure @allenh1 would be very happy about. That's the quick solution. The awesome solution would be to modify these packages so they can be built both in Debian-like systems and Gentoo-like systems. I understand this may be hard to prioritize, but we can all dream.

Future work?

• If provided a server in the cloud, we could host pre-compiled binaries of all packages and serve just a tiny base image. Just like apt-get installing. I've tried to get some sponsor to provide that, but no luck for now.
• I'm actually thinking on re-basing my pepper_os image from this one... Or you may want to officially support this by providing 'perception', 'navigation' and maybe other useful ROS packages?

[allenh1]

I think the best that could be done is that you make a PR to ros-overlay so these are not needed. I'm sure @allenh1 would be very happy about.

:+1:

[mcaniot]

You're right about the "ready to use" versions of the work, we'll provide that with upcoming releases. To get rid of the constant refactoring, we decided to freeze the gentoo prefix pipeline, and build on top of your pre built images:

• awesomebytes/roogp_ros_kinetic_ros_base:1464

• awesomebytes/roogp_32b_ros_kinetic_ros_base:1465

• awesomebytes/roogp_ros_melodic_ros_base:1466

• awesomebytes/roogp_32b_ros_melodic_ros_base:1467

We might update the tag version at some point, but for now freezing it will allow us to focus on the upper ROS layers. Because of our requirements, we also want to build on top of 'ros_base' instead of desktop, so we'll keep it that way.

[awesomebytes]

Fair enough.

Just be warned that by fixing the tags you may run into issues in some time as dependencies get outdated. I've run into this issue in the past. And it is happening more often with python dependencies in upstream gentoo dropping python 2 support.

I would recommend a hybrid approach, when you setup CI for this, where you have a job with fixed tag version and a :latest tag version. Doing so you could switch the fixed tag to whatever is the latest that hopefully still builds easily.

I'd be curious to know what your requirements are, if it can be shared.

P.S.: here I am pretty much off-topic-ing, but, are you planning to support Python 3 on the Python SDK (naoqi)?

On Mon, Mar 30, 2020, 23:15 Maxime Caniot notifications@github.com wrote:

You're right about the "ready to use" versions of the work, we'll provide that with upcoming releases. To get rid of the constant refactoring, we decided to freeze the gentoo prefix pipeline, and build on top of your pre built images:

-

awesomebytes/roogp_ros_kinetic_ros_base:1464

awesomebytes/roogp_32b_ros_kinetic_ros_base:1465

awesomebytes/roogp_ros_melodic_ros_base:1466

awesomebytes/roogp_32b_ros_melodic_ros_base:1467

We might update the tag version at some point, but for now freezing it will allow us to focus on the upper ROS layers. Because of our requirements, we also want to build on top of 'ros_base' instead of desktop, so we'll keep it that way.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/softbankrobotics-research/sbre_robot_ros_gentoo_prefix/issues/4#issuecomment-605963524, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANEK5HQ53KTA2WQUMYURVLRKCEMVANCNFSM4LEVMPSQ .