[question] diff/relation with Gentoo tree

[archenroot]

Hi guys, thanks for this work.

What is the difference and/or relationship between this overlay and standard Gentoo tree packages?

It is not clear to me as some of ebuilds are in different groups (dev-python vs dev-utils, etc.)...

I am not sure which one is better (I understand depends on expectations) to use?

[allenh1]

@archenroot thank you for the question.

What is the difference and/or relationship between this overlay and standard Gentoo tree packages?

The difference is that this overlay is maintained by the Open Source Robotics Foundation, and will, after I have worked out all the little bugs, be synched on a nightly/weekly (have not decided this yet) basis on the ROS buildfarm.

It is not clear to me as some of ebuilds are in different groups (dev-python vs dev-utils, etc.)...

For a LONG time we have had our packages under the dev-python category, and I just don't see a huge reason to change them. They will likely be removed from this overlay entirely in the near future (since they are now in two places: there's a catkin under dev-python/catkin and one under ros-lunar/catkin, for example, and they should not be used simultaneously).

I am not sure which one is better (I understand depends on expectations) to use?

I suppose I'd say that depends on how quickly you are in need of it. We're going to be supporting all non-EOL distros here (jade is soon to be EOL, so I've got builds for kinetic, lunar, and indigo as of now).

The goal here is simple: make Gentoo installations just as easy as Ubuntu installations. You can almost do the following: sudo emerge ros-lunar/desktop_full.

This differentiates from the Gentoo tree in that you can also do ros-lunar/p2os_driver, for example, or whatever package you need (if it's in the rosdistro release yaml, then it's in our repo, unless it lacks dependencies).

The current status is not exactly stable, but I've been on this full time for the past couple weeks at OSRF: allenh1/ebuild-generator.

I'm super close to having things working well. Current builds don't work because of some python problems, but I think I'll have those resolved shortly. I'll post back here when I'm able to install desktop_full without error, should you be willing to test on your end.

[archenroot]

Thanks for detailed answer.

So you don't cooperated with Gentoo ROS Project team?

For example in gentoo tree we have:

ares ~ # emerge --search desktop_full

[ Results for search key : desktop_full ]
Searching...

*  ros-meta/desktop_full [ Masked ]
      Latest version available: 9999
      Latest version installed: [ Not Installed ]
      Size of files: 0 KiB
      Homepage:      http://wiki.ros.org/desktop_full
      Description:   Metapackage for ROS complete desktop install
      License:       BSD

[ Applications found : 1 ]

ares ~ # equery meta desktop_full
 * ros-meta/desktop_full [gentoo]
Maintainer:  ros@gentoo.org (Gentoo ROS Project)
Upstream:    Remote-ID:   ros/metapackages ID: github
Homepage:    http://wiki.ros.org/desktop_full
Location:    /usr/portage/ros-meta/desktop_full
Keywords:    1.2.0:0: 
Keywords:    1.3.0:0: 
Keywords:    1.3.1:0: ~amd64
Keywords:    9999:0: 
License:     BSD

I just wonder why not synchronize the work with Gentoo ROS project team: https://wiki.gentoo.org/wiki/Project:ROS

If we talk about simple packages, then I don't care to go the way which every one would like to go, but Gentoo created a specific project for this to handle, so why having 2 large works not synced? I also checked the ebuild QA (just addhoc look at same ebuilds those ebuilds from default Gentoo tree looks slightly better)?

Thanks for your effort, I just started learning ROS for some SLAM related stuff.

I think it shouldn't be so hard to support in Gentoo different distributions either via group ros-lunar, or either via USE flag similar like in case of PYTHON => pythontargets[lunar|kinetic|etc...] so everybody is happy in the end. Do I understand you like to support multiple distributions? I think the effort is great, but sync with Gentoo ROS project would be more than welcomed.

[allenh1]

We're not synched with the Gentoo people for a number of reasons, but the most trivial reason is the difference in our priorities. The most important difference, to me, is the versioning.

For example, the Gentoo repo has two Gazebo ebuilds: 8.0.0 and 8.1.0. While up to date, these are not yet supported in ROS. The latest release (which was released less than a month ago) is Lunar, which requires Gazebo 7 (at least at the time of writing).

Other small things: There are different versions of packages that are associated to each release. Stage, for example, uses version 4.1.1-1 in Kinetic, but version 4.3.0 in Lunar.

I can think of a few reasons why you'd like to use this repo:

• These installers are not distro agnostic (there's a proven amount of synergy withihn each ROS release, and we don't like to mix them).
• These installers will work like their Ubuntu package equivalents, installing an isolated meta-os in the /opt/ros/distro directory, even isolating individual distributions (should you, like me, be a crazy person that has several ROS versions on their system).
• There are < 160 ebuilds for the Gentoo ROS repo, whereas there are slightly over 4000 ebuilds for this repo (which is a far better representation of ROS -- it definitely deserves to be called an OS).

I also checked the ebuild QA (just addhoc look at same ebuilds those ebuilds from default Gentoo tree looks slightly better)?

Please feel free to file issues! I am not very knowledeable about ebuilds, and I don't know the conventions super well. Just note that the dev-python/* packages wiill soon be removed entirely.

Thanks for your effort, I just started learning ROS for some SLAM related stuff.

Absolutely! And welcome! We're really fortunate to have a really postive and driven community. Please feel free to reach out for help on answers.ros.org, should you need it!

I think it shouldn't be so hard to support in Gentoo different distributions either via group ros-lunar, or either via USE flag similar like in case of PYTHON => pythontargets[lunar|kinetic|etc...] so everybody is happy in the end. Do I understand you like to support multiple distributions? I think the effort is great, but sync with Gentoo ROS project would be more than welcomed.

I'll keep that in mind. For now I'm just trying to get it working consistently. After all is well, I'll ping the Gentoo people.

[archenroot]

@allenh1 Regarding QA I take it back, not relevant and I didn't use REPOMAN, and is anyway not my primary intention here to discuss QA - I just checked 9999 vs regular packages detection and EAPI version, not important in the moment :-)

To me you did great stuff and I like better the idea of having multiple isolated ROS installation at the same time. I am also usually going crazy (in positive meaning) :-)

It might be good to implement later ESELECT which could give you eselect ros list: lunar kinetic etc..

So with this in mind it could be also then handful to use USE_EXPAND variable with something like ROS_TARGETS="lunar|kinetic|etc."

I will definitely open discussion with Gentoo ROS people about merging with this overlay from long term perspective and especially taking into account support of multiple ROS distributions at same time, it is must from my perspective!

Thanks again!

[allenh1]

It might be good to implement later ESELECT which could give you eselect ros list: lunar kinetic etc..

That's a really, really good idea. I'll definitely look into it!

I will definitely open discussion with Gentoo ROS people about merging with this overlay from long term perspective and especially taking into account support of multiple ROS distributions at same time, it is must from my perspective!

Alright, excellent! I'll keep working in the meantime!

Thanks again!

Of course!

Also, if you get a few, it would be super awesome if you could try and emerge ros-lunar/desktop_full from this repo. I'm super close on my system, but I think it's only due to my processor flags.

[archenroot]

Good stuff!

I will try to emerge the package today.

[allenh1]

Thank you very much! Should one fail, please file an issue. In the meantime, is it alright if we close this one?

[archenroot]

I am putting here copy of email with Alexis - maintainer of ROS project at Gentoo - one man show in the moment looks like :-)

Hi Ladislav!

On Thu, 8 Jun 2017 09:38:53 +0200 Ladislav Jech archenroot@gmail.com wrote:

Hi Alexis,

my name is Ladislav Jech, I maintain some unofficial overlay for neural networks and GPU stuff.

I am contacting you because of ROS overlay for Gentoo and idea to merge/sync this overlay and Gentoo portage into one.

Nice work there. When I started all the ROS stuff in Gentoo this overlay was mostly dormant and only contained the basic stuff to later do a manual install of ROS packages you need. It seems to be much more useful these days.

I opened discussion here: https://github.com/ros/ros-overlay/issues/71

1. I think the approach to install multiple ROS distribution at same time makes sense.

It might make sense indeed. The main issue for me is the manpower: ROS in gentoo is a one man show and I can barely keep what's there in good shape. If people start to contribute and want to move in that direction, I definitely see that as a good thing. More on it below.

2. We could come with something like ROS_TARGETS USE_EXPAND variable with values: lunar|kinetic|etc.

Yep. I don't really like what's done in the overlay to have one category for each ROS release. After all, they're mostly different versions of the same package.

We could keep the dev-ros category and do something like what is done for python. ebuilds could have ROS_COMPAT="kinetic lunar" saying they can be installed for either kinetic or lunar or both (I'm not 100% sure there but I think there are a lot of cases where the same version of the same package is shipped with different ROS releases).

Then users could set ROS_TARGETS="kinetic" to install the kinetic version. Just like when you do PYTHON_TARGETS="python3_4" you get only the python 3.4 version of python packages.

This raises the question of where to install ROS packages. I've been with installing into /usr because:

1. Distros usually install stuff there. It's not a strong requirement but it's better.
2. Installing in standard paths avoids having to source some environment file in every shell (like what you have to do when doing a manual ros install). This can be really improved by an eselect module as you suggest on GH, I had not thought about it before and it seems a good idea.

That last point was the one really annoying me.

However, if you install in /usr then you can't have 2 versions of the same package (say roscpp-X for kinetic and roscpp-Y for lunar) as they would collide. I've realized over time that installing into /usr also requires some patching since OSRF assumes it'll always be installed in a somewhat sandboxed /opt. So, in the end, I think it is a very good idea to install ROS into /opt/ros/${TARGET} (like done in the overlay) with an eselect module setting up globally the variables so that currently selected ros target is the one running by default.

There's also the python issue: We'd have to fix a python version for each ROS release. I think that's what OSRF does and is a good thing: who needs two pythons for running ROS as long as one works? It's minor but we have to keep that in mind.

3. Maintainer of ROS overlay is ROS guy who install ROS farm, so he has very good knowledge about package dependencies, so good help
4. they have massive ebuild packages ready

Yep, I noticed. There'd be some changes to do for this to be properly integrated into Gentoo, like the ebuild generator seems to duplicate a lot of code instead of using the eclass, but I imagine the eclass can't be used for now because of the above required changes.

I think even Gentoo tree and ROS overlay go little bit different way, Gentoo offers via ebuild subsystem multiple ways how to sync those 2 into one where all are happy. There'd be some changes to do for this to be properly integrated into Gentoo, like the ebuild generator seems to duplicate a lot of code instead of using the eclass, but I imagine the eclass can't be used for now because of the above required changes.

I think even Gentoo tree and ROS overlay go little bit different way, Gentoo offers via ebuild subsystem multiple ways how to sync those 2 into one where all are happy.

Yep, but don't assume anything on the directions I've taken for ROS in Gentoo: Those are mostly dictated by how much time I'm able to salvage for getting this to work as I need it. I can assure you that if you start contributing (e.g. via github pull requests) those directions will likely change to yours ;)

I'd really love to see the ros overlay to be a staging ground before being integrated in the main portage tree, but that mostly depends on you guys.

Bests,

Alexis.

So that looks like quite positive feedback!.

That was also my idea, that this overlay could become staging area for bleeding edge packages access and QA for Gentoo tree.

I generally agree on all points you suggested as they follow my own. To summarize it looks like good idea to go with:

1. new eclass for ROS to prevent duplicated code and some sanity management for ebuilds
2. install into /opt/ros/{TARGET_OS} to isolate multiple
3. support eselect ros (list | set [n]) so user can pickup in shell one for specific use
4. create new USE_EXPAND variable -> ROS_COMPAT with distro specification. This could be per package/global configurable as Gentoo supports. So for example Global level is "kinect lunar", and on package level it could be like:

• package-1.0.ebuild - supports only kinect
• package-2.0.ebuild - support both kinect and lunar
• package-3.0.ebuild - support only lunar

5. Q: all in overlay bleeding edge will be ~amd64, I am not sure when to switch to stable, but this could be decided by Gentoo package staging process...?
6. Q: I am not sure if we maintain it in the Gentoo tree or we don't move it into Science project, but in general I don't care much?
7. this overlay will in fact become staging area for Gentoo tree - I am not sure about dependency maintenance ::gentoo vs ::ros-overlay, but there definitely some best practices to lower the maintenance need.

So Alen, what do you think about this?

[aballier]

new eclass for ROS to prevent duplicated code and some sanity management for ebuilds

There is already ros-catkin.eclass used by all ROS packages in the gentoo tree. That's what should be improved to handle the cases mentioned here I think.

Q: all in overlay bleeding edge will be ~amd64, I am not sure when to switch to stable, but this could be decided by Gentoo package staging process...?

Here we have our own process: https://wiki.gentoo.org/wiki/Stable_request

Q: I am not sure if we maintain it in the Gentoo tree or we don't move it into Science project, but in general I don't care much?

I wouldn't move it to science overlay; keep your changes here for overlaying, use what can be used from gentoo tree, merge to gentoo tree what can be merged

this overlay will in fact become staging area for Gentoo tree - I am not sure about dependency maintenance ::gentoo vs ::ros-overlay, but there definitely some best practices to lower the maintenance need.

make your changes here, send a PR to the gentoo tree when you're happy with it

in the long term we could talk about getting you write access to the gentoo repo but that's a much longer process than filing a github PR

[allenh1]

@aballier Thank you very much! I'll work on cleaning up the generated ebuilds in the meantime.

You can track that progress on allenh1/ebuild-generator (which, in hindsight, I should probably have name "ros-ebulid-generator"). If you have any specific issues/PR's for simple fixes with the way I am generating the ebuilds, it would be extremely helpful for me if you could file them there.

[Alessandro-Barbieri]

@archenroot @allenh1 @aballier AWESOME!

[Alessandro-Barbieri]

For long I was thinking for a solution to the overly increasing number of ebuilds (some of them duplicates across categories) and my first thought was to merge all the categories into one and use slots to accomodate ROS releases. Can be a working solutions or it's too complicated? What about packages that uses the same version across multiple releases and need to be installed in multiple slots?

@aballier Are you OK to adding the missing ROS dependencies downstream? For example gazebo-7 and other packages that we will figure out soon. See https://github.com/ros/rosdistro/issues/15082 for a list

[aballier]

For long I was thinking for a solution to the overly increasing number of ebuilds (some of them duplicates across categories) and my first thought was to merge all the categories into one and use slots to accomodate ROS releases. Can be a working solutions or it's too complicated? What about packages that uses the same version across multiple releases and need to be installed in multiple slots?

This is part of my initial reply :)

@aballier Are you OK to adding the missing ROS dependencies downstream? For example gazebo-7 and other packages that we will figure out soon. See ros/rosdistro#15082 for a list

Feel free to file github PR to the gentoo tree and mention me there. That'd be too optimistic for me to maintain them all, but proxy maintaining is definitely a way to get this done.

As for gazebo-7 specifically, why not, but it should be slotted and not EOLed upstream. It's better to avoid forcing people to downgrade because some ros packages happen to prefer gazebo-7.

[Alessandro-Barbieri]

There are < 160 ebuilds for the Gentoo ROS repo, whereas there are slightly over 4000 ebuilds for this repo (which is a far better representation of ROS -- it definitely deserves to be called an OS).

Crazy idea: use https://github.com/jauhien/g-sorcery

[allenh1]

@Alessandro-Barbieri that system has deprecated features like 'herd' in it, if you look closely (last updated 2 years ago as well).

I wrote a significant amount of code to get mine to work, and it definitely works for all my purposes (including automatically filling PR's when generation finishes).

[Alessandro-Barbieri]

@Alessandro-Barbieri that system has deprecated features like 'herd' in it, if you look closely (last updated 2 years ago as well).

I see, but that is the default schema, you can provide your xml schema

I wrote a significant amount of code to get mine to work, and it definitely works for all my purposes (including automatically filling PR's when generation finishes).

As far as I have read, gsorcery relies on user written code in order to generate ebuilds, eclasses and metadata.

[allenh1]

That's fine, but I don't see anything it does that mine doesn't. It's just a different API.

[Alessandro-Barbieri]

new eclass for ROS to prevent duplicated code and some sanity management for ebuilds
install into /opt/ros/{TARGET_OS} to isolate multiple
support eselect ros (list | set [n]) so user can pickup in shell one for specific use
create new USE_EXPAND variable -> ROS_COMPAT with distro specification. This could be per package/global configurable as Gentoo supports. So for example Global level is "kinect lunar", and on package level it could be like:

package-1.0.ebuild - supports only kinect
package-2.0.ebuild - support both kinect and lunar
package-3.0.ebuild - support only lunar

We need someone familiar with this kind of eclasses, after a look at the python eclasses I got lost ...

[archenroot]

@Alessandro-Barbieri :D me too :D

@aballier - Are you willing/capable of introducing the eclass which might support what we discussed here? We have theoretical concept now, but to be honest I am also not capable in writing it by myself (or it could take mich longer time as I never did), I am of course reading gentoo wiki and looking into ros-catking.eclass you created But not sure of what where and how should be done in proper way. Thanks for your feedback.

Question: What is also not clear is mapping of ros distro to python versions. @allenh1 - does ROS define strict compatibility on distro level or per package? If on distro level then it is more easy to manage, while making mapping like: kinect - python2_7,python3_4 lunar - python3_4,python3_5, etc...

Maybe I see just a ghost, but this could be a possible issue. Or does it make sense to create such mapping and maintain it? I think so... but not sure.

@allenh1 - with gsorcery it was just idea to consider, I understand both will do same job and you already implemented your own framework/tool, so maybe wasted time to migrate to generic ebuild generator tool.

[aballier]

@aballier - Are you willing/capable of introducing the eclass which might support what we discussed here?

Yes I can but I definitely need help here.

What is also not clear is mapping of ros distro to python versions. @allenh1 - does ROS define strict compatibility on distro level or per package? If on distro level then it is more easy to manage, while making mapping like: kinect - python2_7,python3_4 lunar - python3_4,python3_5, etc...

Yes that's what I was thinking about: For each ROS release, define one python version to be used. Don't even worry about multi-python for now. While multi-python is useful, esp. for testing, I don't think it brings any benefit to ROS and it doesn't work well with ROS anyway (rospack for example does not support it properly).

@allenh1 - with gsorcery it was just idea to consider, I understand both will do same job and you already implemented your own framework/tool, so maybe wasted time to migrate to generic ebuild generator tool.

I would say don't bother with g-sorcery. While it is a noble idea, 95% of the work of writing an ebuild generator is not about writing the ebuild but parsing the proper data from upstream files and having specific features like auto-bump notification or filling PRs. I don't even use it myself when I need to generate some ebuilds.

[archenroot]

Yes I can but I definitely need help here. I think we all are willing to help, so need lets say navigation

Yes that's what I was thinking about: For each ROS release, define one python version to be used. Don't even worry about multi-python for now. While multi-python is useful, esp. for testing, I don't think it brings any benefit to ROS and it doesn't work well with ROS anyway (rospack for example does not support it properly).

This is something @allenh1 could decide as well. To me multi-python is nice. Maybe eclass can define something like stable python mapping record and experimental one. kinect -> python2_7(stable), python3_4(experimental|test|etc.)

I would say don't bother with g-sorcery. While it is a noble idea, 95% of the work of writing an ebuild generator is not about writing the ebuild but parsing the proper data from upstream files and having specific features like auto-bump notification or filling PRs. I don't even use it myself when I need to generate some ebuilds.

I agree, lets forget it... this just naturally came to mind...

[archenroot]

@allenh1 If you in general agree on the way how to handle ros within Gentoo as per above discussion, maybe it could be good time to create new branch grant us access there, I mean @aballier @Alessandro-Barbieri and me initially, so we can have place for new Gentoo ROS integration, or do you prefer different way?

[allenh1]

I'd prefer to give you access to allenh1/ebuild-generator for now, since that's what gets merged into this repo.

I don't mind the discussion above at all. That all sounds super good.

[tfoote]

Hi @archenroot @aballier @Alessandro-Barbieri and @allenh1 thanks for pushing on this. It's great to see that there's interest in forging a path to submitting upstream. I hope with the automation we can keep the individual effort level down for ongoing maintenance for the greater ecosystem on Gentoo.

[allenh1]

Closing for now, just doing some cleanup.