up-to-date status page

[fmauch]

I've posted this over at melodic already, but of course this also makes sense for noetic.

Since switching to archlinux I find it to be a bit tedious to check which ROS packages are available and whether those are up-to-date in the AUR.

Is there some status overview / scripting for this? Otherwise I might be willing to create a scripting tool for this. The idea would basically be to compare the ros-melodic-*-packages in AUR with the one listed in the rosdistro.

I opened the issue here, as this seems to be the most generic place.

[acxz]

That definitely is a good idea, and actually some people have some work on this. Friendly ping to @jwhendy Also check out @bionade24 's aur-ci scripts. There is one that goes through and updates all the packages, (although it can cause errors in my experience)

[bionade24]

Also check out @bionade24 's aur-ci scripts. There is one that goes through and updates all the packages, (although it can cause errors in my experience)

Yeah, it currently only works if the ros tarball is the first source. But afaik we don't have any other cases. @acxz What was your actual problem? Could you maybe then open an issue?

[jwhendy]

Unfortunately I have not done this comparison (edit: between here and AUR. I have done some hackish comparisons between our versions and those in the ROS index). Shouldn't this be as simple as comparing package versions?

I still don't understand what the consistent hold up is in updating. @bionade24 is this a function of you running your server manually? Or per-package? My proposal might be:

• pursue some way to guarantee updates per time (e.g. nightly)
• switch away from depending on AUR at all (why couldn't we just clone/build from here?)
• ditch building from scratch entirely by moving to binaries (again, I'm not clear on the key limitations or hold ups there)

I find building from here pretty similar to Ubuntu adding custom repos. What does AUR give us unless it's updated nearly seamlessly? It seems more trouble than it's worth, at least on something that never even remotely gets "stable."

[bionade24]

I still don't understand what the consistent hold up is in updating. @bionade24 is this a function of you running your server manually? Or per-package? My proposal might be:

No, I always feared it would break more than we would benefit from new versions.

[fmauch]

• switch away from depending on AUR at all (why couldn't we just clone/build from here?)

I think the point is that people might use regular AUR helpers to install their ROS system.

[jwhendy]

bonade24: No, I always feared it would break more than we would benefit from new versions.

I admit I've been ignoring much of the recent activity since there's been so much. I assumed the strategy was that AUR would mirror this org. Is that not the case?

fmauch: I think the point is that people might use regular AUR helpers to install their ROS system.

Indeed, but the point is they are up to date here, but not on AUR. I was thinking out loud if AUR helpers could point here instead. It seems our hurdle is getting them synced to AUR. My first bullet was figuring out why that doesn't happen (and @bionade24 indicates it's intentional?). As a backup if this is not possible, I'd rather just accept that we can never trust that AUR is up to date and figure out how to use this org instead.

What good is it to pretend AUR is reliable if it's not? So, either we get it reliable or we abandon and just get people working with this github org.

[bionade24]

I think the point is that people might use regular AUR helpers to install their ROS system.

Then people could use arch4edu or (if melodic) my repo instead. Both are same outdated as AUR. And theoretically you could use https://github.com/bionade24/ros-aur-helpers as a package manager for @jwhendy 's idea

[acxz]

Yeah, it currently only works if the ros tarball is the first source. But afaik we don't have any other cases. @acxz What was your actual problem? Could you maybe then open an issue?

Ah sorry the problem that I had wasn't something I experienced myself (disc. have never used those update scripts myself), but basically when you pushed out several updates to the ros-melodic packages after the sync to noetic. Some of the _dir variables were rewritten causing package builds to break. You can see the PRs made by @fmauch that reverted many of those changes. I think the packages affected were moveit-* and some other low level utilities like rosbag.

[bionade24]

I admit I've been ignoring much of the recent activity since there's been so much. I assumed the strategy was that AUR would mirror this org. Is that not the case?

I think I misunderstood you. About Mirroring: My hacky CI failed and that's why it didn't worked very often, which leaded to non-updated packages. I originally thought you ment automatically updating PKGBUILD files and committing changes somehow like depend-a-bot.

[acxz]

As for noetic, I can assure you that all packages on the github here are in sync with the packages in the AUR.

AUR provides visibility. PKGBUILDs in AUR when popular enough move the the official repos. PKGBUILDs in AUR can be used by unofficial user repositories to make binaries.

I don't see a reason not to use the AUR, because we are having syncing problems. We shouldn't abandon AUR when the problem is something else. We should fix our syncing problem. Which I think for noetic will hopefully be good since I can spend more time on it and I'm not using CI. As for syncing/updating ros-arch packages with the official ros-noetic packages that still needs to be handled. But due to the difficulties with searching for the right package release version from ROS1 and ROS2 versions it makes it hard for noetic. When we get around to the ROS2 Rolling Distro (https://discourse.ros.org/t/ros-2-rolling-distribution-plans/13227) we can then depend on the release versions and it will be much cleaner to implement.

My hacky CI failed and that's why it didn't worked very often, which leaded to non-updated packages.

For the noetic packages I am not CI'ing at all and just syncing once it builds on my machine. It will be a bit more unstable, but arch4edu will be provided binaries for the stable equivalent solution. I still think your hacky CI was pretty amazing and a really good idea. However, the use of an AUR helper is nice even when debugging issues which is why I prefer to just sync things. (dsic. haven't looked at your other ros-aur-helper repo)

[bionade24]

CI should now work again, I had to comment the pull function out once. I still believe we should do some CI testing for noetic, probably with drone. If no one else cares I'll care about it. We then still can sync anyway directly. Or what about using GH actions ( we wouldn't have packages then) ?

[acxz]

Is there some status overview / scripting for this? Otherwise I might be willing to create a scripting tool for this. The idea would basically be to compare the ros-melodic-*-packages in AUR with the one listed in the rosdistro.

Coming back to the main conversation at hand. @fmauch if you could create a tool that compares the ros-noetic package versions on the AUR with the rosdistro package versions and it outputs a list of outdated AUR packages, that itself would be pretty beneficial. From there we can put up a list of these outdated packages on an issue thread and the community can help update them. And/Or we can come up with an automated solution that updates the PKGBUILDs of the affected packages. Let's start small/split this problem up. I myself will not have the time to create such a script but based on the repos and scripts linked in this discussion you can probably scrape together something.

Just having a list of outdated packages is a pretty good start.

[acxz]

I still believe we should do some CI testing for noetic, probably with drone.

What is drone? Absolutely we should do CI, but the build logs should be visible to everyone (I know you where working on a web frontend, but time is always the limiting factor isnt it :) ) I think we can prob use github actions. We can also have the arch4edu repository inside of our CI to speed up the clean chrooting of the packages. It is totally possible. Also having arch4edu in the CI loop will ensure we are also syncing packages properly. Just some ideas, I myself will not have the time to investigate this further for the foreseeable future.

[fmauch]

Aweseome! I hacked something together very quickly last night, that grabs both versions for a package. Making this pretty and going over all released packages should be a doable task...

[bionade24]

I still believe we should do some CI testing for noetic, probably with drone.

What is drone? Absolutely we should do CI, but the build logs should be visible to everyone (I know you where working on a web frontend, but time is always the limiting factor isnt it :) ) I think we can prob use github actions. We can also have the arch4edu repository inside of our CI to speed up the clean chrooting of the packages. It is totally possible. Also having arch4edu in the CI loop will ensure we are also syncing packages properly. Just some ideas, I myself will not have the time to investigate this further for the foreseeable future.

https://drone.io/ This idea came to me during hacking the webinterface, so I stopped it. Can be self-hosted easily, but I don't know if this could work. Maybe something for noetic.

[acxz]

Aweseome! I hacked something together very quickly last night, that grabs both versions for a package. Making this pretty and going over all released packages should be a doable task...

dope! I'm not sure if the AUR has a nice API for getting package information, it probably does though since repology is able to scrape the packages off of it.

[fmauch]

I've used aurwebrpc with a small python wrapper for querying the package information. That works very nice indeed.

[jwhendy]

@bionade24

I originally thought you ment automatically updating PKGBUILD files and committing changes somehow like depend-a-bot.

Ahhh, that makes sense now. Indeed, I think trying to auto-update based on upstream would indeed be a challenge. I was purely commenting on once we're good here, how might we improve the auto-sync to AUR.

@acxz

We shouldn't abandon AUR when the problem is something else.

I agree. Actually, why not just use something like the simple bash script I originally made? I'd be happy to do that... namely, a script that:

• clones everything from here
• adds a new remote that points to aur/ros-*-foo
• fetch, merge, then push from local to AUR (not sure, but the errors you mentioned before might just be git spitting out information. For example, I often just did git checkout foo and the branch might already have been on foo, or pushing and git will say up to date)

Would that be reasonable? I think CI is great for checking that things can be merged here... but I don't see the need for CI when the purpose is to make sure that AUR mirrors here. [after catching up more above, I see you're basically already doing this so we're on the same page].

if you could create a tool that compares the ros-noetic package versions on the AUR with the rosdistro package versions and it outputs a list of outdated AUR packages

I might be able to dig up what I had, which was at least the second half of this. It was simple, but a bit non-trivial as you have to do some checks for ROS1 vs. ROS2 (often 0.x if 1.x exists, or 1.x if 2.x exists).

[jwhendy]

I found where Oskar said he used my script (lost in my btrfs ssd fail). It's not my original, but has some of the elements. It's where @acxz already pointed to.

[acxz]

[after catching up more above, I see you're basically already doing this so we're on the same page].

Thats p much exactly what I'm doing lol

often 0.x if 1.x exists, or 1.x if 2.x exists).

there are actually some packages that don't follow this structure either I think for noetic let's just use rosdistro For ROS 2 Rolling we can then use upstream releases

[jwhendy]

I think for noetic let's just use rosdistro

Oops, you're right. I think was remembering trying to parse upstream repos (differentiate by version number), but then shifted to rosdistro which removes that ambiguity. My bad.

[fmauch]

Yes, I currently leverage rosdistro which makes it quite simple to get the package information.

I haven't found a way to loop over all released packages, yet. But I haven't done a thorough investigation, either.

On May 28, 2020 6:12:24 PM GMT+02:00, John Hendy notifications@github.com wrote:

I think for noetic let's just use rosdistro

Oops, you're right. I think was remembering trying to parse upstream repos (differentiate by version number), but then shifted to rosdistro which removes that ambiguity. My bad.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/ros-noetic-arch/ros-noetic-desktop-full/issues/9#issuecomment-635445373

[fmauch]

A very first version: https://github.com/fmauch/arch_ros_package_monitor

lists all package version from the ROS distribution and the corresponding AUR version.

I would add options to create lists of missing packages, diverging versions, etc... feel free to make suggestions directly in the repo' issue tracker.

[acxz]

This is amazing!!! I feel like this script should go in ros-build-tools since we have a bunch of helper stuff in there. thx for this utility! Yeah the more you can beef it the up the better. Have at it!

[stertingen]

In the next step, we could generate up-to-date PKGBUILD files from that automatically.

This is amazing!!! I feel like this script should go in ros-build-tools since we have a bunch of helper stuff in there. thx for this utility! Yeah the more you can beef it the up the better. Have at it!

Do we have a ros-build-tools repo that is distro-agnostic? I'm thinking of something to manage all active ROS distributions together.

[acxz]

ros-build-tools is supposed to be distro-agnostic. Back in the day it was used for kinetic. However, now that one uses kinetic on Arch (See: https://pkgstats.archlinux.de/ for proof) melodic was p much the only that used/needed it. Now that we have noetic tho we might need to update it properly.

[fmauch]

I've put in all the features I currently have in mind. Feel free to drop any issues inside the repository. If you'd like to go on integrating this into ros-build-tools, we can leave this issue open to tackle this, otherwise I'd also be fine with closing this.

If we integrate it into ros-build-tools - how should we proceed? Should I simply drop a PR on https://github.com/ros-melodic-arch/ros-build-tools-py3? It would be nice, if that repo could get a license, first. I like to try to keep my OSS projects licensed properly.

[acxz]

it should be BSD licensed from what i memba let's keep it open till we get it in ros-build-tools

[fmauch]

Well, it neither contains a LICENSE file nor are any license headers present inside the script files.

[acxz]

yikes yeah that should prob be added then

[acxz]

Just an update @fmauch changes from ros-build-tools-py3 have been merged into ros-build-tools. From now on we will be using ros-build-tools which is hosted here Feel free to make a PR on the LICENSE. Now that I think more about this though, ros-build-tools should just have the necessary files for building ros packages. Maybe a ros-update-tools repo which contains maintainer scripts would be a better place for this (for example this new repo would have the create PKGBUILD script [although this specifically should be done by superflore at some point as well]).

I'll prob make that repo at some point. bionade24 has been saying we should start fresh.

I do have the ros-arch github organization namespace. Maybe all ros related packages/repos for Arch should go there and only the ros distro specific packages go in the respective ros distro github orgs.

I think that would help conglomerate ros arch related repos that everyone has been creating under their own user profiles. Let me know how that sounds to everyone.

@fmauch would you be willing to be a member for that org? We could then add your arch-ros-package-monitor repo in as well.

[fmauch]

Thanks for the update!

I think, keeping all tools for creating, checking and updating the PKGBUILDs should be at the same place.

I'd be happy contributing to such an org!

[acxz]

Sweet, I'll go ahead and add you to that org. Feel free to include your repo in there and at some point this week I'll move the ros-build-tools repo in there as well.

everyone else on this thread just comment here or email me on wanting to join the ros-arch gh org.

[acxz]

In any case I think the conversation here has come to an end, if anyone wants to continue this then please open an issue over at the ros-arch organization instead of this one. All maintenance related convos should happen there from now on and only package specific issues should be made in the respecitve ros-melodic-arch and ros-noetic-arch orgs.

[bionade24]

merged into ros-build-tools. From now on we will be using ros-build-tools which is hosted here Feel free to make a PR on the LICENSE. N

Just one note about the license. It's legally impossible to add a LICENSE for every part we haven't created. You need to drop all the code that isn't from someone who allowed you to release the code under license X. In the moment the situation is bad (as there is a owner of the code who hasn't released them, but releasing it under a license is more than obvious illegal.

[acxz]

Yeah that is a predicament. However from the beginning the license field of the package has been listed as BSD. Even though no license file was included, technically the creator listed the project as BSD. I believe this also means that anyone that has contributed to the before us such as bchretien and zootboy also agreed to license their contributions under BSD.

While maybe not completely legal, I think it’s fine to add the license file now, before even more contributors contributor. If bchretein and zootboy have any problems I am sure we can peacefully talk it through, although I highly doubt they will.