Why does ros2/navigation show up as being in rolling?

mikeferguson commented 10 months ago

This repo - https://index.ros.org/r/navigation/github-ros2-navigation/ - is only released in the Ardent/Bouncy rosdistro, but still shows up in the index.ros.org as being part of rolling? That seems like a bug in the indexer...

tfoote commented 10 months ago

That does look like a bug. I think that there might be an assumption that rolling is always the latest. Or else there's some caching going on. I'm not sure where it's getting the branch name from.

 -- Adding repo navigation instance: github-ros2-navigation from uri: https://github.com/ros2/navigation.git with version: bouncy

Updating repo / instance navigation / github-ros2-navigation from uri: https://github.com/ros2/navigation.git

[47.86%] Scraping github-ros2-navigation...
[1;34m --- scraping version for navigation instance: github-ros2-navigation distro: rolling[0m
/home/jenkins-agent/.bundle/ruby/2.7.0/gems/rugged-0.27.7/lib/rugged/repository.rb:42: warning: Using the last argument as keyword parameters is deprecated
[1;34m --- scraping version for navigation instance: github-ros2-navigation distro: ardent[0m
/home/jenkins-agent/.bundle/ruby/2.7.0/gems/rugged-0.27.7/lib/rugged/repository.rb:42: warning: Using the last argument as keyword parameters is deprecated
/home/jenkins-agent/.bundle/ruby/2.7.0/gems/rugged-0.27.7/lib/rugged/repository.rb:42: warning: Using the last argument as keyword parameters is deprecated
[1;34m --- scraping version for navigation instance: github-ros2-navigation distro: bouncy[0m
[1;34m --- scraping version for navigation instance: github-ros2-navigation distro: jade[0m
/home/jenkins-agent/.bundle/ruby/2.7.0/gems/rugged-0.27.7/lib/rugged/repository.rb:42: warning: Using the last argument as keyword parameters is deprecated
[1;34m --- scraping version for navigation instance: github-ros2-navigation distro: indigo[0m
/home/jenkins-agent/.bundle/ruby/2.7.0/gems/rugged-0.27.7/lib/rugged/repository.rb:42: warning: Using the last argument as keyword parameters is deprecated
[1;34m --- scraping version for navigation instance: github-ros2-navigation distro: hydro[0m
/home/jenkins-agent/.bundle/ruby/2.7.0/gems/rugged-0.27.7/lib/rugged/repository.rb:42: warning: Using the last argument as keyword parameters is deprecated
[1;34m --- scraping version for navigation instance: github-ros2-navigation distro: kinetic[0m
/home/jenkins-agent/.bundle/ruby/2.7.0/gems/rugged-0.27.7/lib/rugged/repository.rb:42: warning: Using the last argument as keyword parameters is deprecated

I suspect that is something to do with the scraper finding rolling in the branch name or some other heuristic to decide that branch is associated.

The individual packages also are improperly tagged: https://index.ros.org/p/dwa_local_planner/github-ros2-navigation/#rolling

rkent commented 7 months ago

This happens because the repo https://github.com/ros2/navigation has a branch 'always_update_rolling'. rosindex falls back to implicitly look for valid branches if there exists a branch name containing the distro name. That is done in vcs.rb:

      # save the branch as the version if it matches either the explicit
      # version or the distro name
      if explicit_version
        if branch_name == explicit_version
          return branch, branch_name
        end
      elsif branch_name.include? distro
        return branch, branch_name
      end

There is no reason for this repo to be included in the index at all, as it has been archived and marked as obsolete since 2020.

rkent commented 7 months ago

The fix for this is fairly simple: stop searching any repo that ever existed in rosdistro for possible other valid distro, by looking for branch names containing that distro. I ran a full rosindex where I flagged where implicit versions are being created, and came up with 969 examples. I randomly investigated a few of these cases, here are my notes from that:

arni from github/ROS-PSE. There are rosdistro entries for indigo and jade, but none for kinetic or later. Repo has a branch "kinetic-devel"
aruco_opencv from aruco_opencv. Has current versions (humble, iron) with repo fictionlab/ros_aruco_opencv rosdistro has noetic release under fictionlab/aruco_opencv Implicit noetic was located under ros_aruco_opencv due to branch noetic. Appears to be a distro rename in github, as https://github.com/fictionlab/aruco_opencv now resolved to https://github.com/fictionlab/ros_aruco_opencv. rosindex shows duplicate entries under both names due to the noetic entry with https://github.com/fictionlab/aruco_opencv.git in noetic/distribution.yaml
Ekumen-OS repo andino has (now deleted) branch deprecated-noetic. The repo seems to be ROS2 focused. The old branch must have been recently deleted, but checkout is not deleting old branches.
devel-hydro for uri https://github.com/wavelab/ximea_camera.git. rosdistro only has one version, for indigo. Has branch devel-hydro with only 1 commit "initial code dump ...". I assume they started development under hydro, but when finished was released at indigo
galactic for uri https://github.com/Kukanani/vision_msgs.git This repo now resolves to https://github.com/ros-perception/vision_msgs. Rosindex shows two different repos for vision_msgs, both the Kukanani version as well as ros-perception. It appears that the rosdistro repo for vision_msgs switched from Kukanani to ros-perception at galactic. Yet rosindex shows a vision_msgs version for Kukanani at galactic.
kinetic-devel for uri https://github.com/pal-robotics/tiago_simulation.git. There is a single rosdistro entry for this, at humble. Yet the repo has lots of stale branches with different distros: kinetic, indigo, galactic, etc. Should these versions show in rosindex? The only non-stale branch is "humble-devel" which is also what shows in rosdistro
noetic-devel for uri https://github.com/shadow-robot/sr-config.git rosdistro has entries for previous distros (hydro, indigo, kinetic, melodic) but not for noetic. Should the noetic version show in rosindex because the branch exists?
humble for uri https://github.com/clearpathrobotics/zbar_ros.git This seems to be another case of a repository transfer, here to ros-drivers. clearpathrobotics shows in jade, switched to ros-drivers in kinetic

In none of these cases do I feel that a rosindex entry was appropriate. Should rosindex only reflect entries in rosdistro, or should it aggressively try to find possible versions based only on the existence of branch names? I think that clutter is a major problem in rosindex, so I think we should only show "official" entries from rosdistro. But my knowledge of ROS history and lore is somewhat limited, so I'd appreciate comments on this before I do a PR.

The PR could be fairly trivial (maybe only one line changed), but that would orphan a lot of code. I believe the orphaned code should be deleted with a longer PR.

mikeferguson commented 7 months ago

Should rosindex only reflect entries in rosdistro, or should it aggressively try to find possible versions based only on the existence of branch names

Honestly, if it isn't even in rosdistro, how well supported could it be? @tfoote might have more thoughts on this, but I think it's probably a disservice to ROS users to include things that are outside the rosdistro.

tfoote commented 7 months ago

Ahh, @rkent thanks for digging into that! The original vision of rosindex (https://github.com/rosindex/rosindex) was to index not only the mainline versions of packages but also all the branches and forks. It's a great idea, but we've been slowly moving back from that vision and simplifying it to be the one official version in the index. So with that stated I think that removing the implicit branch matching is the right choice for the new behavior. We should both turn that off as well as eliminating the orphaned code at the same time. As can be seen in these incorrect matches the heuristics are getting the results wrong and misleading people.

To that end a PR to disable the fuzzy matching and any extra logic which can be moderately easily confirmed as unused would be great.

rkent commented 7 months ago

So @tfoote what do I need to do to make progress on the PR that I opened? I don't seem to be able to request review. Do I just need to wait, or is there some step I have not done?

tfoote commented 7 months ago

Thanks for the ping. I missed the notification for that.

tfoote commented 7 months ago

This is resolved in #352

https://index.ros.org/p/navigation/#rolling

ros-infrastructure / rosindex

Why does ros2/navigation show up as being in rolling? #327