Closed adlarkin closed 2 years ago
j-rivero
commented
3 years ago If I understand the issue correctly, nightly precedence over stable is the expected behaviour as described in https://github.com/ignitionrobotics/docs/blob/master/releasing/versioning_pre_nightly.md .
adlarkin
commented
3 years ago nightly precedence over stable is the expected behaviour as described in https://github.com/ignitionrobotics/docs/blob/master/releasing/versioning_pre_nightly.md
Thanks for sharing the link - it does say that nightly over stable is expected. So, I think that we have a disagreement of expected behavior somewhere, because if I try this for Citadel, I end up with stable packages for everything except ign-math and ign-cmake (see below).
So, if the documentation in the link above is what we consider "ground truth", I think that we need to check all other Ignition versions to make sure that nightly over stable behavior is preserved. If the linked documentation is incorrect, then we should update that documentation according to the behavior we want and then make sure all Ignition versions mimic the desired behavior.
cc @chapulina
libignition-cmake2-dev/unknown,now 2.8.0+nightly+git20210628+1r3b7870289027ae0fc04c83d5817907ed863a0720-1~bionic amd64 [installed,automatic]
libignition-common3/unknown,now 3.13.2-1~bionic amd64 [installed,automatic]
libignition-common3-av/unknown,now 3.13.2-1~bionic amd64 [installed,automatic]
libignition-common3-av-dev/unknown,now 3.13.2-1~bionic amd64 [installed,automatic]
libignition-common3-core-dev/unknown,now 3.13.2-1~bionic amd64 [installed,automatic]
libignition-common3-dev/unknown,now 3.13.2-1~bionic amd64 [installed,automatic]
libignition-common3-events/unknown,now 3.13.2-1~bionic amd64 [installed,automatic]
libignition-common3-events-dev/unknown,now 3.13.2-1~bionic amd64 [installed,automatic]
libignition-common3-graphics/unknown,now 3.13.2-1~bionic amd64 [installed,automatic]
libignition-common3-graphics-dev/unknown,now 3.13.2-1~bionic amd64 [installed,automatic]
libignition-common3-profiler/unknown,now 3.13.2-1~bionic amd64 [installed,automatic]
libignition-common3-profiler-dev/unknown,now 3.13.2-1~bionic amd64 [installed,automatic]
libignition-fuel-tools4/unknown,now 4.4.0-1~bionic amd64 [installed,automatic]
libignition-fuel-tools4-dev/unknown,now 4.4.0-1~bionic amd64 [installed,automatic]
libignition-gazebo3/unknown,now 3.8.0-1~bionic amd64 [installed,automatic]
libignition-gazebo3-dev/unknown,now 3.8.0-1~bionic amd64 [installed,automatic]
libignition-gazebo3-plugins/unknown,now 3.8.0-1~bionic amd64 [installed,automatic]
libignition-gui3/unknown,now 3.6.0-1~bionic amd64 [installed,automatic]
libignition-gui3-dev/unknown,now 3.6.0-1~bionic amd64 [installed,automatic]
libignition-launch2/unknown,now 2.2.0-1~bionic amd64 [installed,automatic]
libignition-launch2-dev/unknown,now 2.2.0-1~bionic amd64 [installed,automatic]
libignition-math6/unknown,now 6.8.0+nightly+git20210628+1r9519ce74dae8b8c68303ef81e85fda27d9f4f494-1~bionic amd64 [installed,automatic]
libignition-math6-dev/unknown,now 6.8.0+nightly+git20210628+1r9519ce74dae8b8c68303ef81e85fda27d9f4f494-1~bionic amd64 [installed,automatic]
libignition-math6-eigen3-dev/unknown,now 6.8.0+nightly+git20210628+1r9519ce74dae8b8c68303ef81e85fda27d9f4f494-1~bionic amd64 [installed,automatic]
libignition-msgs5/unknown,now 5.7.0-1~bionic amd64 [installed,automatic]
libignition-msgs5-dev/unknown,now 5.7.0-1~bionic amd64 [installed,automatic]
libignition-physics2/unknown,now 2.4.0-1~bionic amd64 [installed,automatic]
libignition-physics2-core-dev/unknown,now 2.4.0-1~bionic amd64 [installed,automatic]
libignition-physics2-dartsim/unknown,now 2.4.0-1~bionic amd64 [installed,automatic]
libignition-physics2-dartsim-dev/unknown,now 2.4.0-1~bionic amd64 [installed,automatic]
libignition-physics2-dev/unknown,now 2.4.0-1~bionic amd64 [installed,automatic]
libignition-physics2-mesh-dev/unknown,now 2.4.0-1~bionic amd64 [installed,automatic]
libignition-physics2-sdf-dev/unknown,now 2.4.0-1~bionic amd64 [installed,automatic]
libignition-physics2-tpe/unknown,now 2.4.0-1~bionic amd64 [installed,automatic]
libignition-physics2-tpe-dev/unknown,now 2.4.0-1~bionic amd64 [installed,automatic]
libignition-physics2-tpelib/unknown,now 2.4.0-1~bionic amd64 [installed,automatic]
libignition-physics2-tpelib-dev/unknown,now 2.4.0-1~bionic amd64 [installed,automatic]
libignition-plugin/unknown,now 1.2.0-1~bionic amd64 [installed,automatic]
libignition-plugin-dev/unknown,now 1.2.0-1~bionic amd64 [installed,automatic]
libignition-rendering3/unknown,now 3.5.0-1~bionic amd64 [installed,automatic]
libignition-rendering3-core-dev/unknown,now 3.5.0-1~bionic amd64 [installed,automatic]
libignition-rendering3-dev/unknown,now 3.5.0-1~bionic amd64 [installed,automatic]
libignition-rendering3-ogre1/unknown,now 3.5.0-1~bionic amd64 [installed,automatic]
libignition-rendering3-ogre1-dev/unknown,now 3.5.0-1~bionic amd64 [installed,automatic]
libignition-rendering3-ogre2/unknown,now 3.5.0-1~bionic amd64 [installed,automatic]
libignition-rendering3-ogre2-dev/unknown,now 3.5.0-1~bionic amd64 [installed,automatic]
libignition-sensors3/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-air-pressure/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-air-pressure-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-altimeter/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-altimeter-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-camera/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-camera-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-core-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-depth-camera/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-depth-camera-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-gpu-lidar/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-gpu-lidar-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-imu/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-imu-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-lidar/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-lidar-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-logical-camera/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-logical-camera-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-magnetometer/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-magnetometer-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-rendering/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-rendering-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-rgbd-camera/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-rgbd-camera-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-thermal-camera/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-sensors3-thermal-camera-dev/unknown,now 3.2.0-1~bionic amd64 [installed,automatic]
libignition-tools-dev/unknown,now 1.2.0-1~bionic amd64 [installed,automatic]
libignition-transport8/unknown,now 8.2.0-1~bionic amd64 [installed,automatic]
libignition-transport8-core-dev/unknown,now 8.2.0-1~bionic amd64 [installed,automatic]
libignition-transport8-dev/unknown,now 8.2.0-1~bionic amd64 [installed,automatic]
libignition-transport8-log/unknown,now 8.2.0-1~bionic amd64 [installed,automatic]
libignition-transport8-log-dev/unknown,now 8.2.0-1~bionic amd64 [installed,automatic]
chapulina
commented
3 years ago nightly precedence over stable is the expected behaviour
Wow that caught me by surprise. I understand that it is done that way for libraries that have nightlies built after their stable release. For example, ign-math 6.8's nightlies are newer than the 6.8 stable release.
But what's happening with ign-gazebo5 is that the nightlies are from a time before the stable release, and since the stable release we'be been building nightlies for the next major. So we're left with stale nightlies in the nightly repo.
What should we do to solve this situation, @j-rivero ? Do we remove the old nightlies once we start building nightlies for a new version? Or maybe we do like on Homebrew and version the nightlies (N-1).999.999?
chapulina
commented
3 years ago Btw, this combination of 2 different "types" of nightlies is one of the reasons that I think it would be simpler to standardize the way we do nightlies, always building them for unstable versions (#319).
j-rivero
commented
2 years ago Resurrecting this:
Quoting https://github.com/ignitionrobotics/docs/blob/master/releasing/versioning_pre_nightly.md:
Nightlies for version X.Y.Z are snapshots of development code merged on top of a released version X.Y.Z. This includes features and/or patches that will be shipped into the next release. Therefore nightlies need to have precedence over stable releases. Another use case for nightlies is to serve as continuous unstable releases when preparing a new version X.Y.Z. In this scenario the stable release should have precedence over the nightlies generated before it. To get this precedence, the nighlty version uses the trick of setting the version to {X-1.99.99} (i.e: if the version to release is 9.0.0, the nightlies used before the version will use 8.99.99).
Answering Louise:
But what's happening with
ign-gazebo5is that the nightlies are from a time before the stable release, and since the stable release we'be been building nightlies for the next major. So we're left with stale nightlies in the nightly repo.
If we keep building nightlies on top of a stable release , they are newer code than the stable so you get the nightlies if you have the nightly repository enabled. Use case 1) described in the document above. I think that we have killed this approach since https://github.com/ignition-tooling/release-tools/issues/319 so we can probably remove it from the document if we don't plan to use nightlies on top of stable releases.
What should we do to solve this situation, @j-rivero ? Do we remove the old nightlies once we start building nightlies for a new version?
Ideally packages should not be removed, we could it from nightlies to solve a particular problem since the packages are experimental. If that is case for some Citadel versions, I can remove them to be consistent with the change in #319
Or maybe we do like on Homebrew and version the nightlies
(N-1).999.999?
I think is what we have done or suppose to be doing according to https://github.com/ignitionrobotics/docs/blob/master/releasing/versioning_pre_nightly.md
To get this precedence, the nighlty version uses the trick of setting the version to {X-1.99.99} (i.e: if the version to release is 9.0.0, the nightlies used before the version will use 8.99.99).
scpeters
commented
2 years ago When installing the Edifice binaries, I end up with the nightlies if I add the sources for both the stable and nightly packages. Since Edifice is a stable release, I should end up with the stable binaries instead of the nightlies if both are specified.
Steps to reproduce:
sudo sh -c 'echo "deb http://packages.osrfoundation.org/gazebo/ubuntu-stable `lsb_release -cs` main" > /etc/apt/sources.list.d/gazebo-stable.list' sudo sh -c 'echo "deb http://packages.osrfoundation.org/gazebo/ubuntu-nightly `lsb_release -cs` main" > /etc/apt/sources.list.d/gazebo-nightly.list' wget http://packages.osrfoundation.org/gazebo.key -O - | sudo apt-key add - sudo apt-get update sudo apt-get install ignition-edifice
I did a slightly different procedure in a clean docker container and did not see the issue of outdated nightlies displacing newer stable packages
# clean ubuntu:bionic image
apt update
apt install curl gnupg
curl -sSL http://get.gazebosim.org | sh
apt install ignition-edifice
# add /etc/apt/sources.list.d/gazebo-nightly.list
apt updateAt this point I did not install anything else, I simply listed the packages that were outdated:
# apt list --upgradable
Listing... Done
ignition-tools/unknown 1.4.1+nightly+git20211108+1rfefc79e3adb643af7528b0db587bc01a6b285f7d-1~bionic amd64 [upgradable from: 1.4.1-1~bionic]
libignition-cmake2-dev/unknown 2.9.0+nightly+git20211108+1r3971db877d5aca63e2ff55e31ff8367af14c8a3d-1~bionic amd64 [upgradable from: 2.9.0-1~bionic]
libignition-common4/unknown 4.4.0+nightly+git20211108+1r8ba9b71526b63193159adc577976949e83a089ee-1~bionic amd64 [upgradable from: 4.4.0-1~bionic]
libignition-common4-av/unknown 4.4.0+nightly+git20211108+1r8ba9b71526b63193159adc577976949e83a089ee-1~bionic amd64 [upgradable from: 4.4.0-1~bionic]
libignition-common4-av-dev/unknown 4.4.0+nightly+git20211108+1r8ba9b71526b63193159adc577976949e83a089ee-1~bionic amd64 [upgradable from: 4.4.0-1~bionic]
libignition-common4-core-dev/unknown 4.4.0+nightly+git20211108+1r8ba9b71526b63193159adc577976949e83a089ee-1~bionic amd64 [upgradable from: 4.4.0-1~bionic]
libignition-common4-dev/unknown 4.4.0+nightly+git20211108+1r8ba9b71526b63193159adc577976949e83a089ee-1~bionic amd64 [upgradable from: 4.4.0-1~bionic]
libignition-common4-events/unknown 4.4.0+nightly+git20211108+1r8ba9b71526b63193159adc577976949e83a089ee-1~bionic amd64 [upgradable from: 4.4.0-1~bionic]
libignition-common4-events-dev/unknown 4.4.0+nightly+git20211108+1r8ba9b71526b63193159adc577976949e83a089ee-1~bionic amd64 [upgradable from: 4.4.0-1~bionic]
libignition-common4-graphics/unknown 4.4.0+nightly+git20211108+1r8ba9b71526b63193159adc577976949e83a089ee-1~bionic amd64 [upgradable from: 4.4.0-1~bionic]
libignition-common4-graphics-dev/unknown 4.4.0+nightly+git20211108+1r8ba9b71526b63193159adc577976949e83a089ee-1~bionic amd64 [upgradable from: 4.4.0-1~bionic]
libignition-common4-profiler/unknown 4.4.0+nightly+git20211108+1r8ba9b71526b63193159adc577976949e83a089ee-1~bionic amd64 [upgradable from: 4.4.0-1~bionic]
libignition-common4-profiler-dev/unknown 4.4.0+nightly+git20211108+1r8ba9b71526b63193159adc577976949e83a089ee-1~bionic amd64 [upgradable from: 4.4.0-1~bionic]
libignition-math6/unknown 6.9.2+nightly+git20211108+1r074aa5346a16777536a669e189c3ec79c53e9940-1~bionic amd64 [upgradable from: 6.9.2-1~bionic]
libignition-math6-dev/unknown 6.9.2+nightly+git20211108+1r074aa5346a16777536a669e189c3ec79c53e9940-1~bionic amd64 [upgradable from: 6.9.2-1~bionic]
libignition-math6-eigen3-dev/unknown 6.9.2+nightly+git20211108+1r074aa5346a16777536a669e189c3ec79c53e9940-1~bionic amd64 [upgradable from: 6.9.2-1~bionic]
libignition-plugin/unknown 1.2.1+nightly+git20211108+1r0da6f078a3b98c3ac962378272dd29c80b70b119-1~bionic amd64 [upgradable from: 1.2.1-1~bionic]
libignition-plugin-dev/unknown 1.2.1+nightly+git20211108+1r0da6f078a3b98c3ac962378272dd29c80b70b119-1~bionic amd64 [upgradable from: 1.2.1-1~bionic]
libignition-tools-dev/unknown 1.4.1+nightly+git20211108+1rfefc79e3adb643af7528b0db587bc01a6b285f7d-1~bionic amd64 [upgradable from: 1.4.1-1~bionic]
libignition-utils1/unknown 1.1.0+nightly+git20211108+1re9c439b65e1d773654e0bd6bed2b45d4f8d9e609-1~bionic amd64 [upgradable from: 1.1.0-1~bionic]
libignition-utils1-cli-dev/unknown 1.1.0+nightly+git20211108+1re9c439b65e1d773654e0bd6bed2b45d4f8d9e609-1~bionic amd64 [upgradable from: 1.1.0-1~bionic]
libignition-utils1-dev/unknown 1.1.0+nightly+git20211108+1re9c439b65e1d773654e0bd6bed2b45d4f8d9e609-1~bionic amd64 [upgradable from: 1.1.0-1~bionic]I wonder if you can run apt list --upgradable on your system and see if it wants to upgrade from those old nightlies onto the stable versions?
chapulina
commented
2 years ago I think is what we have done or suppose to be doing according to https://github.com/ignitionrobotics/docs/blob/master/releasing/versioning_pre_nightly.md
Ahh I think this would solve all our problems, but I don't think this is what's happening right now :thinking:
scpeters
commented
2 years ago I think is what we have done or suppose to be doing according to https://github.com/ignitionrobotics/docs/blob/master/releasing/versioning_pre_nightly.md
Ahh I think this would solve all our problems, but I don't think this is what's happening right now 🤔
from my testing, I'm not sure there's a problem since I couldn't reproduce the stale binaries
chapulina
commented
2 years ago from my testing, I'm not sure there's a problem since I couldn't reproduce the stale binaries
Ah sorry @scpeters , I misunderstood your comment. I'm confused though, your testing seems to suggest that the system wants to install nightlies on top of stable (i.e. libignition-common4/unknown 4.4.0+nightly... [upgradable from: 4.4.0-1~bionic]). Am I reading this wrong? This means the nightlies take precedence, which is ok for those versions.
It shouldn't be possible to reproduce this issue with Edifice right now since all pre-release nightlies have been superseded by higher minor versions (i.e. there are no X.0.0 on Edifice anymore). What happened with Edifice:
X.0.0~nightlyX.0.0 stable - this was never picked instead of the nightlies thoughX+1.0.0 X.1.0 - finally this stable version overrides the old nightliesIn that situation, the nightlies available were older than the stable version and installing them was causing issues.
But now I think we can't reproduce it even with Fortress, because we're currently still building fresh nightlies from it. So we currently have:
X.0.0~nightlyX.0.0 stableX.0.0~nightly every nightThese nightlies still take precedence, but that's ok because they're indeed newer.
scpeters
commented
2 years ago from my testing, I'm not sure there's a problem since I couldn't reproduce the stale binaries
Ah sorry @scpeters , I misunderstood your comment. I'm confused though, your testing seems to suggest that the system wants to install nightlies on top of stable (i.e.
libignition-common4/unknown 4.4.0+nightly... [upgradable from: 4.4.0-1~bionic]). Am I reading this wrong? This means the nightlies take precedence, which is ok for those versions.
the nightlies do take precedence over stable versions if they were built more recently than the stable version. I think that is intended. The problem that I thought @adlarkin was reporting was old nightlies being preferred over newer stable versions:
libignition-gazebo5/unknown,now 5.0.0+nightly+git20210415+1ra41cf80049c07977079d0b215398dc4225b63038-1~bionic amd64 [installed,automatic]
libignition-gazebo5-dev/unknown,now 5.0.0+nightly+git20210415+1ra41cf80049c07977079d0b215398dc4225b63038-1~bionic amd64 [installed,automatic]
libignition-gazebo5-plugins/unknown,now 5.0.0+nightly+git20210415+1ra41cf80049c07977079d0b215398dc4225b63038-1~bionic amd64 [installed,automatic]
These lines in particular seemed to be problematic, since the latest stable release is newer than a 5.0.0 nightly of ign-gazebo. I wasn't able to reproduce this, though. I think if he ran apt upgrade he might get the stable version of ign-gazebo5.
chapulina
commented
2 years ago I think if he ran apt upgrade he might get the stable version of ign-gazebo5.
Now he'll get stable 5.2.0, so the original issue is not currently reproducible
scpeters
commented
2 years ago oh, I didn't realize how old this issue is. I'm just noticing all the dates right now
j-rivero
commented
2 years ago Do you think that we can close this now or is there anything more we need to fix?
adlarkin
commented
2 years ago from my testing, I'm not sure there's a problem since I couldn't reproduce the stale binaries
Ah sorry @scpeters , I misunderstood your comment. I'm confused though, your testing seems to suggest that the system wants to install nightlies on top of stable (i.e.
libignition-common4/unknown 4.4.0+nightly... [upgradable from: 4.4.0-1~bionic]). Am I reading this wrong? This means the nightlies take precedence, which is ok for those versions.It shouldn't be possible to reproduce this issue with Edifice right now since all pre-release nightlies have been superseded by higher minor versions (i.e. there are no
X.0.0on Edifice anymore). What happened with Edifice:
- Release
X.0.0~nightly- Release
X.0.0stable - this was never picked instead of the nightlies though- Next nightlies released from
X+1.0.0- Release
X.1.0- finally this stable version overrides the old nightliesIn that situation, the nightlies available were older than the stable version and installing them was causing issues.
But now I think we can't reproduce it even with Fortress, because we're currently still building fresh nightlies from it. So we currently have:
- Release
X.0.0~nightly- Release
X.0.0stable- Release
X.0.0~nightlyevery nightThese nightlies still take precedence, but that's ok because they're indeed newer.
I just re-did the steps in the original comment of this issue for both edifice and fortress. I can confirm that on edifice, the stable packages take preference over the nightly packages (except for the packages shared between edifice and fortress - for example, math6 - which makes sense because the nightlies are newer, as @chapulina said), and on fortress, the nightlies take preference over stable binaries.
Do you think that we can close this now or is there anything more we need to fix?
I think we can close this, @j-rivero. The tests I just ran/mentioned above gave me the correct/expected behavior (stable packages when they're newer than nightlies, and nightlies when they're newer than stable).
When installing the Edifice binaries, I end up with the nightlies if I add the sources for both the stable and nightly packages. Since Edifice is a stable release, I should end up with the stable binaries instead of the nightlies if both are specified.
Steps to reproduce:
Here's what I see when I check which ignition packages were installed (this is in ubuntu bionic):