search

ceph / chacra

A binary/file REST API to aid in multi-distro|arch|release management
9 stars 18 forks source link

Problem with ubuntu repos and/or package_manager_version #194

Closed zmc closed 8 years ago

zmc commented 8 years ago

As I'm working on teuthology support for shaman/chacra, I've mostly been running tests against centos 7 since there aren't any ubuntu 14.04 builds yet. Today I tried a few tests against 16.04 and am seeing issues.

First, the rest runs. This one uses my wip branch but is using gitbuilder: http://pulpito.ceph.com/zack-2016-09-09_08:40:06-smoke-jewel-distro-basic-vps/ This run uses shaman: http://pulpito.ceph.com/zack-2016-09-09_08:39:27-smoke-jewel-distro-basic-vps/

Focusing on one job, it appears there may something wrong with the repos being generated, and package_manager_version isn't matching what the packages are actually using:

2016-09-09T14:55:05.362 INFO:teuthology.orchestra.run.vpm107:Running: u'echo deb https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default/ ubuntu/16.04 main | sudo tee /etc/apt/sources.list.d/ceph.list'
2016-09-09T14:55:05.382 INFO:teuthology.orchestra.run.vpm107.stdout:deb https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default/ ubuntu/16.04 main
2016-09-09T14:55:05.383 INFO:teuthology.orchestra.run.vpm107:Running: 'sudo apt-get update'
2016-09-09T14:55:05.666 INFO:teuthology.orchestra.run.vpm107.stdout:Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
2016-09-09T14:55:05.759 INFO:teuthology.orchestra.run.vpm107.stdout:Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
2016-09-09T14:55:05.808 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04 InRelease
2016-09-09T14:55:05.830 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04 Release
2016-09-09T14:55:05.839 INFO:teuthology.orchestra.run.vpm107.stdout:Hit:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
2016-09-09T14:55:05.853 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:05.875 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:05.897 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:05.919 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:05.920 INFO:teuthology.orchestra.run.vpm107.stdout:Hit:10 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
2016-09-09T14:55:05.941 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:05.963 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:05.984 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.006 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.028 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.050 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.072 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.094 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.116 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.138 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.160 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.182 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.204 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.226 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.248 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.270 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.292 INFO:teuthology.orchestra.run.vpm107.stdout:Err:6 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main amd64 Packages
2016-09-09T14:55:06.292 INFO:teuthology.orchestra.run.vpm107.stdout:  404  Not Found
2016-09-09T14:55:06.314 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main all Packages
2016-09-09T14:55:06.337 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:8 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en_US
2016-09-09T14:55:06.359 INFO:teuthology.orchestra.run.vpm107.stdout:Ign:9 https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04/main Translation-en
2016-09-09T14:55:06.397 INFO:teuthology.orchestra.run.vpm107.stdout:Fetched 94.5 kB in 0s (118 kB/s)
2016-09-09T14:55:07.397 INFO:teuthology.orchestra.run.vpm107.stdout:Reading package lists...
2016-09-09T14:55:07.439 INFO:teuthology.orchestra.run.vpm107.stderr:W: The repository 'https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04 Release' does not have a Release file.
2016-09-09T14:55:07.439 INFO:teuthology.orchestra.run.vpm107.stderr:E: Failed to fetch https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default/dists/ubuntu/16.04/main/binary-amd64/Packages  404  Not Found
2016-09-09T14:55:07.440 INFO:teuthology.orchestra.run.vpm107.stderr:E: Some index files failed to download. They have been ignored, or old ones used instead.
2016-09-09T14:55:07.441 INFO:teuthology.orchestra.run.vpm107:Running: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=10.2.2-508-g9bfc0cf-1xenial ceph-mds=10.2.2-508-g9bfc0cf-1xenial ceph-common=10.2.2-508-g9bfc0cf-1xenial ceph-fuse=10.2.2-508-g9bfc0cf-1xenial ceph-test=10.2.2-508-g9bfc0cf-1xenial radosgw=10.2.2-508-g9bfc0cf-1xenial python-ceph=10.2.2-508-g9bfc0cf-1xenial libcephfs1=10.2.2-508-g9bfc0cf-1xenial libcephfs-java=10.2.2-508-g9bfc0cf-1xenial libcephfs-jni=10.2.2-508-g9bfc0cf-1xenial librados2=10.2.2-508-g9bfc0cf-1xenial librbd1=10.2.2-508-g9bfc0cf-1xenial rbd-fuse=10.2.2-508-g9bfc0cf-1xenial'
2016-09-09T14:55:07.483 INFO:teuthology.orchestra.run.vpm107.stdout:Reading package lists...
2016-09-09T14:55:07.644 INFO:teuthology.orchestra.run.vpm107.stdout:Building dependency tree...
2016-09-09T14:55:07.645 INFO:teuthology.orchestra.run.vpm107.stdout:Reading state information...
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-mds' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-common' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-fuse' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'ceph-test' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'radosgw' was not found
2016-09-09T14:55:07.680 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'python-ceph' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'libcephfs1' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'libcephfs-java' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'libcephfs-jni' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'librados2' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'librbd1' was not found
2016-09-09T14:55:07.681 INFO:teuthology.orchestra.run.vpm107.stderr:E: Version '10.2.2-508-g9bfc0cf-1xenial' for 'rbd-fuse' was not found
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/parallel.py", line 83, in __exit__
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/parallel.py", line 101, in next
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/parallel.py", line 19, in capture_traceback
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/task/install.py", line 107, in _update_deb_package_list_and_install
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/orchestra/remote.py", line 192, in run
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/orchestra/run.py", line 399, in run
  File "/home/teuthworker/src/teuthology_wip-shaman-2/teuthology/orchestra/run.py", line 166, in wait
CommandFailedError: Command failed on vpm107 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=10.2.2-508-g9bfc0cf-1xenial ceph-mds=10.2.2-508-g9bfc0cf-1xenial ceph-common=10.2.2-508-g9bfc0cf-1xenial ceph-fuse=10.2.2-508-g9bfc0cf-1xenial ceph-test=10.2.2-508-g9bfc0cf-1xenial radosgw=10.2.2-508-g9bfc0cf-1xenial python-ceph=10.2.2-508-g9bfc0cf-1xenial libcephfs1=10.2.2-508-g9bfc0cf-1xenial libcephfs-java=10.2.2-508-g9bfc0cf-1xenial libcephfs-jni=10.2.2-508-g9bfc0cf-1xenial librados2=10.2.2-508-g9bfc0cf-1xenial librbd1=10.2.2-508-g9bfc0cf-1xenial rbd-fuse=10.2.2-508-g9bfc0cf-1xenial'
zmc commented 8 years ago

I realized after I submitted this that half is chacra and half is shaman; I'm happy to file a shaman issue as well if you want!

andrewschoen commented 8 years ago

I see in this in the log that looks suspicious:

2016-09-09T14:55:07.439 INFO:teuthology.orchestra.run.vpm107.stderr:W: The repository 'https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default ubuntu/16.04 Release' does not have a Release file.
2016-09-09T14:55:07.439 INFO:teuthology.orchestra.run.vpm107.stderr:E: Failed to fetch https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default/dists/ubuntu/16.04/main/binary-amd64/Packages  404  Not Found

The path to these repos are not ../default/dists/ubuntu/16.04/main/ they are ../default/dists/xenial/main/. See: https://2.chacra.ceph.com/r/ceph/jewel/9bfc0cf178dc21b0fe33e0ce3b90a18858abaf1b/ubuntu/xenial/flavors/default/dists/xenial/main/binary-amd64/

Do you think that could be a factor here?

zmc commented 8 years ago

Ah, possibly! I'll look into that, thanks.

zmc commented 8 years ago

Yep, it was that. Summary of the issue is that a value we were using to write that line happened to be the same as the codename for GitbuilderProject, but isn't for ShamanProject - so now we explicitly use the codename value.

I'll open a ticket for the shaman issue soon.