Cannot install 1.3.0-1.0.506 on RHEL/CentOS v7.2

[danlochhead]

Previously had no problems installing marathon and have been doing so a couple of times a day for the last couple of months. However, with the latest build:

Downloading packages:
marathon-1.3.0-1.0.506.el7.x86_64.rpm                                                                                                                                                                                 |  17 MB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : marathon-1.3.0-1.0.506.el7.x86_64                                                                                                                                                                                         1/1
Error unpacking rpm package marathon-1.3.0-1.0.506.el7.x86_64
**error: unpacking of archive failed on file /usr/bin/marathon;57daa6d3: cpio: read**
  Verifying  : marathon-1.3.0-1.0.506.el7.x86_64                                                                                                                                                                                         1/1

Failed:
  marathon.x86_64 0:1.3.0-1.0.506.el7

[skarlsson]

same problem, both upgrading and on clean install centos 7

[garrettux]

same, able to reproduce in a clean vagrant centos 7.2 VM. more details:

[vagrant@mesos-00 ~]$ cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)

[vagrant@mesos-00 ~]$ uname -a
Linux mesos-00 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[vagrant@mesos-00 ~]$ sudo yum list marathon
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: repos.forethought.net
 * extras: centos.mirror.ndchost.com
 * updates: mirrors.seas.harvard.edu
Available Packages
marathon.x86_64                                                                                                                  1.3.0-1.0.506.el7                                                                                                                  mesosphere

[vagrant@mesos-00 ~]$ sudo yum install marathon
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: repos.forethought.net
 * extras: centos.mirror.ndchost.com
 * updates: mirrors.seas.harvard.edu
Resolving Dependencies
--> Running transaction check
---> Package marathon.x86_64 0:1.3.0-1.0.506.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                        Arch                                                         Version                                                                   Repository                                                        Size
==============================================================================================================================================================================================================================================================================
Installing:
 marathon                                                       x86_64                                                       1.3.0-1.0.506.el7                                                         mesosphere                                                        17 M

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 17 M
Installed size: 89 M
Is this ok [y/d/N]: y
Downloading packages:
marathon-1.3.0-1.0.506.el7.x86_64.rpm                                                                                                                                                                                                                  |  17 MB  00:00:14
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : marathon-1.3.0-1.0.506.el7.x86_64                                                                                                                                                                                                                          1/1
Error unpacking rpm package marathon-1.3.0-1.0.506.el7.x86_64
error: unpacking of archive failed on file /usr/bin/marathon;57dafadd: cpio: read
  Verifying  : marathon-1.3.0-1.0.506.el7.x86_64                                                                                                                                                                                                                          1/1

Failed:
  marathon.x86_64 0:1.3.0-1.0.506.el7

Complete!

[mesosphere]
name=Mesosphere Packages for EL 7 - $basearch
baseurl=http://repos.mesosphere.io/el/7/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mesosphere

[mesosphere-noarch]
name=Mesosphere Packages for EL 7 - noarch
baseurl=http://repos.mesosphere.io/el/7/noarch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mesosphere

[mesosphere-source]
name=Mesosphere Packages for EL 7 - $basearch - Source
baseurl=http://repos.mesosphere.io/el/7/SRPMS/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mesosphere

[Daniel-ltw]

Any resolution for this issue?

[lloesche]

I'm looking into this right now. At a first glance the packages that were transferred over to the repository import server were incomplete. Their headers were intact but they are truncated. Instead of 80MB they are 10-20MB. Interestingly both dpkg-sig and rpm were able to sign the packages and later verify the signature.

Package on repo:

$ rpm -K marathon-1.3.0-1.0.506.el7.x86_64.rpm
marathon-1.3.0-1.0.506.el7.x86_64.rpm: rsa sha1 (md5) pgp md5 OK
$ rpm2cpio marathon-1.3.0-1.0.506.el7.x86_64.rpm |  cpio -idmv
cpio: ./usr/bin/marathon not created: newer or same age version exists
cpio: premature end of file

Package generated by CI system:

$ rpm -K marathon-1.3.0-1.0.506.el7.x86_64.rpm
marathon-1.3.0-1.0.506.el7.x86_64.rpm: rsa sha1 (md5) pgp md5 OK
$ rpm2cpio marathon-1.3.0-1.0.506.el7.x86_64.rpm |  cpio -idmv
./usr/bin/marathon
./usr/lib/systemd/system/marathon.service
181426 blocks

Looking at the TeamCity Job we're using scp to copy the files from CI system to signing system. It's therefor possible that network issues cause incomplete transfers. I'm just surprised that neither the rpm nor deb repo tools caught this issue and signed and imported the truncated packages without throwing an error (which would have aborted the import). I just confirmed this by splitting a valid RPM in half and being able to rpm --resign the first half without a problem.

I will now: 1) manually remove all broken 1.3.0 and related release packages from the repo and resync it to S3 (where repos.mesosphere.com is hosted) 2) re-import the packages from the CI system 3) switch the transfer process from scp to using rsync over ssh with checksumming so we can't get any broken imports in the future

[liquid-sky]

@lloesche I was able to install 1.3.0-1.0.506.ubuntu1404 just now 👍

[lloesche]

I've identified and removed the broken packages and uploaded new packages for the last release.

List of packages that have been replaced:

marathon-0.15.7-1.0.501.el6.x86_64.rpm
marathon-0.15.7-1.0.501.el7.x86_64.rpm
marathon-1.1.3-1.0.503.el6.x86_64.rpm
marathon-1.1.3-1.0.503.el7.x86_64.rpm
marathon-1.3.0-1.0.506.el6.x86_64.rpm
marathon-1.3.0-1.0.506.el7.x86_64.rpm
marathon_0.15.7-1.0.501.debian81_amd64.deb
marathon_0.15.7-1.0.501.ubuntu1204_amd64.deb
marathon_0.15.7-1.0.501.ubuntu1404_amd64.deb
marathon_0.15.7-1.0.501.ubuntu1604_amd64.deb
marathon_1.1.3-1.0.503.debian81_amd64.deb
marathon_1.1.3-1.0.503.ubuntu1204_amd64.deb
marathon_1.1.3-1.0.503.ubuntu1404_amd64.deb
marathon_1.1.3-1.0.503.ubuntu1604_amd64.deb
marathon_1.3.0-1.0.506.debian81_amd64.deb
marathon_1.3.0-1.0.506.ubuntu1204_amd64.deb
marathon_1.3.0-1.0.506.ubuntu1404_amd64.deb
marathon_1.3.0-1.0.506.ubuntu1604_amd64.deb

List of corrupt packages that have been removed:

marathon-1.3.0-1.0.498.el6.x86_64.rpm
marathon-1.3.0-1.0.500.el7.x86_64.rpm
marathon-1.3.0-1.0.505.el6.x86_64.rpm
marathon_1.3.0-1.0.493.ubuntu1604_amd64
marathon_1.3.0-1.0.496.ubuntu1404_amd64
marathon_1.3.0-1.0.498.ubuntu1404_amd64
marathon_1.3.0-1.0.499.ubuntu1404_amd64
marathon_1.3.0-1.0.500.debian81_amd64
marathon_1.3.0-1.0.500.debian81_amd64
marathon_1.3.0-1.0.502.debian81_amd64
marathon_1.3.0-1.0.502.debian81_amd64
marathon_1.3.0-1.0.505.debian81_amd64
marathon_1.3.0-1.0.505.debian81_amd64
marathon_1.3.0-1.0.505.ubuntu1404_amd64
marathon_1.3.0-1.0.505.ubuntu1604_amd64
mesos-1.0.1-2.0.93.centos65.x86_64.rpm
mesos-1.1.0-0.0.269.pre.20160822gitacb814b.centos701406.x86_64.rpm
mesos-1.1.0-0.0.277.pre.20160830git037a346.centos65.x86_64.rpm
mesos-1.1.0-0.0.289.pre.20160910git8f7bd95.centos65.x86_64.rpm
mesos_1.1.0-0.0.269.pre.20160822gitacb814b.ubuntu1204_amd64
mesos_1.1.0-0.0.269.pre.20160822gitacb814b.ubuntu1604_amd64
mesos_1.1.0-0.0.270.pre.20160823git0a84a58.ubuntu1404_amd64
mesos_1.1.0-0.0.272.pre.20160825gitae513df.ubuntu1204_amd64
mesos_1.1.0-0.0.273.pre.20160826git9d0976f.ubuntu1204_amd64
mesos_1.1.0-0.0.275.pre.20160828git537584c.ubuntu1604_amd64
mesos_1.1.0-0.0.276.pre.20160829git037a346.debian81_amd64
mesos_1.1.0-0.0.276.pre.20160829git037a346.ubuntu1404_amd64
mesos_1.1.0-0.0.283.pre.20160905git3bb51c3.ubuntu1604_amd64
mesos_1.1.0-0.0.290.pre.20160911git3562f95.debian81_amd64
mesos_1.1.0-0.0.297.pre.20160918gitbe81a92.ubuntu1404_amd64

New Marathon and Chronos packages will be transferred from the CI system to the repo import system using rsync instead of scp. For Mesos the improvement is being implemented at this very moment and should be active within the next hour.

I also purged the Cloudfront CDN to make sure there's no dirty copies of replaced packages in the cache.

Please reopen this issue and mention me directly if you're still having errors/trouble installing packages from the repo. Thanks!

[JayH5]

Thank you! 🍰

[drcrallen]

Great! 1.3.0-1.0.506.ubuntu1404 seems to be deploying correctly. Thanks!

[stahnma]

I'm still showing an older package in repos.mesosphere.com/el/7/

(albeit, this is mesos issue and not marathon, Found this issue via google. mesos version is 1.0.1-2.0.93.centos701406)

[lloesche]

@stahnma that's correct, the latest stable release of Mesos is 1.0.1.

We do provide nightly builds if you enable the unstable repo. E.g.

$ yum --enablerepo=mesosphere-unstable list mesos

should show you the latest nightly build (right now that's 1.1.0-0.0.318.pre.20161007git8ad489f.centos701406).

If that repo is not available on your system I recommend installing our repo via rpm -Uvh http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-3.noarch.rpm.

Hope that helps.