Hash Sum mismatch failures

[jamshid]

Does anybody else get "Hash Sum mismatch" errors when using this proxy? Restarting the aptcacherng container doesn't seem to help. Even deleting its data volume (it reached 1.4GB) and restarting the container does not fix it. Nor does docker system prune -f --volumes to clear docker image build layers. The docker server is 19.03.11 and ubuntu 16, but upgrading docker to 19.03.13 didn't help.

Finally the fix seemed to be adding apt-get clean && rm -rf /var/lib/apt/lists/* && apt-get update && to the beginning of every RUN command in my Dockerfile which used apt-get. Hopefully this helps someone.

% docker logs -t buildenv_aptcacherng_1
2020-10-22T20:29:19.464601561Z [warn] Epoll ADD(1) on fd 12 failed. Old events were 0; read change was 1 (add); write change was 0 (none); close change was 0 (none): Bad file descriptor

From my buildenv/docker-compose.yml:

  aptcacherng:
    image: sameersbn/apt-cacher-ng:3.3-20200524
    restart: always
    init: true
    deploy:
      resources:
        limits:
          memory: ${APT_CACHER_MEM-300m}
    volumes:
      - aptcache:/var/cache/apt-cacher-ng
    ports:
      - "${PUBLISH_IP-}3142:3142"

[2020-10-22 20:32:23+0000] [example:latest] #17 ERROR: executor failed running [/bin/sh -c apt-get update && apt-get install -y openjdk-8-jre-headless]: runc did not terminate sucessfully
...
[2020-10-22 20:31:37+0000] [example:latest] #17 16.30   openjdk-8-jre-headless
[2020-10-22 20:31:37+0000] [example:latest] #17 16.59 0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
[2020-10-22 20:31:37+0000] [example:latest] #17 16.59 Need to get 28.9 MB of archives.
[2020-10-22 20:31:37+0000] [example:latest] #17 16.59 After this operation, 105 MB of additional disk space will be used.
[2020-10-22 20:31:37+0000] [example:latest] #17 16.59 Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 java-common all 0.68ubuntu1~18.04.1 [14.5 kB]
[2020-10-22 20:31:37+0000] [example:latest] #17 16.80 Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libnspr4 amd64 2:4.18-1ubuntu1 [112 kB]
[2020-10-22 20:32:23+0000] [example:latest] #17 17.66 Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libnss3 amd64 2:3.35-2ubuntu2.12 [1220 kB]
[2020-10-22 20:32:23+0000] [example:latest] #17 22.18 Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 libpcsclite1 amd64 1.8.23-1 [21.3 kB]
[2020-10-22 20:32:23+0000] [example:latest] #17 22.28 Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 openjdk-8-jre-headless amd64 8u265-b01-0ubuntu2~18.04 [27.5 MB]
[2020-10-22 20:32:23+0000] [example:latest] #17 24.16 Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 openjdk-8-jre-headless amd64 8u265-b01-0ubuntu2~18.04 [27.5 MB]
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44 Err:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 openjdk-8-jre-headless amd64 8u265-b01-0ubuntu2~18.04
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44   Hash Sum mismatch
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44   Hashes of expected file:
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44    - SHA512:49310887ca3d1ec4bb565fc84709c251b300577d729befd0ea4336eaed0b704c314b447d86da76077623e689cd8e77cff0281c43506a50d97657a1e0fa65578c
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44    - SHA256:d0c708242898cce58076dca46245f69a9672ffe00e16120cc9b07c2e7102acba
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44    - SHA1:6d1a84de5eea88e509a57e1c855cd331eb145e60 [weak]
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44    - MD5Sum:3f026505370dff5fcbcc6e928f38ede3 [weak]
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44    - Filesize:27519756 [weak]
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44   Hashes of received file:
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44    - SHA512:986e9328c5667acb803b8d612765f7d197e307650df907912f808aef512ed07f6ef816c7cd854da0e6f1722f1389ee92b49c29258ebcb08e41289834cb7162c0
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44    - SHA256:4d0baf168b060ed29c5fa195b1d292b89410d7314f83ddba1a9b1088cc6476f6
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44    - SHA1:81eafe83a04e02cfecbdfd364fc82662d91b5485 [weak]
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44    - MD5Sum:1069da1d5ee7f28504554e6f8428b841 [weak]
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44    - Filesize:27519756 [weak]
[2020-10-22 20:32:23+0000] [example:latest] #17 63.44   Last modification reported: Wed, 05 Aug 2020 15:33:45 +0000

[bucky2076]

yes I am getting similar errors. I also get tcp/ip address problems randomly occur. When it fails I simply re-start the job. Sometimes have to do this several times before it works. thanks for your fix. I suspect need some kind of timeout parameter in case the container cannot respond quickly enough to the request from the client.

[bucky2076]

errors with this container are escalating... getting hash sum mismatch, and tcp closed connection errors. I've capitulated, and moved acng to a virtual machine. Works great now.

[jamshid]

Hmm @sameersbn do you happen to know what might be causing these errors? Now I'm paranoid about SolarWinds / Sunburst build system hacking.

[bucky2076]

with container, I could not easily get access to the acng.conf file. Perhaps the standard install needs tweaking, but would want the conf file to maintain state after upgrading.

[jamshid]

FWIW I switched my image to mbentley/apt-cacher-ng (https://github.com/mbentley/docker-apt-cacher-ng) and I don't seem to be seeing the weird Hash Sum mismatch errors anymore.

services:
  aptcacherng:
    deploy:
      resources:
        limits:
          memory: 300m
    image: mbentley/apt-cacher-ng
    init: true
    ports:
    - published: 3142
      target: 3142
    restart: always
    volumes:
    - aptcacherng:/var/cache/apt-cacher-ng:rw
volumes:
  aptcacherng: {}

[MaStr]

@bucky2076 's solution is the way to go: run a dedicated VM for that.

I tried everything, from this image, the other named image and creating my own based on recent Debian. Nothing solved the "Hash sum missmatch".

Running in a VM will help, but you have to double check with the underlying filesystem. I was using 9P first to use NAS's storage, but it resulted in the same "has sum missmatch" errors then before.. at least it took longer/bigger files.

You can go with a filesystem on the VM itself. I was successful using CIFS with the following options:

 noserverino,credentials=xxxxx,uid=102,gid=110,dir_mode=0755,file_mode=0755

DisconnectTimeout: 5
DlMaxRetries: 10
ExThreshold: 7
FollowIndexFileRemoval: 1
LocalDirs: acng-doc /usr/share/doc/apt-cacher-ng
LogDir: /var/log/apt-cacher-ng
MaxInresponsiveDlSize: 6400
NetworkTimeout: 300
Remap-alxrep: file:archlx_mirrors /archlinux # ; file:backend_archlx # Arch Linux
Remap-betaArmbian: beta.armbian.com ; fi.mirror.armbian.de/beta
Remap-cygwin: file:cygwin_mirrors /cygwin # ; file:backends_cygwin # incomplete, please create this file or specify preferred mirrors here
Remap-debrep: file:deb_mirror*.gz /debian ; file:backends_debian # Debian Archives
Remap-epel:   file:epel_mirrors # Fedora EPEL
Remap-fedora: file:fedora_mirrors # Fedora Linux
Remap-gentoo: file:gentoo_mirrors.gz /gentoo ; file:backends_gentoo # Gentoo Archives
Remap-klxrep: file:kali_mirrors /kali ; file:backends_kali # Kali Linux Archives
Remap-secdeb: security.debian.org security.debian.org/debian-security deb.debian.org/debian-security /debian-security cdn-fastly.deb.debian.org/debian-security ; deb.debian.org/debian-security security.debian.org cdn-fastly.deb.debian.org/debian-security
Remap-sfnet:  file:sfnet_mirrors # ; file:backends_sfnet # incomplete, please create this file or specify preferred mirrors here
Remap-slrep:  file:sl_mirrors # Scientific Linux
Remap-stableArmbian: apt.armbian.com ; fi.mirror.armbian.de/apt
Remap-uburep: file:ubuntu_mirrors /ubuntu ; file:backends_ubuntu # Ubuntu Archives
ReportPage: acng-report.html
ReserveSpace: 1048576
ResponseFreezeDetectTime: 300
StupidFs: 1
SupportDir: /usr/lib/apt-cacher-ng
TrackFileUse: 1