balena delta image's pull fails when separate delta-data-root is used

Description

When a delta image is pulled in the case that --delta-data-root (and other flags required for this) are set, the pull doesn't seem to succeed with the error:

unexpected EOF. Invalid delta tar archive

Steps to reproduce the issue:

start the balena daemon up with a single base folder: balenad --debug --data-root=$(pwd)/base/balena
create an image (pull in some base image, and just add an extra file, for example)
create a delta between this image and the base image used, tag it, push the image to Docker HUB
stop balena daemon, and restart with a separate delta data root pointing to the previous folder, and a new folder for the data root, such as balenad --delta-data-root=$(pwd)/base/balena --delta-storage-driver=overlay2 --data-root=$(pwd)/top/balena (the correct --delta-storage-driver is needed too)
check that balena now does not report the image (full or delta) in balena images
try to pull the delta pushed earlier

Describe the results you received:

The pull fails with unexpected EOF. Invalid delta tar archive:

debian-delta: Pulling from imrehg/deltatest
4f4fb700ef54: Pull complete 
1c74fe47e344: Ready to download 
Total:  [========>                                          ]      32B/187B
unexpected EOF. Invalid delta tar archive

In the daemon (run with --debug) on the first pull attempt this is the message:

DEBU[2018-08-30T12:35:21.066313192+01:00] Calling POST /v1.35/images/create?fromImage=imrehg%2Fdeltatest&tag=debian-delta 
DEBU[2018-08-30T12:35:21.066492990+01:00] Trying to pull imrehg/deltatest from https://registry-1.docker.io v2 
DEBU[2018-08-30T12:35:22.784842683+01:00] Pulling ref from V2 registry: imrehg/deltatest:debian-delta 
DEBU[2018-08-30T12:35:23.612414067+01:00] pulling blob "sha256:1c74fe47e344589fc1b99d785a44faf71c512bc0aaad2dd71725e887f95b60f1" 
DEBU[2018-08-30T12:35:23.612535111+01:00] pulling blob "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 
DEBU[2018-08-30T12:35:24.277139459+01:00] Applying tar in /home/greg/resin/balenadelta/top/balena/overlay2/8ceed3d52c471f8b60da6d070393889931be3af02b3616eea61f728d3af2b94a/diff 
DEBU[2018-08-30T12:35:24.371635601+01:00] Applied tar sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 to 8ceed3d52c471f8b60da6d070393889931be3af02b3616eea61f728d3af2b94a, size: 0 
INFO[2018-08-30T12:35:24.474578354+01:00] Attempting next endpoint for pull after error: unexpected EOF. Invalid delta tar archive 
INFO[2018-08-30T12:35:24.478186653+01:00] Layer sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 cleaned up

Describe the results you expected:

I'd expect the pull to succeeds, similarly how it works without separate delta root, which is:

debian-delta: Pulling from imrehg/deltatest
4f4fb700ef54: Already exists 
1c74fe47e344: Pull complete 
Total:  [==================================================>]     155B/155B
Digest: sha256:7f24ca3478a8425c00e79d434d8f8778b86f4418fa8b30c02f64c4ebcc3bc23b
Status: Downloaded newer image for imrehg/deltatest:debian-delta

Additional information you deem important (e.g. issue happens only occasionally):

same error both with overlay2 and aufs graphdrivers

Output of balena version:

Client:
 Version:   v17.12.0-2-gc87589c33
 API version:   1.35
 Go version:    go1.10.3
 Git commit:    c87589c33
 Built: Thu Aug 30 13:03:35 2018
 OS/Arch:   linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:  v17.12.0-2-gc87589c33
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.10.3
  Git commit:   c87589c33
  Built:    Thu Aug 30 13:03:35 2018
  OS/Arch:  linux/amd64
  Experimental: false

Output of balena info:

balena info when not running with a separate delta data root:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 3
Server Version: v17.12.0-2-gc87589c33
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null
 Log: journald json-file
Swarm: 
 NodeID: 
 Is Manager: false
 Node Address: 
Runtimes: bare runc
Default Runtime: runc
Init Binary: balena-init
containerd version: 
runc version: 13e66eedaddfbfeda2a73d23701000e4e63b5471
init version: N/A (expected: )
Kernel Version: 4.18.5-arch1-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 19.34GiB
Name: hydra
ID: MM6E:Q4MZ:MGAC:WMBG:IYX5:7UYU:EWM6:U5E7:2XLT:EGCH:JYVU:EMXR
Docker Root Dir: /home/greg/resin/balenadelta/base/balena
Debug Mode (client): false
Debug Mode (server): false
Username: imrehg
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

balena info when running with --delta-data-root etc:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: v17.12.0-2-gc87589c33
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null
 Log: journald json-file
Swarm: 
 NodeID: 
 Is Manager: false
 Node Address: 
Runtimes: bare runc
Default Runtime: runc
Init Binary: balena-init
containerd version: 
runc version: 13e66eedaddfbfeda2a73d23701000e4e63b5471
init version: N/A (expected: )
Kernel Version: 4.18.5-arch1-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 19.34GiB
Name: hydra
ID: MM6E:Q4MZ:MGAC:WMBG:IYX5:7UYU:EWM6:U5E7:2XLT:EGCH:JYVU:EMXR
Docker Root Dir: /home/greg/resin/balenadelta/top/balena
Debug Mode (client): false
Debug Mode (server): false
Username: imrehg
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Tested on ArchLinux, as well as ResinOS 2.14.3 (Intel NUC and Raspberry Pi 3)

balena-os / balena-engine

balena delta image's pull fails when separate delta-data-root is used #98