Bug: max read sized reached when importing layer with very large files ss

stacker version

stacker version stacker v1.0.0-rc13-86ba851 liblxc cb8e38aca27a23964941f0f011a8919aab8bebab

Describe the bug

When a stacker layer has a file > 10G, the following stack trace shows:

error: max read size reached
error: exit status 1
stackerbuild.io/stacker/pkg/container.MaybeRunInNamespace
        /stacker-tree/pkg/container/userns.go:102
main.main.func3
        /stacker-tree/cmd/stacker/main.go:324
github.com/urfave/cli/v2.(*Command).Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/command.go:213
github.com/urfave/cli/v2.(*App).RunContext
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/app.go:332
github.com/urfave/cli/v2.(*App).Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/app.go:309
main.main
        /stacker-tree/cmd/stacker/main.go:329
runtime.main
        /usr/lib/go/src/runtime/proc.go:267
runtime.goexit
        /usr/lib/go/src/runtime/asm_amd64.s:1650

I eventually tracked this down to an error from github.com/udhos/equalfile

https://github.com/udhos/equalfile/blob/90a7461a2a226fa62eca6e350b5207ae9c78fff8/equalfile.go#L390

The default max read size in equalfile is 10^10

https://github.com/udhos/equalfile/blob/90a7461a2a226fa62eca6e350b5207ae9c78fff8/equalfile.go#L12

It looks like there are some equalfile Options struct which can use a "Limited Reader" to avoid the max size limit.

To reproduce

Configuration
Client tool used
Seen error

error: max read size reached
error: exit status 1
stackerbuild.io/stacker/pkg/container.MaybeRunInNamespace
        /stacker-tree/pkg/container/userns.go:102
main.main.func3
        /stacker-tree/cmd/stacker/main.go:324
github.com/urfave/cli/v2.(*Command).Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/command.go:213
github.com/urfave/cli/v2.(*App).RunContext
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/app.go:332
github.com/urfave/cli/v2.(*App).Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/app.go:309
main.main
        /stacker-tree/cmd/stacker/main.go:329
runtime.main
        /usr/lib/go/src/runtime/proc.go:267
runtime.goexit
        /usr/lib/go/src/runtime/asm_amd64.s:1650

Expected behavior

stacker can operate on existing roots/layers even if they have very large files inside (>10G)

Screenshots

No response

Additional context

No response

Put this into a file, extract.sh

#!/bin/sh
cat <<EOF | base64 -d > test-case.tgz
H4sIAAAAAAAAA+1Ze1Mbtxbn7/0Up45bHqn2AX7cca7TIcFlmCaQMeS2cx8D8q7W3rK72q60gCH+
7j3SPrwGUtrgksmNzySAJR2do5/OU5ZMSBLRK5Iy6hERXDPLDRmNTTFZWxbZSN1uW/12uu3t+m9N
rU57zWl1nE532+nauM7p2DudNbCXpsEfUCYkTQHW0glNE5Z+dN1D818oPfvGGgWxNaJiAoRlV8bJ
0bvTvX5zA+0hTKicQPPdz3ubxqv3B2/21ISet0ZZEHrG0esDNXRTTM4s7gbG8Ojo5HhxOOVcCuP4
ZPf1T4Ph4hSi756ztJo8endy3G8QIiRP6ZgROU1Yn1+wNKRTIAQFEC9IcQcte4ZDevNisJCthouN
i4lK9qxRisJRfZa5DtUyxe+xUTaGOadSbAbaN4zPfWlLJHnX/8d8mc6/9qD/t1pOp/D/znZnp4X+
397pdFf+/xR0y/+NZyAmLAzdCXPPwQsEHYWsf/x62/5Hx8BJOQkE4D/2W0ZDPwjZugCP+TQLJaAV
gbIiUFZkvN395fT44N8DDCUbjr215dibm8bXHF3KRcfvX+U6ZCMhA5lJ1p9jVf5V7XP6r8Hw+ODo
sH/hmLZpk9R1doy9wav3+2oPHaQahhH48B/4BogPjWbB2ID/vcDrYrEBSO+Hb/oTKRPRs6xxICfZ
yHR5ZCUp/5W5sjxOqauVMoxzggnL45dxyKln6V2YO+HQKMeCeAwFAzRvqQumaTY0j5ulIRBx7L9B
3VANaw5XsbaCqKGuIJNJJudxd5ZvMom4B3a33b49sxCz8SJFwGPDD5StaitSEABNZeBTV6Lp+hAz
5jGvDhm7SngqTYwDi6Cp49bO/ezPUeM+Xq2LAqxUBVTUeenY+6C86F6eT5NXA6R5ow1ldieL5cj8
V69f4NHGOavNoJmqcEB8GuAtEnQTNHrtNUr/hXWFR6jj9LUAkuNazphTGoXLxtXlyVTBuh7SdMx0
QAI/5VF+RDK/eHTovwvkRWzHKR3dlt7LkbAqJQ0WCrZsLDKhkGBXAYaVv8nSlGNp3r/IV+OpPCFm
l4Bph4EKsb7II8Ynb24sz+7/jM3fsvcFE38sQltbW5hHSZrFcT3ExvwSXJphVNYxTWmXpQxw9WNg
S92+/ZexeyRC8OEDoNzmD0uBqsQHDV8yD5qp++mYfO5q7Onpnvp/ONjdezswI29ZMlSR3+m0Ptr/
t7e7Zf/f3bFtVf93uvaq/n8KOuEYa7AM9DKX9QzDMWGvKO8gpMo4Kv8qakLDODs7M+6pSrXf16q9
R5WbzZvbxeG8NiwL6lpRWA4p1Yyfd4eHB4f7vbxZ0QGTQsglcB9EQl32PYTBOYOOvW9sm4BxVgfU
M933noFw0yDBrFkDxjB2TOCJxNqShuFUs5yVz2RnQGOvCNjlJl9OILnH/z9avH2qjAf8fxsdv/D/
dtveQf/fdnbaK/9/EiouO77ooffqD6c8DqfoPGmmKlRVTPd0Qlf5vaeKSf0J/bwHlY8nWGzGmOqD
CLtqYbo05nHg0lB7PXbV1q80iqYWRoeUxdLKRlksM6IHddNNI6/T0r226sLM8TXKQHfqwQctjKI/
kilkiYchCb77rigxiuEgxisMVSESc1J8QHNG0RGLPVHVey4lLlNFMWqGZl+Ni98yKia+IJLzcD58
dU0yGYTCMG5X8/fAopbI/DMd96BCFYeCSDmSyFcTUK8evQo4N+SZV8KWw6Ixi/koZBVe+hMRLMUG
l2iWIBoXoNFrrALNi4l3L3J3nnfV4GA4PBpWjzPNm5vy1WE2g+fgqHcazQtjhpeqIJ9zAF4BBSpA
NRsh0z2FXh2de0EKVh429MiPh7tvB33rTu+j5q7xCqCK/wVE1lZxBngJTc2d71Noiqvxvl34VhSz
lZ5oaW6m0pUO5HEWjTBfYLTXnHCZBuq6MaBrLvXHy37tSHqTQwXFRu2cVs69uYhIyHkC6sEIDmEj
CuJMAI+ZfvGCS7Z+wYCGKpJOtVTJYj1ditab+DyFjY2rvvMCrv55iD+eP9/cfAEeL+zuIWgWsPFw
e/3HHXAKhS9ZcWpMhbhC5VlZO/v3ys9j5Q8Q6JwnJ2pVEAWFNZezROeGfrOGUE0Oug2h4YJiGZYB
aHNkFHL3PGd2fnpVLvncYW9FBd2T/5eR8hfogfzvqDf/Iv+32i1Hff/Xba3y/5PQo/L/V5L+sTUh
+SPV41J/PQXWjneyO9wfnPQbllTzslFLpySBRjOfb5QpqDyv/j9/2I5Vw+KL2qvey3Jp/TVQzLUo
3ucTDN0X0LiTcebrKh1qY8ZHkv7gl3dHw5NTXN+v4WaWZlNqlnMoxfJDl4+AWFisN2/mm8zW598k
KD8lr+eAqBvPIpYGLsG2Me8QY6afm4iYCskiTIhJ/ZHax2MW5ciCkAaY9TRWnmdB45RF/EIpXGga
xKoXrZRLUVjq376tklmlaVQPLilmWVUAlE1t8R0Fn5da2ofyfT93bFjRila0ohX9/9Lvi1zn+QAo
AAA=
EOF
tar xzvf test-case.tgz

Then you can run it and expand the tarball with the test case


$ chmod +x expand.sh
$ ./expand.sh
 stacker  ./expand.sh
x test-max-read-size/clean.sh
x test-max-read-size/go.sh
x test-max-read-size/README.md
x test-max-read-size/build-export-stacker.yaml
x test-max-read-size/stacker.yaml
 stacker  ls -al test-max-read-size
total 40
drwxr-xr-x@  7 ryaharpe  staff   224 May  8 16:11 .
drwxr-xr-x@ 30 ryaharpe  staff   960 May  8 16:11 ..
-rw-r--r--@  1 ryaharpe  staff   343 May  8 11:56 README.md
-rw-r--r--@  1 ryaharpe  staff  1143 May  8 15:01 build-export-stacker.yaml
-rwxr-xr-x@  1 ryaharpe  staff   309 May  8 11:09 clean.sh
-rwxr-xr-x@  1 ryaharpe  staff  2318 May  8 15:50 go.sh
-rw-r--r--@  1 ryaharpe  staff   924 May  8 14:58 stacker.yaml

project-stacker / stacker