search

google / containerregistry

A set of Python libraries and tools for interacting with a Docker Registry.
https://gcr.io
Apache License 2.0
204 stars 116 forks source link

Support images larger than 2GiB in docker_session.upload() #162

Open eshiroma opened 4 years ago

eshiroma commented 4 years ago

This library has been extremely useful for jobs that run docker commands!

One such job recently started failing with OverflowError: string longer than 2147483647 bytes when it tried to copy a 2.6GB image.

I'm sure that there are reasons behind the 2GiB limit, but would it be possible to increase this limit? (e.g. to 4GiB or 8GiB)

eshiroma commented 4 years ago

It looks like the root cause of the library failure is the limitation of the SSL wrapped HTTP client - a similar issue discussed at https://github.com/psf/requests/issues/2717

Bevisy commented 3 years ago

When I upload an image larger than 2GiB, I see that the reason is that zlib.crc32() does not support more than (2^32-1) bytes in python version 2.7.5, resulting in an error.zlib.crc32() is called here https://github.com/google/containerregistry/blob/8a11dc8c53003ecf5b72ffaf035ba280109356ac/client/v2_2/docker_image_.py#L461

$ python2 -c "import zlib;zlib.crc32('a'*(1<<31))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OverflowError: size does not fit in an int

This means that the size of the uploaded image should not exceed 2GiB per layer. We should upgrade python2 to 2.7.18 or python3

jonjohnsonjr commented 3 years ago

We should upgrade python2 to 2.7.18 or python3

You're saying this is fixed with just a version bump?