Closed rmarianski closed 6 years ago
Hhmm, the odd thing here is that with a local example, it works. But when running it through the rawr generation code, I get an error: I/O operation on closed file
. I'll have to dig into this a little bit more.
The local test I was using is:
from cStringIO import StringIO
buf = StringIO()
rawr_formatter = Gzip(Msgpack())
writer = rawr_formatter.create(buf)
writer.write('foo', 'bar')
writer.flush()
writer.close()
data = buf.getvalue()
buf.close()
ungzip = GzipFile('<file>', 'rb', 0, BytesIO(data))
unpacker = Unpacker(file_like=ungzip)
result = list(unpacker)
print result
I think what we might need is for raw_tiles.formatter.gzip.Gzip:create
to return a rawr_tiles.formatter.gzip.File
object which keeps references to both buf
and gz
and implements flush by calling it on each in turn, and likewise for close.
Alternatively, we could just drop support for the Gzip
formatter. This isn't the first time it has caused headaches! And we're using a .zip
container now, which already includes compression. The compression level for zipfile can't be changed, but luckily is currently defaulted to 6, which is close enough to the 5 we were using before. (Note, if we don't drop Gzip
formatter, we should use zipfile.ZIP_STORED
to make sure we're not double-compressing.)
Closed in favor of https://github.com/tilezen/raw_tiles/pull/12
Connects to https://github.com/mapzen/tile-tasks/issues/290
I optimistically based this pr off of https://github.com/tilezen/raw_tiles/pull/10
This resolves the gzip CRC errors.