tilezen / raw_tiles

Raw tiles
MIT License
12 stars 4 forks source link

Close formatter before getting data #11

Closed rmarianski closed 6 years ago

rmarianski commented 6 years ago

Connects to https://github.com/mapzen/tile-tasks/issues/290

I optimistically based this pr off of https://github.com/tilezen/raw_tiles/pull/10

This resolves the gzip CRC errors.

rmarianski commented 6 years ago

Hhmm, the odd thing here is that with a local example, it works. But when running it through the rawr generation code, I get an error: I/O operation on closed file. I'll have to dig into this a little bit more.

The local test I was using is:

from cStringIO import StringIO

buf = StringIO()
rawr_formatter = Gzip(Msgpack())

writer = rawr_formatter.create(buf)
writer.write('foo', 'bar')
writer.flush()
writer.close()
data = buf.getvalue()
buf.close()

ungzip = GzipFile('<file>', 'rb', 0, BytesIO(data))
unpacker = Unpacker(file_like=ungzip)
result = list(unpacker)
print result
zerebubuth commented 6 years ago

I think what we might need is for raw_tiles.formatter.gzip.Gzip:create to return a rawr_tiles.formatter.gzip.File object which keeps references to both buf and gz and implements flush by calling it on each in turn, and likewise for close.

Alternatively, we could just drop support for the Gzip formatter. This isn't the first time it has caused headaches! And we're using a .zip container now, which already includes compression. The compression level for zipfile can't be changed, but luckily is currently defaulted to 6, which is close enough to the 5 we were using before. (Note, if we don't drop Gzip formatter, we should use zipfile.ZIP_STORED to make sure we're not double-compressing.)

rmarianski commented 6 years ago

Closed in favor of https://github.com/tilezen/raw_tiles/pull/12