Handle uncompressed NBT files

[winny-]

Not all NBT files are gzipped. Minecraft uses two NBT's, without compression: idcounts.dat and servers.dat. More info at Nbt#Uses.

Is there a way to currently parse a NBT without uncompressing it? I get this error trying to open servers.dat with nbt.nbt.NBTFile:

>>> os.getcwd()
'/Users/winston/Library/Application Support/minecraft'
>>> os.listdir()
['.DS_Store', 'assets', 'launcher.jar', 'launcher.pack.lzma', 'launcher_profiles.json', 'libraries', 'logs', 'options.txt', 'output-client.log', 'resourcepacks', 'saves', 'screenshots', 'servers.dat', 'stats', 'textures_0.png', 'versions']
>>> nbt.VERSION
(1, 4, 1)
>>> serversnbt = nbt.nbt.NBTFile('servers.dat', 'rb')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/site-packages/nbt/nbt.py", line 508, in __init__
    self.parse_file()
  File "/usr/local/lib/python3.4/site-packages/nbt/nbt.py", line 532, in parse_file
    type = TAG_Byte(buffer=self.file)
  File "/usr/local/lib/python3.4/site-packages/nbt/nbt.py", line 85, in __init__
    self._parse_buffer(buffer)
  File "/usr/local/lib/python3.4/site-packages/nbt/nbt.py", line 90, in _parse_buffer
    self.value = self.fmt.unpack(buffer.read(self.fmt.size))[0]
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/gzip.py", line 365, in read
    if not self._read(readsize):
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/gzip.py", line 433, in _read
    if not self._read_gzip_header():
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/gzip.py", line 297, in _read_gzip_header
    raise OSError('Not a gzipped file')
OSError: Not a gzipped file

[winny-]

Oops, I finally noticed one may pass a file-like object to the constructor as keyword argument buffer.

Maybe there should instead be a keyword argument gzipped that defaults to True?

[macfreek]

Glad you found the solution!

I'm not a big fan of the current API. In this case, I would rather see a separate classes GzippedNBTFile and UncompressedNBTFile. Preferably with factory methods instead of alternatives in __init__.

E.g.

GzippedNBTFile(UncompressedNBTFile):
    def __init__(self):
        ...
    @staticmethod
    def from_fileobject(self, fileobj):
        ...
    @staticmethod
    def from_filename(self, filename):
        ...

Also, I rather saw that NBTFile has a TAG_Compound, instead that it is a TAG_Compound.

However, these changes were never made, since we liked to retain backward compatibility, and introducing another method to accomplish the same would likely be confusing to most users.

What would you consider a good API?

[winny-]

That is one approach, though I think the path of least resistance might be adding a gzipped keyword argument that defaults to True, and mark the buffer keyword argument as deprecated (but keep it for backward compatibility) since it assumes the file is not compressed.

Additionally, it is possible to detect if a file is gzipped: https://gist.github.com/winny-/6043044#file-mc_change_spawn-py-L50-L56 — so perhaps auto-detection could be implemented.

[macfreek]

Auto-detection is certainly nice, though not very Pythonic (from import this: In the face of ambiguity, refuse the temptation to guess.).

Point about an easier API is taken. Code contributions are, as always, welcome. I may have a look later, but this has low priority for me.

PS: the reason I prefer a distinct RawNBTFile and GZipNBTFile class, instead of a single NBTFile class with a plethora of __init__ parameters is that it is more scalable: I wouldn't be surprised if at some point, a zlib-compressed NBT File sees the light of day (that is the default compression in region files), and in that case the number of __init__ parameters would grow a bit too big.

[winny-]

Thank you for the prompt feedback.

If I find time I'd love to contribute improvements. Soon, maybe? :grinning:

[SKeppinger]

Hello from the future. Would it be possible to get an explanation on how to use the buffer keyword to read an uncompressed NBT file? I keep getting an error saying the first record is not a compound tag.

[winny-]

Hey, I haven't used this codebase in a long time. Glancing at the source code, I saw unit tests using the buffer= keyword argument. (search for buffer=). Source code for NBTFile exists here I believe you can do something like:

with open('uncompressed.nbt', 'r') as f:
    nbt = NBTFile(buffer=f)

I'm going to go ahead and close this - no change intended and there's a work around available.

[lynrayy]

Can't read servers.dat

Re open this one

[winny-]

Can't read servers.dat

Re open this one

Any luck using the buffer= keyword? https://github.com/twoolie/NBT/issues/80#issuecomment-1950273441

[lynrayy]

No but i found a way to avoid this

Pack .dat into .gz archieve and then use it

[lynrayy]