twoolie / NBT

Python Parser/Writer for the NBT file format, and it's container the RegionFile.
MIT License
365 stars 74 forks source link

Too many files open in world.iter_nbt (/w fix) #77

Open Someone3x7 opened 9 years ago

Someone3x7 commented 9 years ago

With a large world world.iter_nbt caused a too many files open exception since it never closes the region files. The simple fix is to add a few lines to close the region files in world.py at line 95 so it looks like this:

def iter_nbt(self):
    """
    Return an iterable list of all NBT. Use this function if you only
    want to loop through the chunks once, and don't need the block or data arrays.
    """
    # TODO: Implement BoundingBox
    # TODO: Implement sort order
    for region in self.iter_regions():
        for c in region.iter_chunks():
            yield c
        if hasattr(region.file,'fileobj') and region.file.fileobj: # <- added
            region.file.fileobj.close() # <- added
        region.file.close() # <- added

edit: Updated fix after encountering again in an even larger world.

macfreek commented 9 years ago

Thanks for the report and the patch, I'll have a look at it shortly. (I wanted to have a look this weekend, but didn't get around it yet. For now, let me just acknowledge your report).

Out of curiousity, how large were these worlds?

Besides your patch, it would be good to write a test case to check if all files are open. However, it seems that the resource module can only list the maximum number of open files, but not the current amount of open files.

Someone3x7 commented 9 years ago

The last few worlds had a little over 1,000,000 chunks each. Takes my old i5 couple days to parse one. I didn't want to get too deep into correcting the issue as I'm loosing my mind slowly and tend to make stupid mistakes with code these days.