search

Fenixin / Minecraft-Region-Fixer

Python script to fix some of the problems of the Minecraft save files (region files, *.mca).
GNU General Public License v3.0
557 stars 101 forks source link

Region fixer crashing when attempting to fix certain worlds #151

Open VL4DST3R opened 3 years ago

VL4DST3R commented 3 years ago

Describe the bug On my multiverse server I have quite a number of maps (a little over 40 atm) of varying ages and levels of minecraft fuckery/corruption affecting them- that I'm currently trying to deal with. While some of the maps were fixed successfully, others would cause Region fixer to throw out an "Ops! Something went really wrong and regionfixer crashed." error. Bug reports below.

Full copied text from the MS-DOS view All worlds run the same command and arguments, only the path being different: python regionfixer.py --fc --fm --fw -p 24 "E:\Minecraft\Minecraft Bukkit Server\worlds\creative"

DO note that i run this from a batch file to help with the iteration for each world, but i doubt it should cause any issues.

Also, since it seems that the Multiverse way of structuring dimensions in their own wold_[dimension_name] structure is not recognized by Region fixer- eg: "Overworld" worlds path can be left as worlds\creative", and Region fixer also detects the level.dat (and all the other) files present there correctly, Nether/End worlds require full path to the dimensions, Ex: worlds\creative_nether\DIM-1" and worlds\creative_the_end\DIM1 respectively to be read at all, and cannot make use of level.dat or other data present in the world folder.

Expected behavior While i understand it can't literally work miracles with all my wonky worlds, i'm expecting it to at least not completely crash when reading some of them? Again, logs attached below.

Screenshots n/a

Files that would help solving the issue if requested i can provide some of the worlds causing those issues

Desktop (please complete the following information):

Additional context

Bug report 1 (and the most common):

<class 'TypeError'>
Traceback (most recent call last):
  File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 374, in fix_problematic_chunks
    chunk = region_file.get_chunk(*local_coords)
  File "E:\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 599, in get_chunk
    return self.get_nbt(x, z)
  File "E:\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 574, in get_nbt
    data = self.get_blockdata(x, z) # This may raise a RegionFileFormatError.
  File "E:\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 566, in get_blockdata
    raise ChunkDataError(err)
nbt.region.ChunkDataError: Error -3 while decompressing data: incorrect data check

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "regionfixer.py", line 588, in <module>
    value = main()
  File "regionfixer.py", line 548, in main
    fix_bad_chunks(args, w)
  File "regionfixer.py", line 71, in fix_bad_chunks
    counter = scanned_obj.fix_problematic_chunks(problem)
  File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 1398, in fix_problematic_chunks
    counter += regionset.fix_problematic_chunks(status)
  File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 936, in fix_problematic_chunks
    counter += self._set[r].fix_problematic_chunks(status)
  File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 391, in fix_problematic_chunks
    out += dc.decompress(i)
TypeError: a bytes-like object is required, not 'int'

Bug report 2 (just one instance):

**********
*** Exception while scanning:
*** 0f31e1bc-88f0-3d0a-95c5-47cd965cb541-6507148097825114692.dat
**********
*** Printing the child's traceback:
*** Exception:<class 'EOFError'>Compressed file ended before the end-of-stream marker was reached
**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\scan.py, line 746, in scan_data
***   _ = nbt.NBTFile(filename=s.path)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 628, in __init__
***   self.parse_file()**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 655, in parse_file
***   self._parse_buffer(self.file)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 493, in _parse_buffer
***   tag._parse_buffer(buffer)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 486, in _parse_buffer
***   name = TAG_String(buffer=buffer).value**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 345, in __init__
***   self._parse_buffer(buffer)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 350, in _parse_buffer
***   read = buffer.read(length.value)**********
*** File C:\Users\Vlad\AppData\Local\Programs\Python\Python37-32\lib\gzip.py, line 276, in read
***   return self._buffer.read(size)**********
*** File C:\Users\Vlad\AppData\Local\Programs\Python\Python37-32\lib\_compression.py, line 68, in readinto
***   data = self.read(len(byte_view))**********
*** File C:\Users\Vlad\AppData\Local\Programs\Python\Python37-32\lib\gzip.py, line 482, in read
***   raise EOFError("Compressed file ended before the "
**********
VincyChannel commented 3 years ago

Same for me

mawoka-myblock commented 3 years ago

And for me! Any fix so far?

VL4DST3R commented 3 years ago

This project is very much dead, so don't expect any fixes anytime soon. Also given 1.8 is soon approaching which will probably completely change how worlds are saved yet again, this tool will most likely die with it.

VincyChannel commented 3 years ago

This project is very much dead, so don't expect any fixes anytime soon. Also given 1.8 is soon approaching which will probably completely change how worlds are saved yet again, this tool will most likely die with it.

Yes I know, now this project is dead, rip

Fenixin commented 3 years ago

It's dead for now. I don't have the free time to update this and the NBT library and there are not many people cooperating so, yeah, it's dead for now. I kind of feel bad saying this but right now it feels like a chore (it was fun before).

This project is very much dead, so don't expect any fixes anytime soon. Also given 1.8 is soon approaching which will probably completely change how worlds are saved yet again, this tool will most likely die with it.

I didn't know that 1.8 was changing the save format. Do you have any information/source about this?

VL4DST3R commented 3 years ago

I said "probably"; but it does seem reasonable to expect that, considering world saves now have to deal with a completely new generation algorithm (and probably new parameters to set up for it), increased height limit and introduction of the 3d biomes to achieve these new terrain features.

Fenixin commented 3 years ago

We already had changes in the game like that without changes in the world format. For example: generation algorithm changed a few times, the height limit changed at least once withouth repercuting.

A really hope they don't change the world format.

mawoka-myblock commented 3 years ago

Although the project is dead for now, your tool really helped me and fixed my issue by deleting, not fixing the chunk. Thanks for your great work, @Fenixin ❤️ !

Fenixin commented 3 years ago

Although the project is dead for now, your tool really helped me and fixed my issue by deleting, not fixing the chunk. Thanks for your great work, @Fenixin ❤️ !

Yeah, maybe ut should be called Chunk-remover, because I think that's the most used feature :P

Thaks a lot for the kind comment, it means a lot!

Fenixin commented 3 years ago

@VL4DST3R , ok, so this may be too late but, could you provide onen of the worlds that crash regionfixer? At least, could you provide a region file for type of crash?

You can use PM or post here a link.

VL4DST3R commented 3 years ago

@VL4DST3R , ok, so this may be too late but, could you provide onen of the worlds that crash regionfixer? At least, could you provide a region file for type of crash?

You can use PM or post here a link.

Sure thing, I'm sure i have a couple with both flavors of errors. I'll post a Dropbox link tomorrow when i finish work since I'll need to batch re-run regionfixer on all the maps we have to find them again.

Fenixin commented 3 years ago

@VL4DST3R, thanks! When scanning, please, pull the latest master branch.

VL4DST3R commented 3 years ago

Using the latest build, (no changes in errors btw) here are two levels that have the first issue: overworld and a nether dimension to differentiate a bit, and finally with the latter, which turns out i have only one map affected by it. Do tell me when you get around to downloading them so i can delete them from db.

Fenixin commented 3 years ago

Thanks! I've got the files. You can delete them.

VL4DST3R commented 3 years ago

Great, im curious if you manage to find what the issue is!

VL4DST3R commented 3 years ago

Thanks! I've got the files. You can delete them.

Any updates on this?

Fenixin commented 3 years ago

Not really, sorry. I'm in the end of term marking examns like crazy. I hope soon.

l1m3r commented 2 years ago

I ran into the same fault (exact same error) a few days ago but I found a solution for my specific case. To upgrade from spigot 1.17.1 to 1.18 the spigot team recommended to --forceUpgrade first on 1.17.1 and then again on 1.18 Problem was my 3277-Spigot-3892929-2071671 "failed to read some chunks" leading to Error upgrading world.

Running regionfixer.py after that failed --forceUpgrade lead to the crash documented by OP.

Solution was to update to 3284-Spigot-3892929-0ebef35. Upgrading the map worked flawlessly that way (with regionfixer.py cleaning up some minor stuff before the upgrade).

Fenixin commented 2 years ago

I have pushed a change that adds EOFError to the list of exceptions. Those files that crashed regionfixer should be now correctly flagged as broken data files.

I think this error could be handled better but for now this is enough.

I'll leave this as open for some time, I want to test a few things.

Thanks for the worlds and the reports!

ab-tools commented 2 years ago

Hello Alejandro, first thanks a lot for all your efforts with this great tool! :-)

Not sure if you're still interested in new reports for this issue, but for me the tool still crashes using latest master GIT branch version and this region file while trying to fix the 4 broken chunks: r.-1.-1.zip

Nothing critical in this region, so I can just go ahead and delete them, still wanted to let you know this problem seems to still persist.

Thanks again Andreas

Fenixin commented 2 years ago

Hello @ab-tools!

Thanks very much for your report! I really appreciate it. I have downloaded the file and you can remove it if you want.

I'll look into it when I get the time (who knows when...).

Thanks again!

ab-tools commented 2 years ago

Thanks for your reply, Alejandro, you're welcome and no hurry from my side!

Fenixin commented 2 years ago

Just pushed a new branch with a possible fix for this issue. Could anybody test it?

This is the branch: https://github.com/Fenixin/Minecraft-Region-Fixer/tree/bytes-fix

VL4DST3R commented 2 years ago

Just pushed a new branch with a possible fix for this issue. Could anybody test it?

This is the branch: https://github.com/Fenixin/Minecraft-Region-Fixer/tree/bytes-fix

On it, testing now. BTW tested the one you pushed a few hours ago in the hopes it would handle the corruption better, same issue on that one, but i guess you knew that already.

EDIT: nevermind, it seems its the same with the one i tested, which means that no, the issues i'm experiencing are still here, here is some of the output log (same error shows up for all affected worlds)


Unreadable player files:
No problems found.

Unreadable data files:
No problems found.

Chunk problems:
---------------------------------------------
| Problem | Corrupted  Wrong located  Total |
---------------------------------------------
|  Count  |    16           107       21942 |
---------------------------------------------

Region problems:
No problems found.

######### Repairing chunks with status: Corrupted ##########
Repairing chunks in regionset "Level/Region files for Overworld":
The chunk 25,31 in region file region\r.0.-2.mca couldn't be fixed.
The chunk 27,31 in region file region\r.0.-2.mca couldn't be fixed.

Ops! Something went really wrong and regionfixer crashed.

Bug report:

<class 'TypeError'>
Traceback (most recent call last):
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 375, in fix_problematic_chunks
    chunk = region_file.get_chunk(*local_coords)
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 622, in get_chunk
    return self.get_nbt(x, z)
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 597, in get_nbt
    data = self.get_blockdata(x, z) # This may raise a RegionFileFormatError.
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 589, in get_blockdata
    raise ChunkDataError(err)
nbt.region.ChunkDataError: Error -3 while decompressing data: invalid distance too far back

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "regionfixer.py", line 574, in <module>
    value = main()
  File "regionfixer.py", line 534, in main
    fix_bad_chunks(args, w)
  File "regionfixer.py", line 71, in fix_bad_chunks
    counter = scanned_obj.fix_problematic_chunks(problem)
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 1453, in fix_problematic_chunks
    counter += regionset.fix_problematic_chunks(status)
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 971, in fix_problematic_chunks
    counter += self._set[r].fix_problematic_chunks(status)
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 392, in fix_problematic_chunks
    out += dc.decompress(i)
TypeError: a bytes-like object is required, not 'int'
VL4DST3R commented 2 years ago

I sent you 2 worlds a while back, and those two are still some of those causing errors on the latest build. Have you by chance tried running the fixer on those maps yourself?

If you need i can provide them again.

Fenixin commented 2 years ago

I sent you 2 worlds a while back, and those two are still some of those causing errors on the latest build. Have you by chance tried running the fixer on those maps yourself?

If you need i can provide them again.

Those are the world I'm testing this with, and they work. The error you posted tells me that you are not in the branch bytes-fix, I really think you downloaded/didn't check out the correct branch. Could you please check that?

VL4DST3R commented 2 years ago

Are you sure-sure the build there is the one thats updated? I double-checked and it acts the same?

Fenixin commented 2 years ago

Ooops... sorry about that, give me a moment...

VL4DST3R commented 2 years ago

LMFAO i was about to go crazy!

Fenixin commented 2 years ago

I'm sorry about that! It should be there now.

VL4DST3R commented 2 years ago

Well well well, new stuff!

Unreadable player files:
No problems found.

Unreadable data files:
No problems found.

Chunk problems:
---------------------------------------------
| Problem | Corrupted  Wrong located  Total |
---------------------------------------------
|  Count  |    16           107       21942 |
---------------------------------------------

Region problems:
No problems found.

######### Repairing chunks with status: Corrupted ##########
Repairing chunks in regionset "Level/Region files for Overworld":
The chunk 25,31 in region file region\r.0.-2.mca couldn't be fixed.
The chunk 27,31 in region file region\r.0.-2.mca couldn't be fixed.
The chunk 28,3 in region file region\r.0.-2.mca couldn't be fixed.
The chunk 30,0 in region file region\r.0.-2.mca couldn't be fixed.
The chunk 31,11 in region file region\r.0.1.mca couldn't be fixed.
The chunk 5,12 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 6,6 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 6,9 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 17,17 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 23,23 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 26,25 in region file region\r.1.-1.mca couldn't be fixed.
The chunk 3,22 in region file region\r.1.-2.mca couldn't be fixed.
    Repaired 0 chunks in this regionset.

Repairing chunks in regionset "POIs files for Overworld":
    Repaired 0 chunks in this regionset.

Repairing chunks in regionset "Entities files for Overworld":
    Repaired 0 chunks in this regionset.

Repaired 0 chunks with status: Corrupted
No chunks to fix with status: Missing Entities tag

####### Repairing chunks with status: Wrong located ########
Repairing chunks in regionset "Level/Region files for Overworld":
    Repaired 107 chunks in this regionset.

Repairing chunks in regionset "POIs files for Overworld":
    Repaired 0 chunks in this regionset.

Repairing chunks in regionset "Entities files for Overworld":
    Repaired 0 chunks in this regionset.

Repaired 107 chunks with status: Wrong located

I let it run on all my maps, so it will be a hot minute until its done with all of them but so far so good!

Those mentions about chunks that cannot be fixed, what does that mean? Or what should i do?

Fenixin commented 2 years ago

I let it run on all my maps, so it will be a hot minute until its done with all of them but so far so good!

Those mentions about chunks that cannot be fixed, what does that mean? Or what should i do?

A chunk is a compressed file, if something is changed it's usually lost. Although very infrequent, sometimes you can decompress the file byte by byte and when you reach error you have something useful. But this is very infrequent.

So, if those chunks cannot be fixed they should be replaced or removed.

Also, I found a bug, regionfixer says that the chunk is fixed, but nothing happens, will look into it (hopefully) in the next days. Please, take this version of regionfixer as beta, be careful and bakcup all your worlds!

Fenixin commented 2 years ago

And thanks a lot for testing!

VL4DST3R commented 2 years ago

Also, I found a bug, regionfixer says that the chunk is fixed, but nothing happens, will look into it (hopefully) in the next days. Please, take this version of regionfixer as beta, be careful and bakcup all your worlds!

💀

I hope i didn't shoot myself in the foot from all this excitement. You.. sure it doesn't do anything? I do have backups but id rather not wipe everything and restore all worlds.

And thanks a lot for testing!

Any time you need! This is very important to me as well, if it wasn't obious. Do tell me when you get around to making fixes stick.

Fenixin commented 2 years ago

💀

I hope i didn't shoot myself in the foot from all this excitement. You.. sure it doesn't do anything? I do have backups but id rather not wipe everything and restore all worlds.

I hope I didn't shoot you in the foot! AFAIK Regionfixer is writing the new fixed chunks in the region file but if you rescan the world the chunk appears as broken again... which is strange to say the least. Right know I don't trust fixing chunks at all, but the worst will happen to your worls is that nothing will be fixed. I really need to investigate this further.

Any time you need! This is very important to me as well, if it wasn't obious. Do tell me when you get around to making fixes stick.

I'll do!

VL4DST3R commented 1 year ago

hey @Fenixin long time no see XD Hope this finds you well, I just came checking to see if this project had any updates/will ever be updated or if you dropped it entirely? Fingers crossed for your return!

Fenixin commented 1 year ago

Hello!

@VL4DST3R, Thanks for asking. My life is chaos right now. I'd love to have the time for this project. That said I hope to do some stuff this summer, not earlier than july.

See you then!

VL4DST3R commented 1 year ago

Rooting for you to get it all sorted out!

VL4DST3R commented 10 months ago

Any updates on this issue? I recently needed to fix some corrupted worlds and there's nothing out there as far as i know that could (potentially) help besides this, but it still has the same issue as per my original issue above. Please don't abandon this :<