search

Podshot / MCEdit-Unified

Combined MCEdit & Pymclevel repository.
http://podshot.github.io/MCEdit-Unified/
ISC License
485 stars 109 forks source link

Support MCPE Leveldb worlds #131

Closed L3ice closed 9 years ago

L3ice commented 10 years ago

I want to port MCPE worlds but I cant. Please update it with the new leveldb format!

https://github.com/Mojang/leveldb-mcpe

Khroki commented 10 years ago

We'd love to, we just need someone who knows both C and Python, and actually uses pocket edition. A combination that has proven hard to come by.

L3ice commented 10 years ago

Ahh :/ I understand that. Maybe I find someone...

TrazLander commented 10 years ago

Ya feel free to keep a look out. Currently I don't think we're much planning to look into it until we have a full blockstate plan somewhat put into place (WIP but slow going). It would make MCPE a little easier to figure out from what I've gathered. But if someone wants to update MCPE support to the current build that would be good, but we're basically built around people who want to do these updates, so we would need someone that wants to do it. So currently it'll hopefully come, but not for some time.

L3ice commented 10 years ago

Found 2 people that are maybe interested. I wait for an answer from them :)

jocopa3 commented 10 years ago

I really don't know Python enough to do it, but I have created a Java World Editor test for the new MCPE format initially to replace MCEdit, but shelved that project due to the shear size of work involved and little time. While I won't be much help in programming a Python version, I can explain most of the LevelDB format.

A good place to start would be the Python LevelDB bindings: https://code.google.com/p/py-leveldb/

Lechros commented 10 years ago

I wish devs would focus on this first... But nobody does since they don't play MCPE

L3ice commented 10 years ago

Well.. Thats sadly true. But other 20-30 million people play mcpe.

naor2013 commented 10 years ago

I can try looking into it next week if it's that important but no promises.. I don't think it would come soon though, sorry

Lechros commented 10 years ago

Well, it's important because you just can't open maps at all. MCEdit doesn't work when it can't open maps...

naor2013 commented 10 years ago

Is there any way to get the map in IOS without jailbreak?

Lechros commented 10 years ago

1.) Download and extract the Minecraft: PE map that you want to play.

2.) Download and install I-FunBox (free).

3.) Connect your iPhone and PC together, and then open I-FunBox, and click on the Managing App Data tab, then click on the Minecraft PE icon under All My Apps.

4.) Click on the following folders in order: Documents > games > com.mojang > minecraftWorlds.

5.) Paste your PE map into the minecraftWorlds folder. Enjoy playing your custom maps!

naor2013 commented 10 years ago

Thanks I think I'll buy MC:PE today just to test it and see what I can do about it (But no promises) I'll try my best..

freundTech commented 10 years ago

@jocopa3 Could you maybe share the code of your unfinished Java world editor with us? I think it would be a good startingpoint.

L3ice commented 10 years ago

(Maybe) its this? https://github.com/jocopa3/PocketViewer

L3ice commented 10 years ago

@naor2013 Did you test it?

naor2013 commented 10 years ago

I bought the game and I had a few tests but I didn't start anything for now.. I'm waiting for 1.2.1.1 to be released and then this is going to be my main thing to work on.. I really hope 1.3.0.0 would be support MCPE (But no promises!)

Lechros commented 10 years ago

I hope it comes early!

jocopa3 commented 9 years ago

Since I have some spare time to work on other projects, I'd like to offer assistance in adding Pocket Edition support. I don't know Python aside from basics, so I won't be of much help there, but I know the Pocket Edition format well-enough to help with it.

The hardest obstacle in adding pocket edition support will be finding and using the specific modified version of LevelDB that Mojang uses in MCPE, and then using it with Python.

You'll need a LevelDB library with ZLib compression, since all worlds are now saved exclusively with that library under that compression algorithm. There are python bindings for LevelDB: (https://code.google.com/p/py-leveldb/), however, the version of LevelDB that it's binded to does not support ZLib compression, so you'll have to build the LevelDB found in Mojang's repo (https://github.com/Mojang/leveldb-mcpe) and bind to it instead.

After those hideous couple of steps, it should be an easy home-stretch to reading/writing Pocket Edition worlds. There is already a lot of documentation on the new PE format and its LevelDB Keys (http://minecraft.gamepedia.com/Pocket_Edition_level_format), but here's the basics:

Each chunk is stored using a 9 byte key (as the documentation cited says). The first 4 bytes are the x position of the chunk, the next four bytes are the z position, and the last byte is the type of data to be accessed. The last byte can be 4 values:

The order of the bytes is little endian.

In Java (I don't know python very well), I get each key using this simple code snippet:

public static byte[] compileKey(int ChunkX, int ChunkZ, byte data){
        return ByteBuffer.allocate(9).order(ByteOrder.LITTLE_ENDIAN).putInt(ChunkX).putInt(ChunkZ).put(data).array();
    }

If the chunk for the key used doesn't exist, then LevelDB will return null. Otherwise, if the chunk does exist, it returns the data for that chunk.

There are other keys as well for local and remote player data, and also flat worlds, which is again explained in the documentation cited.

If anyone needs worlds using the new format to test with, I can offer to upload some. I don't check github often, so you'd have an easier time catching me on reddit (/u/jocopa3) or twitter (@jocopa3).

naor2013 commented 9 years ago

Thanks for the info, that would really help but like you said, we're stuck on changing the levelDb library to be in python..

jocopa3 commented 9 years ago

Well, you don't have to find someone who knows C, Python, and plays pocket edition; you'll have the toughest time doing that since I only know two or three people who fit that criteria. Find someone who knows enough C that they can compile or explain how to compile Mojangs LevelDB library for you, then find someone who can modify the bindings for the library from the py-leveldb project (that is, if the current bindings don't work; if the current bindings do work, then great!).

Lechros commented 9 years ago

stackoverflow...?

Khroki commented 9 years ago

All of the python leveldb wrappers we found either weren't windows compatible (almost all of them) and/or didn't support mojang's particular version of leveldb, so I think we're talking writing a new library or heavily modifying one. The original leveldb isn't windows compatible, so I guess the wrapper makers didn't bother with support.

jocopa3 commented 9 years ago

This may be a stupid question but I'll ask anyway: is it possible to wrap or use Java libraries in Python? If so, I can share the pure-java version of LevelDB I use that is modified to work with MCPE worlds. I doubt such a setup would be close to efficient, but it's probably a lot better than writing your own python port or bindings for Mojang's LevelDB.

Podshot commented 9 years ago

If you can give us the java library, we might be able to work with it.

jocopa3 commented 9 years ago

Here is the compiled LevelDB library and LevelDB API jar files, as well as the source jars to each. I also included a basic example using the LevelDB library in the LevelDB.java source file: https://dl.dropboxusercontent.com/u/94880601/LevelDB-0.8-Tinfoiled-Jocopa3.zip

The version of LevelDB I modified is from someone who goes by the username Tinfoiled, and can be found here: https://github.com/Tinfoiled/leveldb

Their library works for reading worlds, but writing worlds wouldn't work (presumably because of the compression parameters), so I had modified it to use similar parameters as MCPE.

If you have any problems with the library itself, feel free to ask.

naor2013 commented 9 years ago

So it's not Mojang's version, right?

Lechros commented 9 years ago

Yes. Maybe. Or not.

jocopa3 commented 9 years ago

It's not Mojang's, but it should work the same as theirs. Mojang's modified LevelDB is in C++, the one I gave is in Java. The Java version is a modified port of an earlier version of LevelDB, where Mojang's is a modified version of the newer LevelDB, but I haven't faced any issues with compatability between the two yet.

L3ice commented 9 years ago

So will it now be implemented?

Joeschmoh commented 9 years ago

This thread hasn't seen an action in a while, any progress on this?

I'm an old enough dinosaur to be fluent in C, I have worked with python before, and I have kids that play MCPE who want this to work.

It sounds like what you are asking for is a Windows port of the Mojang LevelDB library so you can call it from Python? That sounds plausible, but I'd need more information on exactly what is needed.

Anyway, PM me if are interested.

Khroki commented 9 years ago

Github doesn't have a PM system anymore, so I'll respond here. Basically mojang's LevelDB is already Windows compatible, the issue is we don't have a python library/wrapper to read it, as all the preexisting ones are for the non-mojang version. And without the knowledge of both C++ and Python, we don't have anyone who can make one.

naor2013 commented 9 years ago

Actually, C++

Khroki commented 9 years ago

Fine C++ then. :P Edited last post.

Joeschmoh commented 9 years ago

Okay, I've done most of my programming in C/C++. I've worked with Python before, can't say I'm fluent but probably have enough skills there. I'm willing to help, in any event. I guess the next big question is where to get started.

naor2013 commented 9 years ago

We need the leveldb system to work with Python to allow us to get to the files and read them correctly so we need the link below to work in Python: https://github.com/Mojang/leveldb-mcpe

Joeschmoh commented 9 years ago

Okay, let me dig into this. I have a better idea of what is needed. Looking over the link above, I'm guessing we will want to build their LevelDB interface into a DLL (if it doesn't build that way already) that we can then write some python wrapper code around to act as a glue layer between MCEdit and LevelDB.

My initial guess is that memory management might be trickiest part. But there seem to be a lot of examples on how to handle this (i.e. I googled it). Also, we'll have to consider if Mojang makes different versions of LevelDB that aren't backwards compatible. And finally, if you are just using Python I'm guessing adding in a C++ component means a big change to your build process.

I'm saying this just looking over docs and stuff, obviously I need to actually check out code and see how this all fits together.

Khroki commented 9 years ago

Currently the build process doesn't have any C or C++ compiling at all, Windows is built using Pyinstaller, Mac with Py2App, and linux essentially runs from source using an installer. Seeing as all 3 are different, having to build any C or C++ code may affect our build capabilities, especially with the sensitivity of the Mac environment. So something that directly interfaces or can be pre-compiled for inclusion in the repo (like our NBT library) would be ideal. Not sure if a DLL would work for the unix-based OSes, maybe someone else knows on that.

LaChal commented 9 years ago

DLLs won't work on Linux/OSX systems(*). Moreover, on Linux, you can't just download a dynamic library and have it work. It can work, but it depends on the specific distribution. (Mainly kernel and system libraries versions). So, the source code have to be recompiled on Linux machines. And the code written on Windows uses Windows specific dependencies. It won't compile on Linux or OSX as-is, modifications have to be made...

The most 'simpliest'(**) and effective solution is to have system independent code.

FYI, Python is built over C.

(*) There are some emulators/interfaces to run Windows programs on Linux. But those solutions are tricky to use and configure, and some are expensive. And most of users won't mess with VMs or dive in Wine settings to run a sole program (even wether Wine is a very good stuff :) )...

(**) Porting from a language to another can be hard, very hard. But once done, only one code source is maintained, with some tweaks to fit the target system. This is what we do with MCEdit...

Joeschmoh commented 9 years ago

I've been looking through this. One of the challenges is I can't find any Mojang specific build instructions for the LevelDB code. Sure there is a Makefile and a dependency script, but that alone doesn't answer a lot of the questions on building C/C++ objects. So that'll take time for me to try various combinations and get it to build on each platform.

I'm making an assumption here, but the code is supposed to allow building as a DLL on Windows. I can't see any documentation for how it builds on Linux, but it looks like a static library (the "Linux" support is there, but mostly its written to support Android from what I can tell). Usually, though it's not to hard to make a static library into a ".so" or shared object on Linux. That would be the Linux equivalent of a DLL. Mac is a different beast, with any luck the Linux code will largely build on Mac (but that said, I don't have a Mac to test with).

Based on LaChal's comments, are you suggesting re-writing the C/C++ LevelDB code in Python? That would certainly make it more portable, but that is a LOT of work! But this does beg the question of exactly what are you hoping in the final product here?

As I see it, there are two solutions --

Either way, it's an important question -- what is the desired outcome?

LaChal commented 9 years ago

An other solution is to use Python.h in the code source and write the Python itenrface in the library to obtain a .pyd object.

The interface to use is described in the Python documentation, in the 'Extending and Embedding' and 'Python/C API' topics (see https://docs.python.org/2.7/).

Khroki commented 9 years ago

Well, the second solution would be cleaner and more ideal, but anything you want to do is better than it is now, only thing is compiled builds can't really use external python libraries, only the ones built into the executable, so we would have to include it in all win/mac builds. Not a huge issue assuming compatibility is there. Also Mac isn't fully compatible with .so files as far as I'm aware. I believe the mac equivalent is .dylib, maybe someone who knows mac better can confirm. Also LaChal's solution may work too, up to you really.

TrazLander commented 9 years ago

You look into this anymore @Joeschmoh? More people always asking us for this: http://www.reddit.com/r/MCEdit/comments/31rdda/support_for_minecraft_pocket_edition_please/

Bananamon commented 9 years ago

I was told to bug you guys here. Lol :smile:

I think MCPE support is overdue. There are millions of people who would love to be able to edit their maps. I used to love MCEdit till 0.8.x update, till when it was possible.

I hope you guys will implement support for it ASAP. :+1:

Joeschmoh commented 9 years ago

This is still on my agenda! I apologize for not having anything yet, I'm behind where I wanted to be and I'll blame my work. I promised this at a time my job suddenly got really busy. I hear the message, and I want MCPE support too! I have 3 kids bugging me for it.

Bananamon commented 9 years ago

@Joeschmoh listen to your kids! :stuck_out_tongue_closed_eyes:

L3ice commented 9 years ago

@Joeschmoh Any news?

Bananamon commented 9 years ago

PING

Bananamon commented 9 years ago

I know I am spamming and I am sorry for that but MCPE support has been overdue. It's been almost 1.5 years since 0.9.0 update came out which introduced infinite worlds. We are missing out on too much

Rubisk commented 9 years ago

I started work! Sort of. Took me 3 days to compile mojangs leveldb and get boost setup, I just started wrinting the wrapper. After that's done we should be able to start working on the actual code. It's a long way...

Bananamon commented 9 years ago

YAAAAAAAAAAAAAAAYYYYYYYYYYYYY!!!!!!!!!!!!!!!!!!!!!!!!!!!!! :+1: :smile: