Ancurio / mkxp

Free Software implementation of the Ruby Game Scripting System (RGSS)
GNU General Public License v2.0
514 stars 131 forks source link

MKXP build for Android #56

Closed moocow1452 closed 9 years ago

moocow1452 commented 9 years ago

More of an issue tracker than anything else. I'll see if I can get a compile going, and report back on where I can get the code running and broken, and how I can otherwise put off actually getting things done. Anyone who cares to join me, get something running, or take credit for stuff already done can do so here.

moocow1452 commented 9 years ago

Well, remember when I said that would be working on a mobile port?

Here is what I got so far. Since MKXP using a ton of libraries, we're going to have to do a bit of work porting them into Android since it seems that at least Physfs doesn't have anyone who made inroads on it. Once we complete the libraries, we figure out a way to reference them in either CMake or Qmake to build a project proper with Android NDK. iOS, godspeed on that one.

Ancurio commented 9 years ago

Not sure if an iOS build makes sense, considering the iOS platform is incompatible with the GPL (unless you're specifically targeting jailbroken devices).

moocow1452 commented 9 years ago

Right. No point in making a port if it's going to illegal upon entry. Might be the sort of reason we could try and incorporate into the Retroarch framework, but again, those external libraries are going to be a pain in the ass.

AFAIK, the best bet for Android would be to reconstruct it from the SDL angle, since that seems to be the most developed, then build with CMake, as of here... http://wiki.libsdl.org/Android. I'll see if I can get on the boards for further help, but an autotool enviroment for Android seems like it would be a godsent for these kinds of projects.

Ancurio commented 9 years ago

Might be the sort of reason we could try and incorporate into the Retroarch framework

How would that help with iOS?

moocow1452 commented 9 years ago

RetroArch is a emulator framework for jail broken iOS among other platforms, and is more or less the golden standard for cross platform emulation , cores and framework being able to be built on damn near anything. I guess that's a separate issue and effort from a straight Android build, and we would run into the same issues as above, read as: Libraries are a bitch. Which leads back to that SDL link I posted earlier and think that may be our best bet for an Android build. On Oct 21, 2014 1:52 AM, "Jonas Kulla" notifications@github.com wrote:

Might be the sort of reason we could try and incorporate into the Retroarch framework

How would that help with iOS?

— Reply to this email directly or view it on GitHub https://github.com/Ancurio/mkxp/issues/56#issuecomment-59882288.

moocow1452 commented 9 years ago

Wanna know something fun? There is no pkg-config on Android, meaning you got to manually include directories one after another in the qmake project. I managed to scavenger hunt Android compatible libs and add all the .h files and what have you, but am stuck on RGSS references, and will see what I can do for Ruby on Android when I get another chance. On Oct 21, 2014 8:33 AM, "Joey Carlini" moocow1452@gmail.com wrote:

RetroArch is a emulator framework for jail broken iOS among other platforms, and is more or less the golden standard for cross platform emulation , cores and framework being able to be built on damn near anything. I guess that's a separate issue and effort from a straight Android build, and we would run into the same issues as above, read as: Libraries are a bitch. Which leads back to that SDL link I posted earlier and think that may be our best bet for an Android build. On Oct 21, 2014 1:52 AM, "Jonas Kulla" notifications@github.com wrote:

Might be the sort of reason we could try and incorporate into the Retroarch framework

How would that help with iOS?

— Reply to this email directly or view it on GitHub https://github.com/Ancurio/mkxp/issues/56#issuecomment-59882288.

Ancurio commented 9 years ago

qmake has support for the NDK?

moocow1452 commented 9 years ago

Qt Creator natively has the ability to take the NDK/SDK and pull toolchains out to compile an Android APK to the best of it's ability.

http://qt-project.org/doc/qt-5/deployment-android.html

It is anything but pretty, but it should work just fine for what we want it to do. On Oct 27, 2014 2:51 PM, "Jonas Kulla" notifications@github.com wrote:

qmake has support for the NDK?

— Reply to this email directly or view it on GitHub https://github.com/Ancurio/mkxp/issues/56#issuecomment-60648642.

Ancurio commented 9 years ago

Heh, that's interesting =) I would probably just end up writing a custom Makefile.

moocow1452 commented 9 years ago

Cool beans. Can't wait to see it. On Oct 27, 2014 5:10 PM, "Jonas Kulla" notifications@github.com wrote:

Heh, that's interesting =) I would probably just end up writing a custom Makefile.

— Reply to this email directly or view it on GitHub https://github.com/Ancurio/mkxp/issues/56#issuecomment-60669857.

moocow1452 commented 9 years ago

RGSS fun I mentioned earlier...

00:33:57: Running steps for project mkxp...
00:33:57: Configuration unchanged, skipping qmake step.
00:33:58: Starting: "/usr/bin/make" 
/opt/Qt/5.3/android_armv7/bin/qmake -spec android-g++ CONFIG+=debug -o Makefile ../mkxp/mkxp.pro
/home/user/tools/android/android-ndk-r10c/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -c -Wno-psabi -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -ffunction-sections -funwind-tables -fstack-protector -fno-short-enums -DANDROID -Wa,--noexecstack -std=gnu++0x -g -g -gdwarf-2 -marm -O0 -fno-omit-frame-pointer -Wall -Wno-psabi -W -D_REENTRANT -fPIE -DBINDING_MRI -I/opt/Qt/5.3/android_armv7/mkspecs/android-g++ -I../mkxp -I../mkxp -I../mkxp/src -I../mkxp-android/openal-soft/jni/OpenAL -I../mkxp-android/SDL/include -I../mkxp-android/SDL_image -I../mkxp-android/SDL_ttf -I../mkxp-android/adosbox-read-only/jni/sdl_sound/include -I../mkxp-android/libsigcpp-android/jni -I../mkxp-android/Boost-for-Android/boost_1_53_0 -I../mkxp-android/android-cairo/jni/pixman/pixman -I../mkxp-android/android-cairo/jni/pixman-extra -I../mkxp-android/libphysfs-android/jni -I../../tools/android/android-ndk-r10c/sources/cxx-stl/gnu-libstdc++/4.9/include -I../../tools/android/android-ndk-r10c/sources/cxx-stl/gnu-libstdc++/4.9/libs/armeabi-v7a/include -I../../tools/android/android-ndk-r10c/platforms/android-9/arch-arm/usr/include -I. -o filesystem.o ../mkxp/src/filesystem.cpp
In file included from ../mkxp/src/filesystem.cpp:24:0:
../mkxp/src/rgssad.h:27:14: error: 'PHYSFS_Archiver' does not name a type
 extern const PHYSFS_Archiver RGSS1_Archiver;
              ^
../mkxp/src/rgssad.h:28:14: error: 'PHYSFS_Archiver' does not name a type
 extern const PHYSFS_Archiver RGSS2_Archiver;
              ^
../mkxp/src/rgssad.h:29:14: error: 'PHYSFS_Archiver' does not name a type
 extern const PHYSFS_Archiver RGSS3_Archiver;
              ^
../mkxp/src/filesystem.cpp: In function 'size_t SDL_RWopsRead(SDL_RWops*, void*, size_t, size_t)':
../mkxp/src/filesystem.cpp:95:64: error: 'PHYSFS_readBytes' was not declared in this scope
  PHYSFS_sint64 result = PHYSFS_readBytes(f, buffer, size*maxnum);
                                                                ^
../mkxp/src/filesystem.cpp: In function 'size_t SDL_RWopsWrite(SDL_RWops*, const void*, size_t, size_t)':
../mkxp/src/filesystem.cpp:107:62: error: 'PHYSFS_writeBytes' was not declared in this scope
  PHYSFS_sint64 result = PHYSFS_writeBytes(f, buffer, size*num);
                                                              ^
../mkxp/src/filesystem.cpp: In constructor 'FileSystem::FileSystem(const char*, bool)':
../mkxp/src/filesystem.cpp:353:27: error: 'RGSS1_Archiver' was not declared in this scope
  PHYSFS_registerArchiver(&RGSS1_Archiver);
                           ^
../mkxp/src/filesystem.cpp:353:41: error: 'PHYSFS_registerArchiver' was not declared in this scope
  PHYSFS_registerArchiver(&RGSS1_Archiver);
                                         ^
../mkxp/src/filesystem.cpp:354:27: error: 'RGSS2_Archiver' was not declared in this scope
  PHYSFS_registerArchiver(&RGSS2_Archiver);
                           ^
../mkxp/src/filesystem.cpp:355:27: error: 'RGSS3_Archiver' was not declared in this scope
  PHYSFS_registerArchiver(&RGSS3_Archiver);
                           ^
make: *** [filesystem.o] Error 1
00:34:22: The process "/usr/bin/make" exited with code 2.
Error while building/deploying project mkxp (kit: Android for armeabi-v7a (GCC 4.9, Qt 5.3.2))
When executing step "Make"
00:34:22: Elapsed time: 00:25.

Same thing I'd imagine with no Ruby in the first place.

Ancurio commented 9 years ago

error: 'PHYSFS_Archiver' does not name a type

Are you sure you have latest hg physfs?

moocow1452 commented 9 years ago

No, it's the one specificly made for Android. Dummy.

EDIT: I'm the dummy. Hang on... I'll get a new one cooking.

moocow1452 commented 9 years ago

Okay, gonna give up for the night, got the dreaded "No rule to make target shader/transSimple.frag', needed bysprite.frag.xxd'. Stop." Which is about where I get with the latest x86 builds as well, so I guess that's a good sign.

moocow1452 commented 9 years ago

Here is what I got off the SDL mail group. Thoughts?

••• Hi Joey,I'll share what I know, but at the end of the day, Android is really,really painful.I built SDL_sound for Android several years ago. I did not use theCMake build system, but rewrote the build process by hand in anAndroid.mk because at the time, it was the easiest way to build itinto the product I was working on at the time because they had amonolithic Android.mk. (This was Android 1.6 days, so the toolchainsucked even more back then.)The hardest part were the dependencies, like mpg123. I recall x86assembly code in that library so figuring out how to reinvent thebuild system and deal with arm was a pain.Fast forward a good number of years, and thanks to 3rd party CMaketoolchains, CMake is useable for Android. However, all the SDL_sounddependencies don't necessarily have a CMake build system, so that willbe a pain.Honestly, I find build systems much harder/complicated than code alarge deal of the time. So I'm actually not using SDL_sound anymore,but rolled a native Android OpenSL ES decoder into my audio libraryALmixer (which used to solely rely on SDL_sound).So you will need to figure out how to build the SDL_sound dependenciesand SDL_sound itself. I think SDL_sound is the easy part.I now have the CMake build process for Android down to a point whereI've been able to document it and repeat it for multiple projects.There are still a bunch of hoops you have to jump through. (Thebiggest are setting up the standalone toolchain and settingenvironmental variables.) But I find it much better than using theAndroid.mk/ndk-build system.I setup an example with ALmixer here so other people could learn from it. https://bitbucket.org/ewing/hello-android-almixerAdditionally, OpenAL-Soft is a dependency that ALmixer depends on,though this uses its own supplied Makefile system. This is actually agood example of how to get CMake to find 3rd party dependencies onAndroid.Once these components are built as dynamic libraries, the final HelloWorld app build process relies on the official NDK external module system tofind ALmixer and OpenAL and build appropriately.Because there are a lot of disjoint steps, all of this is strungtogether by some Perl scripts.The README.txt contains a lot of information on how to setup things.Android NDK development is really painful so take your time on this.Don't expect to be able to grasp everything in 10 minutes. That said,I think this example is one of the smoothest examples ever made ofbuilding and using complicated 3rd party dependencies on Android withthe NDK.I also built JavaScriptCore, which is one of the hardest, mostcomplicated projects I've ever had to build. I documented it here andmight be used to cross-reference what I did with ALmixer. https://github.com/appcelerator/hyperloop/wiki/Building-JavaScriptCore-for-AndroidAnyway, I hope some of this might be of some help.Thanks,EricP.S. I love To the Moon, and just played A Bird Story. Beautiful.--Beginning iPhone Games Developmenthttp://playcontrol.net/iphonegamebook/

Ancurio commented 9 years ago

Started working on this myself because I was bored. My system: MT6572 Dual Core 1.3GHz CPU with Mali-400 graphics

Things I've set up so far:

Things that I'm working on / that definitely need attention:

Furthermore, considering APKs are limited to 50 MB by default, dealing with games bigger than that is a curious exercise I have no interest in as of now.

fdelapena commented 9 years ago

Furthermore, considering APKs are limited to 50 MB by default, dealing with games bigger than that is a curious exercise I have no interest in as of now.

When needed: http://developer.android.com/google/play/expansion-files.html

Ancurio commented 9 years ago

When needed: http://developer.android.com/google/play/expansion-files.html

Yeah, I've studied that process already. The concept of "we might download all expansions for you, but we also might not and you'll have to do it yourself" sounds really annoying. I'm not too keen on digging into anything Java related at the moment.

But thanks for linking it.

Ancurio commented 9 years ago

Added point about wrapping SDL_RWops.

Ancurio commented 9 years ago

Good news, by simply moving the GL setup and drawing into a separate thread in my small test program, I was able to reproduce the same exact context restoration problem, so it's not something inherently related to what mkxp does.

For reference, after the app re-enters the foreground, the log is spammed with

E/BufferQueue(  137): [SurfaceView](this:0x4142f5d8,api:1) query: SurfaceTexture has been abandoned!

Edit: For reference, a minimal C app that does threaded GL and causes this: http://pastebin.com/yzNAVK47

moocow1452 commented 9 years ago

Can you package the app as a strict interpreter, and if people want to release their own games in the Google Play Store, they can worry about size requirements?

On Thu Dec 04 2014 at 1:25:31 PM Jonas Kulla notifications@github.com wrote:

When needed: http://developer.android.com/google/play/expansion-files.html

Yeah, I've studied that technique already. The concept of "we might download all expansions for you, but we also might not and you'll have to do it yourself" sounds really annoying. I'm not too keen on digging into anything Java related at the moment.

— Reply to this email directly or view it on GitHub https://github.com/Ancurio/mkxp/issues/56#issuecomment-65678023.

Ancurio commented 9 years ago

Ayy, I just had a crazy idea, and it did indeed solve the context restoration problem. I remembered that SDL at some point during the move-to-background phase clears (nulls) the current context in order to save it, here. But that doesn't make any sense for mkxp because no EGL context is ever current on the main thread, as our context's entire lifecycle is contained to the RGSS thread.

So I tried replicating this behavior by clearing the context on the RGSS thread before sleeping the thread, and making the context current again after waking up, and voila, it works, yay! Now I just have to investigate when exactly SDL makes the context current on the main thread so we don't end up in a situation where it is snatched from the RGSS thread after wakeup.

Ancurio commented 9 years ago

Can you package the app as a strict interpreter, and if people want to release their own games in the Google Play Store, they can worry about size requirements?

Hmm, not sure what you mean with "strict interpreter". What about mkxp is currently not a "strict interpreter"?

Also, I don't think you understand the problem. Apps bigger than 50 MB need special handling and store their files in a different place (external storage, SD cards and such). mkxp can't access these files if it doesn't know about them or where they are. Also the app would have to ensure that everything is downloaded and if not, manually download the rest. There is coding involved no matter what.

Edit: Of course, if the game fits below 50 MB with everything else, it should be as easy as putting all game files into the asset folder and have it work. But users would still have to use ant to build the final apk.

moocow1452 commented 9 years ago

K, forgot that we weren't just dealing with an emulator. Carry on.

On Fri Dec 05 2014 at 3:12:25 AM Jonas Kulla notifications@github.com wrote:

Can you package the app as a strict interpreter, and if people want to release their own games in the Google Play Store, they can worry about size requirements?

Hmm, not sure what you mean with "strict interpreter". What about mkxp is currently not a "strict interpreter"?

Also, I don't think you understand the problem. Apps bigger than 50 MB need special handling and store their files in a different place (external storage, SD cards and such). mkxp can't access these files if it doesn't know about them or where they are. Also the app would have to ensure that everything is downloaded and if not, manually download the rest. There is coding involved no matter what.

— Reply to this email directly or view it on GitHub https://github.com/Ancurio/mkxp/issues/56#issuecomment-65759071.

Ancurio commented 9 years ago

Actually an interesting idea just hit me. I could zip up the entire game folder, put it into assets/, open that zip using SDL_RWops, wrap that handle in a PhysFS_Io* and then mount it. The zip file would not get further compressed by ant, meaning read/seek operations on it would map directly to raw fd calls.

Ancurio commented 9 years ago

Fixed a few SDL bugs in the process:

2797 2802 2808

moocow1452 commented 9 years ago

K, for the sake of not letting this one die. I'll just report back and say that Ancurio does have a working build for MKXP for Android, and ported TtM's Holiday Special as a proof of concept. APK's are renamed zip files, so you could in theory put a new game in there, but I guess he wants to clean it up before TtM hits the Play Store or the Humble Android app.

Ancurio commented 9 years ago

I've been working with "The Mirror Lied" for the most part, but since most games that work on desktop mkxp would work on the mobile build, I also put in the holiday special since unlike TML it crashes my 256MB phone due to low memory (TML crashes too, but only very close to finishing the game). I was planning on releasing a TML Android version sometime before Christmas, but when I tested the game on a friend's phone which had a Vivante GPU, the graphics driver was completely broken, you couldn't see the tilemap and text rendering would create garbage data. I have no idea what's causing it and I can't borrow my friends phone long enough to even attempt to debug it.

So that's the point where I kinda said "screw it, I'd have to work around all 5 or so major GPU vendors and their broken as *\ drivers", and laid down work on this for now. The GPU on my phone, a Mali-400, does have a glitch too, but it was similar to one I previously encountered on desktop GPU drivers so it was easier to work around it. I'm still in the process of sorting out all the changes I made while working on this into individual patches (around 20 or so) since they're very generic and will then hopefully push them to github, but that's about it. I also don't think I'll push the code for the touch input overlay since it's not screen size independent and pretty much only works on smaller handheld-sized phones (on bigger ones it would be upscaled and too large for fingers I think, but not sure, definitely won't work with tablets).

APK's are renamed zip files, so you could in theory put a new game in there

Yeah pretty much, as long as you don't go over 50MB. You'd need to re-align the zip though (the SDK has a tool for that). If you manually put a bigger game on an sd card and then point to it in mkxp.conf it should work too, although I haven't personally tested that.

TtM hits the Play Store or the Humble Android app.

haha!

moocow1452 commented 9 years ago

Worth an ask. Might be time to take this to XDA if I get a spare moment. Wider selection of devices to black and white list, and someone might know something about SDL not working with some devices. More heads and that jazz.

Ancurio commented 9 years ago

It's not that SDL doesn't work on devices (I doubt it honestly), it's just that mobile graphics drivers are %$§!. I've been told this so many times on the internet and didn't want to believe it, but it's actually true. If XDA has a big database of "things that are broken" sorted per OEM, that would help a lot actually.

Ancurio commented 9 years ago

@moocow1452 Just a correction: the build I sent you can't run games on sdcards, I just fixed that locally.

xperia64 commented 9 years ago

In terms of game data, I would recommend doing what this app does: https://play.google.com/store/apps/details?id=net.kernys.rgss

Have a folder on /sdcard or somewhere where users can place games, and provide a simple menu to switch between them. If RPGMaker developers want to port their games officially, they can deal with the headaches of .obb files themselves. Also, if you need testing on more GPU's, I have some PowerVR's, Adrenos, a Mali-400MP, and a Tegra K1 available. The K1 does support full OpenGL, but it appears only Dolphin Emulator has figured out how to get it to work since there is no libGL in the ndk.

moocow1452 commented 9 years ago

If I wanted to bang some drums to try and get help and/or ask for testers or Reddit, XDA, or anywhere else we all can think of, do you have a premier test apk that can be distributed through FDroid or a Dropbox link that you are totally okay with having go out into the wild and representing the work you've done thus far? I'll have some time this weekend, and think I could get a crowd riled up if we wanted to make a compatiblity push.

moocow1452 commented 9 years ago

Also, I have a Moto G I can play with if we need something with more juice.

moocow1452 commented 9 years ago

@xperia64 This is the version I got from @Ancurio over Twitter 5 weeks ago. So far it's, Mali Good, Vivante and Adreno need work. https://www.dropbox.com/s/v7cjdh91cln5m2m/mkxp-debug.apk

moocow1452 commented 9 years ago

PS: Ancurio may get his own version to you soon enough.

xperia64 commented 9 years ago

It's excellent on the Tegra K1. Only issue is if you press the back button and relaunch the app, it crashes. I'll be waiting for a universal build that can play games on the sdcard (the game I have in mind is compiled with mkxp for Linux and Mac so it should work)

pulsejet commented 9 years ago

Can someone please give a step to step guide, right from the ide used, to compile mkxp for android?

pulsejet commented 9 years ago

Forgot to mention that it is working excellent on a mtk 6589 phone

moocow1452 commented 9 years ago

That makes two of us.

The complicated part is that all of the Ruby, SDL and other libraries need to be rebuilt for Android. I think that you can rename the APK to a .zip and extract all your libraries out to build another copy, but that's not the "true FLOSS way." (TM) Whether you were willing to put in the effort to scavenge the interwebs, or be a dirty cheat, you could modify the stock MKXP.pro file to point to the Android libraries, and use QT creator and the Android SDK and NDK to build your own iteration of the app on your own. I can get more detailed instructions out when I have time to, and if Ancurio has more officially official instructions for an Android build, those obviously take precedence.

On Mon, Jan 26, 2015, 9:51 AM radialapps notifications@github.com wrote:

Forgot to mention that it is working excellent on a mtk 6589 phone

— Reply to this email directly or view it on GitHub https://github.com/Ancurio/mkxp/issues/56#issuecomment-71471693.

pulsejet commented 9 years ago

I tried to replace the sigcorp.zip in the apk. However, the other game never runs. I also tried using the original scripts, and the scripts from the game already there but no success. By the way, I get a small graphical glitch when a text box is displayed but it is not important. Otherwise, it works awesome.

Ancurio commented 9 years ago

@radialapps Do you know your way around autotools / GNU binutils? Otherwise it'll be very complicated for you to build mkxp. I can tar up my local prefix containing all the dependencies so you'd only have to run the build script, but if all you want is the finished binaries (ie. you don't intend to hack on the source code), then I'd just recommend to take them from my build (contains both standard armeabi and armeabi-v7a binaries).

To run your own game, you have to edit mkxp.conf in the assets folder and set the RTP= entry to point at your game zip.

pulsejet commented 9 years ago

@Ancurio, If it is too complicated, I would rather prefer the compiled version. However, the game I was testing (Knight blade) did not work when I modified mkxp.conf... Here's what I did: 1) Zip the game. 2) Add it to the APK 3) Direct mkxp.conf to it. 4) Sign the Apk. I get a blank black screen for a long time before it crashes. Did I miss anything?

It would be good if someone made an android project for it. (Even better if it is in a complete package that can be compiled even in small IDEs like AIDE)

Ancurio commented 9 years ago

@radialapps Knight Blade requires both the RMXP RTP and midi audio support, so it's a bit unfit for testing (the RTP is less of a problem, but I haven't looked into how to do midi on android yet, not to mention that it's rather computationally intensive).

It would be good if someone made an android project for it. (Even better if it is in a complete package that can be compiled even in small IDEs like AIDE)

What are the benefits from this?

xperia64 commented 9 years ago

In the event you were to add midi support, one of my other projects happens to be a port of TiMidity++ to android in the form of a music player so there's that.

Ancurio commented 9 years ago

@xperia64 Thanks =) But mkxp already uses fluidsynth for midi rendering.

pulsejet commented 9 years ago

@Ancurio, I forgot to mention that I had copied the entire required rtp and removed all midi files before trying out. I also tried out with a new blank project but no luck...

As for the project, that would make direct device testing much easier and also make experimentation easy.

Ancurio commented 9 years ago

@radialapps And this works on desktop mkxp yes? Is there anything suspicious in logcat when the app hangs and crashes?

Regarding the IDE project, can you give some specific examples? What is "direct device testing", and how does it differ from using ant/adb? Does it include things like attaching a debugger to a running instance?

Ancurio commented 9 years ago

@radialapps Actually, just send me the apk that you've created and which doesn't work, and I'll debug it from there.

moocow1452 commented 9 years ago

I think that there is a debugger available on Android called GROPG. Not sure how useful it is to on device building, but it's trouble shooting without a PC, which is nice.

http://cseweb.uta.edu/~tuan/GROPG/