Go-NoGo + architecture changes for support of apktool

[mougino]

Hello to all, I have been successfuly investigating a brand new direction that would allow to compile a BASIC! program into an APK either 1) from a computer much more easily than the current AppBuilder (no need of JDK+Android SDK etc.) and/or 2) directly from within BASIC! on the Android device. This direction is called apktool: https://code.google.com/p/android-apktool/ https://code.google.com/p/apktool/

There are however some cons, it's not all pros:

• the principle of apktool is to make changes to an existing APK without altering it (it still can be installed after the changes). This means that the keystore and signature would not be touched. In other words, BASIC! APKs produced with apktool will be signed by Paul's BASIC! key. I don't know the ramifications in terms of Google Play store but it is worth mentioning
• the Android version would need to download a 60MB zip onto the sdcard at first build, this could be done via WiFi only but it has significant end user impact to consider if it's a viable solution
• in order for both solutions (PC and cellphone) to work the current Basic.java would have to be modified, because it works better to only decompile the /res and not the /src then replace everything in /res to match user program then recompile we would have to export the following values into an xml:

    public static String AppPath = "rfo-basic";             // Set to the path name for application directories

    public static boolean isAPK = false;                    // If building APK, set true

    public static boolean apkCreateDataDir = true;          // If APK needs a /data/ directory, set true

    public static boolean apkCreateDataBaseDir = true;      // If APK needs a /database/ director, set true

What do you guys think? Does this sound promising enough to continue investigating or current solution is good enough?

[jMarcS]

That's right, Stefano. I have tried it with two .bas files (my_program and one INCLUDE), but it should work with more. The old way is comfortable because you see exactly what you are getting. The new way you have to trust the loader. Since I am still working on it, it may not be all that trustworthy! Still, I think you will find it is easier than the old way. Especially if you can just put the .bas files in AppBuilder's resource window. Please try it -- it will be a valuable test of my changes. Real users do things developers never think of.

Hi Nicolas. Yes, I see what you mean about security. I think it would be safe if an APK is allowed only to RUN programs in resources. It's not really extra work, it work I missed before! I forgot all about AutoRun.java when I did the others, because it has no reference to resources. But it has a BufferedReader, so I have to visit it anyway.

I will have to be a little careful, though. AutoRun.java is a kind of a specialized mini-Basic.java. It will be tempting to try to pull it back into Basic.java, but I don't think it would be good to take that time now.

[jMarcS]

I had a lot of trouble getting AutoRun, LoadFile, and Editor to share common utilities. After that it was a very small change to let an APK RUN another program in res/raw. I put in a check to make sure an APK can not RUN a program file in the file system.

[jMarcS]

I did not intentionally close this.

[stefano-px]

Hello Marc.

A little request about /res/values/setup.xml. Could you edit the note about string-array name="load_file_names" ?

It's not clear enough for non developers how to add files in the list. I suppose the way is:

myfilename

without quoting myfilename. Can you confirm and add a little sample in the note?

Thanks.

[stefano-px]

Hello Marc. Me again.

Another little doubt (I just downloaded the latest master zip from github and looking into changes for building APK - sorry for stupid questions)

In setup.xml I see:

name="my_program">my_program<

refering to assets/rfo-basic/source/my_program.bas

I know android strips extensions in file name, but to be sure I'm asking: is it ok the value my_program, or I have to rename my_program.bas to my_program? Thanks.

[jMarcS]

Hi Stefano. I intended to explain the file list in Appendix D of the manual. But I think you are right; an example in setup.xml is helpful.

About "my_program" -- that's a bug, thank you for catching it. As a resource (res/raw), "my_program" was correct, because the extension is stripped from a resource. As an asset, the name must be exactly the same as the asset name.

I am trying to figure out how to write Appendix D to describe using assets. I put the sample program in "assets/rfo-basic/source/my_program.bas", not "assets/rfo-basic/source/Sample_Programs/my_program.bas" because I thought it would be easier when changing to an APK. But there is no place to put the sample program where both standard BASIC! and an APK can find it.

After I figure out what to say in Appendix D, I will fix setup.xml as you suggest.

[stefano-px]

Thanks Marc. I agree that "assets/rfo-basic/source/my_program.bas" is the best location when changing to an APK.

[mougino]

Marc, just to confirm: if one changes the <string name="my_program"> content in the new setup.xml for, say, hello.bas, then BASIC! would load hello.bas at startup and not my_program.bas, right?

[jMarcS]

Yes. It would look for assets/<my-app>/source/hello.bas. At least that was the idea.

But, as I used to tell my intern, "If it ain't been tested, it don't work." (Yes, the atrocious grammar is intentional!)

[mougino]

I will be able to test it in the next hours, just finished the beta version of the apktool-based AppBuilder, it can decompile make changes and compile but so far APKs refuse to install (I think it's a certificate issue, working on it), when I solve this, I can create an APK from the most recent BASIC! code in a matter of seconds.

[jMarcS]

Woo-hoo!

[mougino]

Let us woo-hoo when it's done, shall we? ;-) Like we say in French il ne faut pas vendre la peau de l'ours avant de l'avoir tué... Changing the certificate didn't do it (not an easy task btw, needed a PEM+PKCS8 instead of a standard Java KeyStore). Marc, you have an Android emulator, if I send you my helloworld.apk can you try to install it and have the details of the reason(s) why installation fails?

[mougino]

FYI apparently you can get more info from the Logcat although it's huge, I'm trying to investigate via an adb logcat ActivityManager:E -v long > logcat.txt

[jMarcS]

The bear might have an opinion on the subject of its pelt?

Nicolas, normally I would love to help with the debugging -- I like it WAY better than writing documentation! But I have spent the entire morning on Appendix D and I am out of time. I'm sorry I did not look sooner, perhaps the manual could have waited.

Go ahead and attach it to an email. I will check again later, but this may be all for the weekend. I have to learn some more music for a performance tomorrow and then we are going out of town for a couple of days.

Meanwhile, I have a draft of Appendix D for review. How do I make it available? I will start by emailing a copy to Mike Leavitt; he will want to rework it before putting it in the git repository, but maybe he can check it into a branch?

[mougino]

Send it to me too, would like to review it (always funny to read Appendix D and compare it to AppBuilder code ^_^). I didn't know you were playing music? What instrument do you play? I used to play the piano/synthesizer but many years have passed since I practiced.

[jMarcS]

This is a draft of De-Re-Basic Appendix D, modified for all the changes Nicolas and I have been making. I'm sending it as a reply to GitHub, so I don't know who will get it.

Mike, I especially need to get this to you. Are you out there?

If I am flooding a lot of mail boxes with a big file, I apologize; I'm learning the hard way...

[stefano-px]

Hello

New questions here. Subject: make an APK.

In 1.76 we have /res/raw. When creating the APK we just put all files needed by App in /res/raw and edit my_program accordingly. I took a look in last Basic-master from github and I saw the new assets phlosophy. I know everything will be explained in the guide, but in meantime I can try to modify my linux GUI for next 1.77 version. My questions are:

• When building an APK, can I delete assets/rfo-basic/source/Sample_Programs/ ?
• Can I delete all NOT needed files in assets/rfo-basic/data as I'm doing now with /res/raw/ ?
• When building an APK, user's files must be copied in assets/rfo-basic/data? Right?
• And: if user have a database in its App files, we need to make a assets/rfo-basic/databases and put the database in there? Does every user's file go in assets/rfo-basic/data, included databases?

Finally, line 799 in Basic.java shows you were thinking/talking/listening to a girl/woman in that moment... ;)

Thanks for attention.

[jMarcS]

Did you get the file in email? Sorry, I don't know what I'm doing.

I dabbled in a few instruments when I was much younger, my best and favorite was the Appalachian mountain dulcimer (not the same as the trapezoidal hammered dulcimer!). But now I'm singing. I was in a choir for almost 20 years before I left Minnesota almost 20 years ago. After a long break, I've joined a small chorale called Jubilate!. I am very excited about it; fun group and I love the music, but one loses one's edge after 20 years...

[stefano-px]

Sorry, I did not read about new Appndix D. Is it possible to have a copy and read it?

[mougino]

Hello Stefano, Marc might be busy in "real life" so let me answer:

• yes you can safely delete assets/rfo-basic/source/Sample_Programs/ (in fact you can safely delete all assets/rfo-basic/ folder)
• when building an APK user's files must be copied in assets/<app_path>/data: the folder must correspond to the app_path value in the new res/values/setup.xml
• database file (*.db) must go in assets/<app_path>/databases

[stefano-px]

Thank you, NIcolas. It's a big change from 1.76, but now is clear! I suppose, just to confirm, that user's program must be copied in /assets//source/my_program.bas. Right? And a little compulsive question: if a user has the bad idea to name its database wiithout extension *.db, still goes to databases directory?

[jMarcS]

Yup, exactly what I was typing when your response arrived, Nicolas. Thank you. Stefano, I get all of these GitHub posts in email, so I replied to that email with a copy of Appendix D. I think it got to Nicolas that way, but if you don't get the emails, I don't know how to get you the file.

And yes, line 799 refers to "she" :{)} That is a convention Paul uses a lot, to say "she" for generic "he or she". There was quite a commotion in the U.S. a decade or two ago about what pronoun to use. Somebody, maybe a science fiction author, coined the word "yx" for a non-gender-specific pronoun, but I don't think anybody paid any attention!

[jMarcS]

It doesn't matter what you call it. The SQL.OPEN command takes a file name argument. It looks in the "databases" directory for that file.

The automatic routing of files with ".db" to databases doesn't happen with assets. You put files in the directory where you want them to go. You just have to know where the commands are going to look for them.

[EDIT: That is not quite correct. If you put a file in the "load_file_names" array, you do not specify the path. A file named "meow.wav" is copied from assets/<app_path>/data/meow.wav to <base_drive>/<app_path/data/meow.wav, but a file named "stats.db" is coped from assets/<app_path>/databases/stats.db to <base_drive>/<app_path/databases/stats.db. So to load files at program start, the automatic routing for the ".db" extension does happen.]

[mougino]

@stefano-px copy user's program to assets/<app_path>/source/and either rename it my_program.bas or keep its original name and copy said original name as the value inside tag <string name="my_program> of the new setup.xml

@jMarcS replying from email to a Github issue drops the attachment so we didn't get your Appendix D. I sent you a mail to what I hope is your email address to tell us to send it individually to our own email addresses. Else you can put it on Paul's FTP maybe and tell us where to get it?

[stefano-px]

Thanks a lot Marc and Nicolas!

[jMarcS]

Ah... if you list files in "load_file_names", then then files that end in ".db" are supposed to be copied to the databases directory on the SDCARD. But I don't remember if I implemented it that way. [EDIT: I had not done it right. I fixed it on January 23.]

Nicolas, your backticks didn't work. Stefano, the tag Nicolas is referring to is <string name="my_program"> in setup.xml. The name in the "my_program" string and the file in assets/<app-name>/source must match

assets/my_app/source/main_program.bas
<string name="my_program">main_program.bas</string>

[EDIT: fixed the path!]

[stefano-px]

Yes Marc, we talked about my_program string some hours ago.

I didn't receive Appendix D. I don't care about spam at this address: rospolosco AT tiscali DOT it

If someone could send me a copy, I will appreciate. Thanks.

[mougino]

You can send to me too at mougino AT free DOT fr

[jMarcS]

Coming right up!

[stefano-px]

Thanks Marc. Received Appendix D.

[mougino]

Yes, me too, well received. I've made promising progress in APK hacking in the last weeks, I might be able (once I identify the current problem why recompiled APKs don't install) to allow compiling from within the Android device itself... You know what it would mean if I succeed... No more Appendix D!!

[mougino]

Guys, just to let you know that I solved the install problem after a whole night of insomnia (2am - 8am >_<) the apktool version of AppBuilder is up and running, now in beta stage (not a release candidate yet). My first tests with the new architecture work ok (decompile-tune-recompile-install-run) with simple BASIC! programs. After my nap I will test it against my RPG and other complex programs.

[stefano-px]

Hello Marc.

I've a problem with "load_file_names" in setup.xml. It seems that the generated APK does not load files in that list. The APK is the simple kitty example. APK is installed correctly on my device and it runs well. My setup.xml looks like this:

database.db meow.wav

Just to be precise: I know kitty example does not need meow.wav to be loaded, but I'm testing my GUI to ant and android tools. As you see I added thinking that was the problem, but it wasn't. Any clue? files are stored in assets/app_path/data and assets/app_path/databases. More info: nor even the message "Standby for initial..." is shown. And: I completely deleted assets/rfo-basic. Thanks for attention.

[mougino]

On my side, my RPG compiles and installs fine with latest BASIC! code and all resources in assets but it hangs at startup after choosing the language, the problem comes from the creation of the preference file: Text.open w, tmp, "../prefs.txt" maybe the function tries to create for write a text file in the assets?? Should I open a specific issue Marc?

[mougino]

Oops, found where the problems comes from, my bad: I accidentaly set apk_create_data_dir to false, of course it needs to be set to true for BASIC! to be able to access files on the SD-Card.

After changing, next error is strange, app hangs when trying to load a soundpool from a file name with a different case from specified: the Linux file is "Blesse.mp3" with a capital B but my program does a soundpool.load snd, "blesse.mp3" (small b) while it works correctly in standard BASIC! it doesn't work in APK from assets... I will have a look on the code.

[mougino]

@stefano-px Reviewing Marc's code in Basic.LoadGraphics() and Basic.copyBinaryAssetToFile() I think you need to list your files as following:

<string-array name="load_file_names">
    <item>databases/database.db</item>
    <item>data/meow.wav</item>
</string-array>

And no need to add the last empty <item></item> anymore, Marc made change so that it is well handled without it.

[stefano-px]

Not my case. Data dir and databases dir are created. I'm investigating. Nicolas, I need your little help for console colors. I thought that changing value in xml/setting.xml from BW to WB make console White text on Black screen, but nothing happens. So I swapped color1 and color2 values in setup.xml and finally have my black console. Is it normal? The value in settings doesn't effewct the APK? I have to manage color1 and color2 ? It's strange.

[stefano-px]

Nicolas, I read now your suggestion. Going to test now. Thanks.

[mougino]

@stefano-px about colors yes I have had the same experience: settings.xml's BW and WB correspond to menu entries in the BASIC! Editor (menu > Preferences > Screen Colors) so swapping them will affect the menu, not the colors :-) Indeed the good way is to change the 3 colors in setup.xml

[stefano-px]

Bravo Nicolas! You were right about load_file_names. I made changes you suggested and now file are copied at startup with initial message.

[mougino]

@stefano-px will you be interested in the apktool-way to build an APK? apktool and the other tools I use now are cross platform, there are Linux binaries available. I can share my source code of Quick APK: in order first to try to replicate its behavior with a script, then to embed everything in a nice GUI similar to Quick APK made with Gambas ;-) Le 19 janv. 2014 16:40, "stefano-px" notifications@github.com a écrit :

Bravo! You were right about load_file_names. I made changes you suggested and now file are copied at startup with initial message.

— Reply to this email directly or view it on GitHubhttps://github.com/RFO-BASIC/Basic/issues/118#issuecomment-32711039 .

[stefano-px]

Thanks, but not for the moment ;) I have a curiosity about your new apktool: did you solve the problem about Google Play Store and Paul sign?

[mougino]

Yes, I am able to sign with whatever certificate I want, or if user doesn't has one, with a default one different from Paul's.

[stefano-px]

Great! Congrats!

[mougino]

@jMarcS After investigation I confirm there could be a problem for the users: all BASIC! commands in standard BASIC! that open or load a file are case insensitive because said files are located in /sdcard which is a FAT file system thus is case insensitive, BUT when building their APK, the /asset file system is case sensitive thus if calling a different case version of the filename, the APK will throw an error (while the program in editor did not). Source: http://stackoverflow.com/questions/6502712/isnt-android-file-exists-case-sensitive

Do you have a recommendation of how to make BASIC! users aware of that?

[mougino]

Going on with my tests, renaming all calls to resources in my RPG was a big pain in the app but now it runs! The behaviour of the app is rather strange thought, compared to its counterpart compiled with previous version of BASIC! and resources in res/raw... The speed first: some parts are way faster than usual (the intro scene for example, or writing the map to a big byte file when creating a new game), I don't know why.. The antialias fonts and some colors then: they look really different, while the code is strictly the same. Did you touch to these commands between v01.76? That is very strange...

[stefano-px]

Hi all. Found a bug (?). INCLUDE does not work when building an APK with 1.77. Then I tried to build an APK with 1.76 and the old way to paste all source files into the main .bas file files worked. So, back with 1.77 and tried the new method: copying all .bas file into assets/app_path/source. I got an error again. Exactly: error_include_file(myfile)not_found, but the file is there as I can see when unzipping the APK. I suppose is something similar to the load_files problem we talked few hours ago. Missing some path...

[mougino]

Retracing the steps: when BASIC! finds an INCLUDE keyword we go through AddProgramLine.doInclude() > Basic.getBufferedReader(SOURCE_DIR, filename) > Basic.streamFromResource(dir, path) > Basic.getAppFilePath(subdir, subpath). So you got it right: if using an INCLUDE "myprog2.bas" then you should copy myprog2.bas in assets/<app_path>/source. Careful though! As stated above the filename in the INCLUDE call must strictly respect the case sensitivity, i.e. if you try to include myprog2.bas via a INCLUDE "myProg2.bas" it won't find it and throw the error_include_file. Does it help?

[stefano-px]

Good night Nicolas. Sorry for delay in my reply. I took a pause. I spent all day building and testing APK. No, it does not help, because that's exactly what I did. I also tried quoting the filename and without quoting; and I paid attention to sensitive and insentive (this comes naturally on linux pc). In 1.76 DE RE Basic I read that the path of the included file must be expressed without quoting. Has this changed in 1.77? Anyway that does not solved. Same project, same files: 1.76 INCLUDE works, 1.77 does not. I can't catch it. I will try tomorrow evening. Thanks and good night.

[stefano-px]

Finally catched the problem. My included file was named 2.bas. For some reason this name lead to the error. Then I renamed it 2a.bas and NO error. Renamed a.bas, NO error. Pasting doesn't work anyway. But OK, don't name included bas files as a single "digit".
[EDIT] Also main bas fails to start if named 1.bas. I suppose is something related to java resources naming rules. I learnt something new.