Go-NoGo + architecture changes for support of apktool

[mougino]

Hello to all, I have been successfuly investigating a brand new direction that would allow to compile a BASIC! program into an APK either 1) from a computer much more easily than the current AppBuilder (no need of JDK+Android SDK etc.) and/or 2) directly from within BASIC! on the Android device. This direction is called apktool: https://code.google.com/p/android-apktool/ https://code.google.com/p/apktool/

There are however some cons, it's not all pros:

• the principle of apktool is to make changes to an existing APK without altering it (it still can be installed after the changes). This means that the keystore and signature would not be touched. In other words, BASIC! APKs produced with apktool will be signed by Paul's BASIC! key. I don't know the ramifications in terms of Google Play store but it is worth mentioning
• the Android version would need to download a 60MB zip onto the sdcard at first build, this could be done via WiFi only but it has significant end user impact to consider if it's a viable solution
• in order for both solutions (PC and cellphone) to work the current Basic.java would have to be modified, because it works better to only decompile the /res and not the /src then replace everything in /res to match user program then recompile we would have to export the following values into an xml:

    public static String AppPath = "rfo-basic";             // Set to the path name for application directories

    public static boolean isAPK = false;                    // If building APK, set true

    public static boolean apkCreateDataDir = true;          // If APK needs a /data/ directory, set true

    public static boolean apkCreateDataBaseDir = true;      // If APK needs a /database/ director, set true

What do you guys think? Does this sound promising enough to continue investigating or current solution is good enough?

[mougino]

@stefano-px well this is odd indeed. This might be a bug in BASIC! code because others have been able to access files from assets named as numeric digits only: see http://stackoverflow.com/questions/11303360/prevent-mediaplayer-from-play-all-mp3-files-in-the-folder-automatically

@jMarcS @mrleavitt I forked De Re BASIC! to propose some updates on the limitations @stefano-px and I have discovered in APK mode i.e. "1.bas" (or any numeric-only name before the extension) fails to load or be INCLUDEd, file names need to respect the case (all of them concern APK mode but standard BASIC! works well, tested it)

[stefano-px]

@mougino Also, remember the issue about “load_file_names" array in setup.xml. In current 1.77 test version, relative path data/file.txt or database/file.db must be specified for files to be loaded. This is not explained in DE RE Basic 1.77 Appendix D., so I don't know if this a bug to fix, or a new rule to respect.

[mougino]

Yes, I agree. However this is specific to Appendix D and @jMarcS has sent us a copy with a (wrong) screenshot of the content of load_file_names (<item>cats.png</item> instead of <item>data/cats.png</item>). I don't have Eclipse on my computer, so I'll let Marc fix this and make a new screenshot if that's ok with him.

[jMarcS]

Hi, all. I am back home and back online. I'll try to get caught up on all of your comments as soon as I can.

About case-sensitivity: I assumed all real file names would be case-sensitive because it's a Linux file system, but you're right: sometimes it will be a (case-insensitive) FAT file system. There was a problem a while ago when Cartman.png somehow got changed to cartman.png -- sometimes the shortcut-maker cared and sometimes it didn't. We'll have to figure this out: we can't ask users to change all their file references.

I will investigate the single digit file basename issue. I can't think of any good reason for that to be a problem, but I don't know the internals of the assets folder.

And about load_file_names: Don't go changing the manual if there's a bug in the code! The design should drive both the code and the manual. If the code is wrong, let's fix it.

Question: has anybody tried building an APK with resources (the old way) instead of assets? I tried to make it work both ways.

I'll have more comments when I get caught up.

[stefano-px]

@jMarcS Hello Marc. I have a little request for next 1..77, or 1.78. Is it posibble to make the APK don't show lines in console (or lined console) output? I tried to edit Basic.java in the part of the colors, editing lineColor = backgroundColor, but nothing happens. Also editing xml/settings.xml, switching from BW to WB, does not affect the console colors: we have to swap color1 and color 2 in setup.xml, but that's OK. lined_console true or false doesn't work. It seems that values in xml/settings.xml do not affect APK. Some suggestion? Thanks.

[jMarcS]

Wow! Nicolas, that was quite a marathon! Congratulations on your success with apktool.

This one-line change in Basic.java:LoadGraphics() (line 704)

old: String path = AppPath + File.separatorChar + fileName;
new: String path = getAppFilePath((fileName.endsWith(".db") ? DATABASES_DIR : DATA_DIR), fileName);

fixes the load_file_names error. You can write it into your source file to try it. I have not committed it because we are not out of the woods yet.

Suppose you want a file like assets/<my-app>/other/file.txt. You would expect to put ../other/file.txt in load_file_names, right? But it does not work! I can not find documentation about assets path rules, but I can't make a path with .. in it load. I don't know if Google decided to prevent using .. to walk out of the assets folder or if it was an accident of how the assets folder is handled. It has an interesting side-effect: we can prevent users from using load_file_names to write outside of their sdcard/<my-app> folder.

I can think of two options.

One is to do just what Nicolas said: have the path relative to assets/<my-app>, like data/meow.wav and other/file.txt. Pro: no more special handling of the .db extension. Con: I don't like requiring users to repeat data/ for most of their files. So:

<item>data/meow.wav</item>
<item>databases/info.db</item>
<item>other/file.txt</item>

The other is more complicated: if the path starts with / it is relative to assets/<my-app>, otherwise it is relative to assets/<my-app>/data (or databases). Pro: simple syntax for most users, flexibility for advanced users. Con: not intuitive for any users. So:

<item>meow.wav</item>
<item>info.db</item>
<item>/other/file.txt</item>

[mougino]

quote: About case-sensitivity: [...] We'll have to figure this out: we can't ask users to change all their file references.

I'm afraid there is no other choice, I already spent some time yesterday searching for a workaround in Android forums / stackoverflow all the advice were to change the call to match the perfect case of the filename and all users ended up in doing that...

About the non recognition of "../" in assets, that is a real bummer! I really counted on it for users working with resources outside of the base path be able to create an APK without any trouble. Couldn't we work around this by first constructing a File that will resolve the "../": File file = new File(Basic.getDataPath(fileName)); or File file = new File(Basic.getDataBasePath(fileName)); depending on the .db extension, then transform it to String: String filepath = file.getName(); and finally removing the base path: filepath = filepath.replace(Basic.getBasePath(), "") ??

[jMarcS]

Nicolas, instead of changing all of the file names in your program,can you change the names in the assets folder? If your program always looks for Meow.WAV, then make the asset Meow.WAV. If it is copied to the SDcard, the program can find it whether it looks for meow.wav or MEOW.WAV or MeOw.WaV, but the asset name must exactly match the what the program wants.

There is another way. At start-up, we can get a complete list of the files in assets -- I do something like that already to recursively copy assets/rfo-basic to the SDcard. Then when a user asks for a file from assets, we do a case-insensitive search of the list. It's a little messy, sure, but it's better than asking users to change their programs.

About "../": Yes, that works! Thank you. Here is a rough sketch:

String path = getAppFilePath((fileName.endsWith(".db") ? DATABASES_DIR : DATA_DIR), fileName);
File file = new File(path);
path = file.getCanonicalPath();

If fileName is ../other/file.txt, then path is initially myapp/data/../other/file.txt, and getCanonicalPath() returns /myapp/other/file.txt. We use that directly for the output file, and strip off the leading '/' for the asset:

file = new File(getBasePath() + path);  // output file
path = path.substring(1);
inputStream = getAssets().open(path); // input from asset

An asset is not a file, but the File class doesn't care. It can operate on a path String even if the file doesn't exist.

[mougino]

The thing is I sometimes called "Blesse.mp3" sometimes "blesse.mp3" in the program, so it was faster for me to replace all occurences of "blesse.mp3" with "Blesse.mp3" in the program. Also, it's not that easy to rename a file from "Blesse.mp3" to "blesse.mp3" on the internal memory of my Android device, because it's a FAT file system, it is the same as wanting to do that on Windows, I don't know if you already experienced? Trying to rename directly "Blesse.mp3" to "blesse.mp3" doesn't work, Windows (or Android in that case) considers it's the same filename regardless of its case, so does not rename it. You need first to rename "Blesse.mp3" to "Blesse2.mp3" (different name) then rename "Blesse2.mp3" to "blesse.mp3"...

Great job for the "../" workaround in assets ;o) I was sure you could do it!

[jMarcS]

@Nicolas: I converted the "rough sketch" for "../" to real working code -- for instance, before writing a file we now have to create its parent directory. I think we agreed in the other thread that it's okay to let the user write anywhere on the SDcard she has permission to write, so I did not restrict the path to stay inside assets/<app-path>.

@Stefano: I found the bug that made program names like 1.bas and 2.bas fail. It works now. I am glad you caught that. It is caused by the way Android gets the raw resource IDs. My fix has a cost: your program can no longer read from a resource folder other than "res/raw". But nobody knew you could do that -- I certainly did not know! -- so it is not much cost.

[mougino]

Hello @jMarcS what is the cost for the fix of INCLUDE 1.bas exactly? Your double negation confuses me sorry: "your program can no longer read from a resource folder other than res/raw" The consequence is that we cannot put the "1.bas" inside assets anymore? We have to put it in res/raw? Or the consequence is that the resources used in "1.bas" must be in res/raw not in assets? When you say "your program", are you talking specifically of the program including the "1.bas" or are you talking generally for all programs? Thanks for your clarification ;-)

[jMarcS]

I'm sorry, Nicolas. You're right, that was not clear at all.

I was talking only about restrictions on resources in the res/ folder. Normally you will put your programs in assets/<app_path>/programs. The change is that you can use a program named with a number, like 1.bas, either as your main program or in an INCLUDE.

I learned something new when I was making that work. The method that gets a resourceID from a resource name first parses the name as a numeric string. If the string, or even just the beginning of the string, can be parsed as a non-zero integer, it is assumed to be a valid resource ID! I do not know why they did that.

If the resource name does not start with a number, the method parses it in the format "package:type/name". The method takes three arguments: name, default_type, and default_package. If the parser does not find a ':', it uses the default_package arg. If it does not find a '/', it uses the default_type arg. That means these two calls are identical:

getIdentifier(packageName, "raw", fileName);
getIdentifier(packageName + ":raw/" + fileName, null, null);

We used to do the first; my change does the second. The fileName may start with a number, but the packageName never does, so the whole string does not start with a number.

But with the first format, a user (a BASIC! program) could have specified a file name like "icons/cartman". The default type, "raw", would be ignored. The program would load res/icons/cartman. (I have left off the ".png" extension, but it would work with the extension, too -- the extension is ignored.)

This "cost" I was talking about? It is that users can't do that any more. If they ask for "icons/cartman", the method argument would be something like rfo-cats/raw/icons/cartman. The change means a user cannot get a res/ resource that is not in res/raw.

I don't think anybody knew you could do that in the old code. If I tell you that you cannot do something you did not know you can do, have I taken something from you? Philosophically, yes: future potential opportunity. Practically, no.

And, of course, the whole argument is irrelevant. We don't expect users to use res/raw any more. We expect them to use assets/. That's one hand. On the other hand, I did not take away the ability to use res/raw if you really want to -- or if you have a legacy apk builder that still uses res/raw. But on the third hand (the "gripping hand", Larry Niven would say), I have not tested to verify that res/raw still works. And I have already told you my maxim about things that have not been tested.

Have I clarified the situation, or have I just drowned you in words? :{)} American salesmen say, "If you don't have a good story, baffle them with bullshit."

[mougino]

Got it. Thanks

[jMarcS]

I've gone through the comments in these two issues (118 and 123). I think the only outstanding issues for me are the console colors and console lines. I will work on them this weekend.

Stefano, I updated the comments in setup.xml as you suggested. That will get committed and pushed with the console fixes.

Is there anything else you guys need from me?

I think we are very close to release. Paul, Mike, and Nicolas will have to coordinate.

[stefano-px]

Thanks Marc!

[jMarcS]

Nicolas said:

The antialias fonts and some colors then: they look really different, while the code is strictly the same. Did you touch to these commands between v01.76?

I did not change them directly. Can you give me an example to investigate? I don't know where to start.

You also talked about case-sensitive file names. I did not implement that complicated work-around I told you about -- it feels very fragile. This may be one time when we have to break backwards-compatibility.

[jMarcS]

Console colors and lines: Stefano got it exactly right with this comment:

It seems that values in xml/settings.xml do not affect APK.

BASIC! was not loading the default values from settings.xml unless the user selected Settings from the Editor menu.

I changed the code so BASIC! does load the default values when it starts up. If it is standard BASIC!, it will not overwrite the existing preferences -- whatever the user changed stays changed. But in an APK, all of the settings are set to the defaults in xml/settings.xml, every time the app runs. There is some overhead there, but it is small. I don't think app users will notice the delay.

There was another problem when lined console is turned on. In some themes, with the default BW setting, the console lines are invisible. I overrode the theme default so the console lines will be the same color as the editor lines. For "BW" and "WB", that's the text color at half-alpha. For "WBL", that's "color1" at half-alpha (for "WBL", text is "color2" and background is "color3").

I haven't tried it, but this should let you change the typeface in an APK, too. You can change the csf_pref to choose between MS, SS, and S. If you want to actually specify a typeface, you'd have to modify the Java code in Settings.java:getConsoleTypeface(). Some day we will have to make that easier.

I know there are other problems with the Console output that this change does not help.

[jMarcS]

I keep saying that my changes should not prevent using resources (res/raw) instead of assets. The one place that was absolutely not true was loading files from "load_file_names"! Oops... (I need the embarrassed emoticon from the forum). That works in this commit, too.

[stefano-px]

@jMarcS Hello Marc. Just downloaded latest Basic-master zip from github and built a test APK. It seems that console color switching works well and console lines on/off too. Great! Thanks a lot. Even tested load_file_names, and all seems to work here.

[mougino]

Hi @jMarcS @stefano-px I thought I could easily use the internet from China but that is not true. I will be mostly offline until February 11th.

About v01.77 release I tested a bunch of stuff and current code seems great. I will have to give you specifics on the antialias when I'm back, I'm not even sure this is something neｗ. I judge that not critical enough to delay the new release. The fix of console customization is a good news too!

[mougino]

Have I just said that that a WiFi modem was installed a few minutes ago in our appartment ;-) So count me in for the new release: I can help and test, don't hesitate to ask me until Chinese New Year (which is in 3 days). Just be aware of the 16 hours time difference...

Back to the subject you are right about the old AppBuilder, it won't be able to make an APK from a v01.77 source code, but users can still use offline mode with v01.76 source code. As for the new AppBuilder "Quick APK" it is not 100% ready yet, and FYI there will be a new version containing the corresponding Basic.apk each time BASIC! has a new release. For the moment only beta testers of Quick APK will be able to create APKs with the equivalent of BASIC! v01.77 (but to be honnest that will be 100% transparent for them).

[jMarcS]

Changes in BASIC! released in released in v01.77 on 2014-02-09.

Nicolas, do you want to close this issue?