RFE: Resources directory structure

[GoogleCodeExporter]

It is well known (Shai reminds us often) that you can't embed resources in a 
directory structure in your app because on iOS (and possibly Windows Phone) all 
resources are just copied flat in to the top directory of the app bundle.

It doesn't need to be this way.  For my offline builds of iOS I have been using 
full resource path structures for quite some time... I just updated my offline 
build project to do this too.  I do it by not letting XMLVM deal with the 
resources at all.  Instead I add a runscript build phase to the Xcode project 
that copies the resources into the bundle during the Xcode build step, 
maintaining the directory structure.

Relevant portions of my offline build script are as follows:

1. Copy all resources from the class path into a resources directory.
       <copy todir="${xcoderes.path}">
           <fileset dir="src" excludes="**/*.java"/>
           <fileset dir="lib/impl/cls" excludes="**/*.java"/>
       </copy>

2. Do the XMLVM build normally.

3. Add the runscript build phase to the xcode project:
        <exec executable="/usr/bin/php" dir="${buildscripts.path}">
           <arg file="${xcodeadd.path}"/>
           <arg file="${xcodeproj.path}/project.pbxproj"/>
       </exec>

The ${xcodeadd.path} variable refers to a PHP script that actually does the 
processing of the xcode project and adds the runscript.  I have attached the 
script for reference.  Ultimately it just adds the following 1-line runscript:

cp -r "${PROJECT_DIR}/../resources/" "${BUILT_PRODUCTS_DIR}"

Justification for this change:

1. One of the strengths of Java is its namespace ability.  Without the ability 
to also include complementary resources for Java libraries it limits the 
potential of third-party libraries.
2. Specific case is for libraries that wrap BrowserComponents and provide 
HTML-based components.  We need to be able to embed stylesheets, images, html 
files, and Javascript files - or if "need" is too strong a word, it is much 
nicer to be able to embed these things without having to encode them into a 
Java class.
3. Eliminate possibility of conflicts.  WIth third party libraries, even if 
they just use a flat directory structure, there is potential for conflicts if 
two libraries use resources with the same name.

Possible Pitfalls:

1. Optimization and Obfuscation.  Currently unused java classes and methods are 
stripped out of the resulting binaries which helps to keep the binary size to a 
minimum.  Resources don't naturally conform to this type of optimization so if 
the CN1 core gets bloated with lots of resources, it will result in the 
application binaries growing larger whether or not the additional resources are 
used.

      Solutions:  
          a. Don't use resources in the CN1 core.  They are still very useful for external libraries.  That way application developers can opt to use the library only if they need it.  Thereby not bloating their application with unneeded resources.
         b. Require CN1 classes to declare which resource directory trees they depend on using either an annotation or some other method.  Then use s proguard (or similar) as a preprocessing step to strip out resources which are not needed.

         (I think solution a is perfectly fine and it doesn't require any special development).

2. Consistency across platforms.  If we make this change to iOS, then what 
about Windows Phone, Blackberry, J2ME, and Android?  
        Solution:
          I would propose that all platforms should retain directory structure of resources.  Android already does.  I haven't tried Windows Phone but I'm sure a similar solution to the iOS solution would not be difficult to create.  I presume since JME and Blackberry are built on Java that they already retain directory structure of resources.

3. Backward compatibility:  What if current apps depend on the flat directory 
structure?
       Solutions:
          a. Add a built flag to make this optional.
          b. Deprecate the old behavior and ask people to change it.  I doubt many people depend on this behavior, and if they do, they are probably placing their resources in the root anyways, which would still function the same way.

This one change will drastically increase the potential for a third-party 
library eco system.

References:
1. The offline build tools projects build file.  (It's a mess, but you can 
refer to the "build-for-ios-device-locally" target to see the IOS build rules:
http://codenameone-incubator.googlecode.com/svn/trunk/shannah/offline-build-tool
s/templates/build-ios.xml

2. PHP Script for adding runscript build phase to Xcode Project:
http://codenameone-incubator.googlecode.com/svn/trunk/shannah/offline-build-tool
s/scripts/prepare_xcode_project.php

Steve

Original issue reported on code.google.com by st...@weblite.ca on 29 Jul 2013 at 8:37

Attachments:

• prepare_xcode_project.php

[GoogleCodeExporter]

Whoops.  Just noticed a bug in the copy resources snippet I posted here.

The step 1 to copy all resources into a resources directory should be:
       <copy todir="${xcoderes.path}">
           <fileset dir="src" excludes="**/*.java"/>
           <fileset dir="lib/impl/cls" excludes="**/*.class"/>
       </copy>

Because the lib/impl/cls directory will contain .class files, not .java files.

Original comment by st...@weblite.ca on 29 Jul 2013 at 8:39

[GoogleCodeExporter]

I don't understand how this works. The problem is that xmlvm flattens the 
hierarchy not us. Where do you update xcode with the files if XMLVM doesn't do 
that?

Original comment by shai.almog on 31 Jul 2013 at 12:37

[GoogleCodeExporter]

XMLVM only needs the .h, .m, and .class files in order to to its work.  It 
outputs an Xcode project.

With the resulting Xcode project, I add a build phase that tells Xcode to copy 
all of the runtime resources into the project build directory during its 
compilation.

I did this by:
1. Convert the .pbxproj file into an XML property list (instead of the old 
format) using plutil (because XML is easier to work with)
2. Add a PBXShellScriptBuildPhase entry to the .pbxproj file.  It looks like 
the following in the .pbxproj:
<key>e4e5a3214a4462a5fd77109d</key>
<dict>
    <key>buildActionMask</key>
    <string>2147483647</string>
    <key>files</key>
    <array/>
    <key>inputPaths</key>
    <array/>
    <key>isa</key>
    <string>PBXShellScriptBuildPhase</string>
    <key>outputPaths</key>
    <array/><key>runOnlyForDeploymentPostprocessing</key>
    <string>0</string>
    <key>shellPath</key>
    <string>/bin/sh</string>
    <key>shellScript</key>
    <string>cp -r "${PROJECT_DIR}/../resources/" "${BUILT_PRODUCTS_DIR}"</string>
</dict>

The buildActionMask just needs to be exactly that value (2^32-1).  The key just 
has to be unique among keys in the project file.

So this just tells Xcode to copy all of the files in the resources directory (a 
directory that I have previously filled with *only* the runtime resources) to 
the ${BUILD_PRODUCTS_DIR} which is an Xcode variable for the application bundle.

This results in a bundle that has all of the runtime resources in it with 
preserved package structure.

3. Build the xcode project (I guess you guys are using pbxbuild).

Note 1: I had originally thought that I needed to convert the .pbxproj file 
back to its original form for Xcode to accept it, but it turns out you can just 
leave it as an XML property list and it will still work fine.

Note 2: The PHP script I attached, handles all of the parsing and converting of 
the .pbxproj file and insertion of the build phase.  The closest thing to a 
main() method in that script is:

/**
 * A build script that prepares the XMLVM xcode project.  Adds the
 * build phase to import resources into the app bundle, and also
 * adds file references for the specified files.
 * @param type $path
 * @param type $filesToAdd
 */
function prepareXcodeProject($path, $filesToAdd = array()){
    $proj = new XCodeProject();
    $proj->load($path);
    $proj->addShellScriptBuildPhase(array(
        'shellPath' => '/bin/sh',
        'shellScript' => 'cp -r "${PROJECT_DIR}/../resources/" "${BUILT_PRODUCTS_DIR}"'
    ));

    foreach ( $filesToAdd as $file ){
        $proj->addFileReference(array(
            'name' => basename($file),
            'path' => $file
        ));
    }

    $proj->save();
}

What it does is probably self explanatory.

Original comment by st...@weblite.ca on 31 Jul 2013 at 4:37

[GoogleCodeExporter]

Oh.. one more thing.  After adding the build phase to the project file, I 
needed to add this build phase to each target in the project.

This is the PHP code that does this:
// Now we need to add it to our target
        $xpath = new DOMXPath($this->dom);
        $keys = $xpath->query('//key');
        $arr = null;
        foreach ( $keys as $key ){
            if ( $key->nodeValue == 'buildPhases'){
                $arr = $key->nextSibling;
                while ( $arr != null and (!$arr instanceof DOMElement) ){
                    $arr = $arr->nextSibling;
                }
                break;
            }
        }

        if ( $arr === null ){
            throw new Exception("No targets with build phases found");
        }

        $strEl = $this->dom->createElement('string');
        $strEl->nodeValue = $buildPhase['key']->nodeValue;
        $arr->appendChild($strEl);

Essentially it looks for a tag like <key>buildPhases</key> which will always be 
followed by an <array> tag with <string>ID</string> children.

I just add a <string>xxx</string> tag as a child of this where xxx is the key 
of my build phase.  In the example I posted, it would be 
<string>e4e5a3214a4462a5fd77109d</string>

Original comment by st...@weblite.ca on 31 Jul 2013 at 4:47

[GoogleCodeExporter]

We are using xcodebuild and going into those scripts probably wouldn't be 
something we will do in the near future unless we need to completely overhaul 
everything. There is so much functionality on top of this its just not 
practical to convert it.

Original comment by shai.almog on 31 Jul 2013 at 4:51

[GoogleCodeExporter]

I can understand the reluctance to want to tinker.  Hopefully at some point 
you'll have other use cases that will be more compelling for you to make this 
change.  To me, this is a pretty big deal but it's not a show stopper since I 
don't need to use the build server.  The build server preserves package 
structure for Android, and I can perform offline builds for iOS.

Original comment by st...@weblite.ca on 31 Jul 2013 at 4:58

[GoogleCodeExporter]

I'd like to have a solution for this but I'm looking for something that is 
guaranteed to be portable. E.g. character type X on os Y might fail for me. JAR 
solved these things nicely but you are asking to do this with the native 
structure which I don't want to rely on.
Furthermore, the advantage of being able to package resources into a RES file 
will solve one of the more painful aspects of HTML programming since we would 
be able to use multi images for image resources.
No its not the standard but neither is doing something different for every OS, 
it could work exactly as it does in the simulator which means consistency.

Original comment by shai.almog on 31 Jul 2013 at 5:16

[GoogleCodeExporter]

If there is a solution to, say, copy a directory structure with subdirectories, 
etc.. into a resource file and allow them to be addressed using a namespacing 
mechanism, and have this compatible with native components like the browser 
component, that that would be sufficient.  To me that sounds like a difficult 
thing to do.

I'm not sure I understand the issue of portability.  Can you elaborate on the 
"character type X on os Y might fail for me" example and how it might relate 
maintaining directory structure of resources?  Are you talking about characters 
in file names causing problems for the file system?

Original comment by st...@weblite.ca on 31 Jul 2013 at 5:32

[GoogleCodeExporter]

Adding the hierarchy to the res file should be pretty easy. The only issue 
would be in supporting the URL type for browsers, I was hoping you would have 
some input on that assuming I can get an input stream...

E.g. local UTF-8 characters are very common and people use characters like % in 
file names. I wouldn't mind if this fails or succeeds just that it would act 
fail/succeed consistently between the device and the simulator.

Original comment by shai.almog on 31 Jul 2013 at 6:24

[GoogleCodeExporter]

I will have to look into each platform to see if it supports it.  It looks like 
this may be possible with iOS using an NSURLProtocol implementation (but I've 
just started reading about this).  
http://stackoverflow.com/questions/9301611/using-a-custom-nsurlprotocol-with-uiw
ebview-and-post-requests

For android I have run through some people asking about this, but haven't seen 
a definitive solution yet.
E.g. https://groups.google.com/forum/#!topic/android-developers/INijIZ4F5G8

It will take some tinkering to find out one way or the other.

The JavaFX web view supports loading resources to a WebView from a Jar file.  
It does this by using URL.toExternalForm() on the specific resource.  But it is 
able to correctly load sub-resources in the jar's child directories (e.g. if 
images are in a subpath of the displayed page in the jar file.  This tells us 
that *at least* it will be possible to create this behaviour in the simulator.

I haven't even touched a Windows phone yet so I don't know where to begin on 
there.

Original comment by st...@weblite.ca on 31 Jul 2013 at 11:05

[GoogleCodeExporter]

This seems like a bit too much effort. Maybe a better solution is to just 
seamlessly unpack the resource the first time it is shown into a temporary 
folder and just work from there?
This should be reasonably portable and require absolutely no change to our 
build scripts. The only concern I have here is if a resource changes, I 
wouldn't want to unpack the data every time the app runs.

Original comment by shai.almog on 1 Aug 2013 at 5:01

[GoogleCodeExporter]

Just did a bit of research regarding custom URL handlers.  It looks like it 
isn't that hard on iOS, Android, JavaSE, and WinPhone.  Haven't found a way yet 
on JavaME or BlackBerry.

Here are some links:

Android  & JavaSE

Mentions using a Custom URLStreamHandlerFactory for loading images:
http://blog.wu-man.com/2012/10/androidimageloader-loading-images.html

setURLStreamHandlerFactory API docs
http://developer.android.com/reference/java/net/URL.html#setURLStreamHandlerFact
ory(java.net.URLStreamHandlerFactory)

More Discussion of Loading resources from a resource file using custom URL 
stream handler
http://stackoverflow.com/questions/8938260/url-seturlstreamhandlerfactory
http://stackoverflow.com/questions/861500/url-to-load-resources-from-the-classpa
th-in-java

More on custom url stream handlers
http://www.unicon.net/node/776

Windows Phone 7 & 8 

WebRequest Creator
http://msdn.microsoft.com/en-us/library/windowsphone/develop/system.net.browser.
webrequestcreator(v=vs.105).aspx
(Looks like it can be used to register custom protocols)

iOS

Custom NSURLProtocol
http://stackoverflow.com/questions/9301611/using-a-custom-nsurlprotocol-with-uiw
ebview-and-post-requests

Java ME

Haven't found a way to make custom protocol handlers yet

Unpacking to a temp directory may work... I haven't tried loading any web pages 
from any locations other than the main bundle on iOS yet... we'd want to make 
sure we haven't traded one difficult problem for another one :)

Original comment by st...@weblite.ca on 1 Aug 2013 at 6:24

[GoogleCodeExporter]

Interesting... J2ME doesn't have a native browser so naturally you won't find 
anything.
HTMLComponent has its own URL code which already supports loading from 
resources.

Original comment by shai.almog on 1 Aug 2013 at 9:57