Support for Dynamic Feature Modules

fruitedev commented 4 years ago

Hello,

I know there is support for AppBundles which makes things easier, but i couldn't find whether Dynamic Feature Modules is supported or not. Support of Dynamic Feature Modules will be really really helpful for us so please share any plan you have.

Thanks,

jonathanpeppers commented 4 years ago

@fruitedev I remember there might have been some limitations when I first implemented app bundles:

https://github.com/xamarin/xamarin-android/blob/master/Documentation/guides/app-bundles.md#what-are-app-bundles

I remember something about needing to support "Instant Apps" for dynamic modules to work, which has a hard file-size limit. The Mono runtime was over the limit.

We should still look into this again. Anyone interested, please give a thumbs up above and that will help us know the demand, thanks!

johnthiriet commented 4 years ago

@jonathanpeppers Let's consider that the module I want to load dynamically is an aar archive with some jar dependencies and I have a binding over this one. Would that make the process easier on your side ? I mean I do want to load this aar dynamically since it's pretty heavy.

evansmj commented 4 years ago

Now that Apple has introduced App Clips, clients are now more interested in Android Instant Apps. Using app bundles may not be so possible due to the Mono runtime limit and that is fine, but if there was a way we could at least attach natively written instant app apk's as feature modules to the Xamarin.Android app that would be great.

johnthiriet commented 4 years ago

Still really really interested in the dynamic delivery for our applications. We are open for a quick call to explain more about why if you want. @jonathanpeppers @davidortinau.

dellis1972 commented 4 years ago

I found that a community member has already bound the PlayCore library, https://github.com/PatGet/XamarinPlayCoreUpdater. This might have enough support to allow you to deliver a natively written apk as a dynamic feature and keep the main app as a Xamarin App.

johnthiriet commented 4 years ago

@dellis1972 Hum that might be but actually "publishing" this module to the play store will maybe be tricky if even possible as the application is in Xamarin. Even by creating a native application just for packaging this module I think it will not work well. Have you heard about someone doing it ?

dellis1972 commented 4 years ago

@johnthiriet I haven't heard of anyone doing it yet. The developer it seems only used it for in app updates.

dellis1972 commented 3 years ago

I did some investigation last week as to what might be needed to support this feature. Firstly the API's required do use this feature are not currently working in https://github.com/PatGet/XamarinPlayCoreUpdater. When trying to use them you will get a javac error at build time. This is down to the actual binding itself. One of the problems is that the SplitInstallManager uses Listener types which make use of Generics. Our binding story for generics is not completely working at this time. So that will be the first hurdle, get the binding working somehow. This might end up being something like a custom .jar which has a wrapper with a simplified API. We then bind the simplified API rather than use the code google provides. While this is not a great approach, it might be our only option.

The next issue is how do we build a "feature". Some investigation into this shows that it is not going to be straight forward. The "feature" apk/zip needs to be built against the apk of the application. So we need to build any features after we have built the main application , but before we do the final bundling. The best option is to run a custom target before _PrepareBuildApk which will allow us to build the features, and gather any outputs.

The reason we need to build the features after is the packaged_resources zip file that gets produced by aapt2 as part of our build process MUST be passed as a -I argument to aapt2 when building the "feature"

aapt2 link --allow-reserved-package-id --package-id 0x7e --auto-add-overlay -o $(LevelPackageTemp) -I $(JavaPlatformJarPath) -I $(_PackagedResources) --manifest AndroidManifest.xml -A assets

This is required because the entires in the AndroidManifest.xml we need to include in the "Feature" MUST use resources from the main app for its distribution elements. This is a sample of a working AndoridManfiest.xml for an "Asset Feature"

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
        xmlns:tools="http://schemas.android.com/tools"
        xmlns:dist="http://schemas.android.com/apk/distribution"
        android:versionCode="1"
        android:versionName="1.0"
        package="com.infinitespacestudios.adtest"
        featureSplit="level1"
        android:isFeatureSplit="true">
    <dist:module dist:title="@string/level1" dist:instant="false">
        <dist:delivery>
            <dist:on-demand />
        </dist:delivery>
        <dist:fusing dist:include="false" />
    </dist:module>
    <uses-sdk android:minSdkVersion="21" android:targetSdkVersion="30" />
    <application android:hasCode="false" tools:replace="android:hasCode" />
</manifest>

Note the @string/level1 resource. That item must be present in the main app strings.xml. So we need to build the main app before we build the feature. Also we cannot build the resources using the --proto-format it seems. We need to build as a normal apk then use the aapt2 convert call to change the format later.

aapt2 convert --output-format proto -o $(LevelPackageProtoTemp) $(LevelPackageTemp)

We then need to manipulate the output file to move things around a bit. By default aapt2 places the compiled AndroidManifest.xml in the root of the zip file. However for aab files it needs to be in a manifest subdirectory within the zip. So we need to fix up the zip. Luckily we already have a task that can do that, so we can probably reuse some code to handle this. Any assemblies we include in the "feature" zip file need to be in the root/assemblies folder as per our normal aab code. At this time the runtime does not support "reloading" our assembly list so we cannot support dynamic delivery of assemblies at this time. However with some additional investigation we might be able to support it.

Once we have this "feature" zip file build we can then pass that to the bundletool via the AndroidAppBundleModules ItemGroup (see here)

The current idea on how an end user might use this is that we allow for an additional piece of meta data on the ProjectReference ItemGroup.

  <ItemGroup>
      <ProjectReference Include="Levels\Level1\Level1.csproj">
         <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
         <AndroidDynamicFeature>true</AndroidDynamicFeature>
      </ProjectReference>  
  </ItemGroup>

In this case ReferenceOutputAssembly HAS to be false otherwise our build system will just include the assembly in the final package. AndroidDynamicFeature is the new item and if 'true' means that during packaging, the feature package will also be produced. The nice thing about using ProjectReference is that the feature will be built as part of a normal build.

At runtime once a user has installed a "feature" they can then use CreatePackageContext to get a new instance of an AssetManager which will contain the new assets.

var ass = CreatePackageContext(PackageName, 0);

For assemblies we need to decide what to do. For java apps they can call StartActivity to kick off an activity assuming the feature contains android activities. The AndroidManifest.xml for the feature would also need to have those activity elements in it so android can pick those up. This might work once we have runtime support to detect new split installs. However we are currently unsure how to do this.

The other option is to use Assembly.Load to load the new "feature" assembly. This might be required if you want to call C# code since there is no reference between the main app and the feature. Reflection will probably be needed to create an instance of a feature class and execute methods. We might be able to hook into the runtime Assembly.Load method do so a search in additional apks/directories if we don't find an assembly in our current mapping list. However this will need us to be able to known "where" the new things are installed.

So we have a number of things we need to implement and investigate before we can fully support this. We will probably have to break this up into a smaller set of projects. The initial goal will be to support dynamic delivery of "assets" the follow that up with dynamic delivery of assemblies later since the runtime changes will not be easy.

UPDATE: So more investigation has turned up some interesting info. Each "Feature" needs to have a different package-id. By default the package-id is 0x7f for an app. What gradle seems to do is DECREMENT this value for each "feature" that is in the project. So your first "feature" will be 0x7e , you second 0x7d etc. So there is a hard limit on the number of features you can have. This value will need to be passed to aapt2 via the --package-id argument in addition to the --allow-reserved-package-id.

Some thoughts on using ProjectReference. I don't think we can use that now, mostly because we probably need to be able to reference the main app from the feature. To do this we need to pass the feature a path to the main app assembly which needs to be built first. So using a ProjectRefernce will not work since they will be built before the main app. So we will probably need to create a new ItemGroup AndroidDynamicFeature which we can then build after the main app.

<ItemGroup>
     <AndroidDynamicFeature Include="Features/Asset/AssetFeature.csproj" />
</ItemGroup>

This will allow us to pass in not only the main app dll but also all the references which the main app have. This will include things like AndroidX.* and GooglePlayServices. So there should be no need to add those things to the "feature" as PackageReference .

So a new todo items are.

[x] Make sure all calls to BuildAppBundle pass the AndroidAppBundleModules ItemGroup
[ ] Update Aapt2Link task to support multiple items for the JavaPlatformJarPath or provide a new property to allow us to pass additional -I items.
[ ] Create a new Aapt2Convert task to handle converting from apk to aab formats.
[ ] Update Tasks like BuildApk and BuildBaseAppBundle so that we don't always include runtime native libraries and assemblies. These are not required when building a feature zip.
[x] Update BuildApkSet to allow users to pass modules #5327
[x] Update BuildApkSet to allow users to use the --local-testing argument which can be used to test dynamic features without having to upload to google #5327.
[ ] Investigate what might be needed to detect when new split installs are done and refresh our assemblies mapping in the runtime.
[ ] Create a new set of .targets which are specific to dynamic features and are geared to its requirements.
[ ] Investigate the idea of auto generating the AndroidManifest.xml for features as the format is slightly different.
[ ] Create a new template for dotnet new for a dynamic feature?
[ ] See if there is a way to allow debugging support when using aab files. This will be required in order to let users test downloading features etc. This cannot be done using apk files since we must build an aab for any of this to work.
[ ] Add support for --package-id and --allow-reserved-package-id to Aapt2Link.
[ ] Update the Resource.designer.cs generator to take in multiple R.txt files.
[ ] Compile the "Feature" against the assemblies from the main app. This will give it access to all the package and references of the main app (like AndroidX). We only want to pack the "Feature" assembly and nothing else.
[ ] Make sure Java binding code is generated and included in the Feature manifest file
[ ] A whole bunch of other stuff which I have not thought about or has not come up yet.

This is just a brain dump of all the stuff I have found other the last couple of days. There are probably more things we need to do but this is a good starting point. To manage some expectations, this is not going to be a quick process. We have not even mentioned IDE integration or the actual end user experience yet. So this is going to be an on going process. But we are looking at it, which is a start :)

jonathanpeppers commented 3 years ago

Ah, so we should be able to knock this out in an afternoon, right??? 👀

johnthiriet commented 3 years ago

@dellis1972 That is the most interesting thing I have read in weeks :-)

chc7042 commented 3 years ago

Hi @dellis1972 and all i saw the arcicle https://github.com/hinojosachapel/DynamicModules and it working correctly. as i know the android app-bundle has an limitation unlike with WPF dynamic loading. main purpose of android app bundle is seperate some size of binary as bundle for reduce initial time for loading. and it could not be splitted because app bundle should be determined at compile time. so it really different with WPF dynamic bundle. how do you think about this? and more what i want to know is (i am beginer of WPF(C#)) is there no way to loading module look like https://github.com/hinojosachapel/DynamicModules?

dellis1972 commented 3 years ago

I'm working on the initial support for this on https://github.com/xamarin/xamarin-android/pull/5890 .

It would be nice to get some community feedback on how I plan to implement this. You can read the PR and some initial documentation. All of this is subject to change, but it's a good start. I have something working for AndroidAsset locally but we are still a way off from getting other parts like AndroidResource or even Activities etc.

dellis1972 commented 3 years ago

@chc7042 thanks for the feedback. What we are discussing here is adding support to Xamarin.Android for the Dynamic Feature support that google provides though its tooling. So the idea is to expose what google provides to XA developers in a way that makes sense for Android. That WPF system makes use of things like Prism which we would want to avoid using because of the large overhead it would have. Also as an SDK developer I should not be forcing developers to use a particular type of framework.

I'm sure we will figure out the ability to load code for dynamic features at some point. Its just going to take some time to figure out the best way to implement it.

ghost commented 3 years ago

@dellis1972 are you including Install Time Play Asset Delivery as well?

dellis1972 commented 3 years ago

@IainCoSource in theory google should handle all of that. As long as the manifest has the right xml elements. I plan to have a FeatureDeliveryType MSbuild property which will let people decide what type of delivery the feature will get. https://github.com/xamarin/xamarin-android/pull/5890/files#diff-f59539dc083cf43b32a9a4fc94a4b71436dfe549137f8ec866964851bbd718a2R53

ghost commented 3 years ago

@IainCoSource in theory google should handle all of that. As long as the manifest has the right xml elements. I plan to have a FeatureDeliveryType MSbuild property which will let people decide what type of delivery the feature will get. https://github.com/xamarin/xamarin-android/pull/5890/files#diff-f59539dc083cf43b32a9a4fc94a4b71436dfe549137f8ec866964851bbd718a2R53

So, for each file in the desired feature type i.e. FeatureDeliveryType.Install_Time_Asset_Bundle would get included in the aab.

But how would you name the Asset Bundles?

Here is a little mock-up in VS, based on how the process works in android studio.

The aab unzipped would look as follows

dellis1972 commented 3 years ago

So the FeatureDeliveryType is a reflection of the diet:delivery settings that end up in the feature manifest. It is set at the Project level. Assets with just have the normal AndroidAsset build action as they do in normal projects. The dis:delivery element in android supports dist:install-time , dist:on-demand. So the the FeatureDeliveryType will have values for InstallTime and OnDemand. I've not figured out how we will support conditional delivery yet, that might be something users will need to do via a AndroidManifestOverlay.

As for naming , the current plan is to use the $(ProjectName) name as the feature name although this will be customisable via FeatureSplitName.

Note that we currently can't support the InstantApps feature because you cannot ship native libraries in those and we require the mono runtime in order to operate.

ghost commented 3 years ago

Some of that is a bit beyond my current knowledge. But if there is something I can do to help please let me know.

dellis1972 commented 3 years ago

While we are working on support for this in the main app. I published rather hacky sample which shows how to implement "Dynamic Asset Delivery" over at https://github.com/infinitespace-studios/XamarinLegacyDynamicAssetsExample. This sample uses a bunch of custom targets to get the job done. So it will work on the current Stable release of Xamarin.Android.

NOTE: This will only work for "Dynamic Asset Delivery" it will NOT be able to handle activities or resources. It also contains a little wrapper binding which fixes the issue were we cannot use the SplitInstallStateUpdatedListener Java class because it uses Java Generics. I've added documents on how this hack works.

I've tested it locally and it seems to work for me. I have no idea if it will work on Google Play or not though.

So take a look while you wait :D

ghost commented 3 years ago

While we are working on support for this in the main app. I published rather hacky sample which shows how to implement "Dynamic Asset Delivery" over at https://github.com/infinitespace-studios/XamarinLegacyDynamicAssetsExample. This sample uses a bunch of custom targets to get the job done. So it will work on the current Stable release of Xamarin.Android.

NOTE: This will only work for "Dynamic Asset Delivery" it will NOT be able to handle activities or resources. It also contains a little wrapper binding which fixes the issue were we cannot use the SplitInstallStateUpdatedListener Java class because it uses Java Generics. I've added documents on how this hack works.

I've tested it locally and it seems to work for me. I have no idea if it will work on Google Play or not though.

So take a look while you wait :D

Will take a look. :)

Eversor commented 2 years ago

Looking forward to this feature, happy to know that a Monogame veteran is doing the job ;) may i help you testing it?

dellis1972 commented 2 years ago

@Eversor its still a work in progress I'm afraid. I have had to put it on hold for a while because of other more important tasks on Xamarin.Android. That said I did find some time to put a MonoGame specific example together at https://github.com/infinitespace-studios/MonoGameAndroidAssetPackExample for those who are interested 😄

dellis1972 commented 9 months ago

For those of you who are interested in asset packs I have a new PR up at https://github.com/xamarin/xamarin-android/pull/8631.

We could use some feedback on the approach we plan to take on this if anyone has the time :)

dellis1972 commented 7 months ago

Asset Pack support has been merged, should be in the next .NET 9 Previous.

Still trying to figure out how we can even support feature packs :/

dotnet / android

Support for Dynamic Feature Modules #4810