Dynamic module loading

[DartBot]

This issue was originally filed by @dam0vm3nt

---

This feature was initally described in

Issue #3819: Provide a Method to Dynamicaly Load Classes

But it has been closed pretending it was done. IT IS NOT as Mr. gbracha honestly admits: "... where the code being loaded is not known statically. This will require the ability advanced capabilities like mirror builders. This should be a separate issue."

So I'm posting that separate issue hoping sometime in the future will be adressed as it is the only reason stopping me to adopt dart.

In other word: We need some mechanism to dyamically load classes not know at compile time by another module. Something like java class loaders / Flex Module / JS AMD.

[4stern]

not without unnecessary effort ... imho a single migration file should look like this:

part of my_fancy_project;
class MyMigration extends AbstractMigration {
    Future doMigration(databaseConnector) async {
        //.... doing migration stuff with the databaseconnector
        //.... ex: databaseConnector.query('CREATE TABLE ... ');
    }
    Future undoMigration(databaseConnector) async {
        //.... undoing migration stuff with the databaseconnector
        //.... ex: databaseConnector.query('DROP TABLE ... ');
    }
}

the main migration-tool create the db-connector, loads all files and execute the "doMigration" function with the connector as parameter. if your project need a new migration, just add a file that looks like the snippet above and go on ...

imho you have no choice to take over a db-connector to an isolate? so the isolate has to manage all the stuff for its own ... right? a big unnecessary boilerplate :(

[zoechi]

Why do you need to load that dynamically?

[jodinathan]

Well, I am not sure, but it looks like you could either use an isolate or create a cmd with the migration stuff

2018-01-25 16:41 GMT-02:00 4Stern notifications@github.com:

not without unnecessary effort ... imho a single migration file should look like this:

part of my_fancy_project; class MyMigration extends AbstractMigration { Future doMigration(databaseConnector) async { //.... doing migration stuff with the databaseconnector //.... ex: databaseConnector.query('CREATE TABLE ... '); } Future undoMigration(databaseConnector) async { //.... undoing migration stuff with the databaseconnector //.... ex: databaseConnector.query('DROP TABLE ... '); } }

the main migration-tool create the db-connector, loads all files and execute the "doMigration" function with the connector as parameter. if your project is need a new migration, just add a file that looks like the snippet above and go on ...

imhp you have no choise to take over a db-connector to an isolate? so the isolate has to manage all the stuff for its own ... right?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dart-lang/sdk/issues/10530#issuecomment-360559802, or mute the thread https://github.com/notifications/unsubscribe-auth/AF6Y3hycGrPDp5k-lo4ndepSMa_C1cL7ks5tOMrFgaJpZM4E3_KU .

-- Jonathan Rezende

[4stern]

Why do you need to load that dynamically?

its easier instead of create the file, import it, instantiate it and add it to the migration-tool ... (this will create a long long import list on bigger projects) i just want to create a file in a special folder and the tool do the rest ...

but it looks like you could either use an isolate

maybe i have to think about it again, but other languages provide an much easier way ...

[nex3]

Providing any kind of in-depth API over the isolate interface is a nightmare. You have to manually forward all function calls, and everything needs to be asynchronous. We did this to load transformers in pub, which is a pretty simple API all things considered, and even that is a maintenance nightmare.

[dam0vm3nt]

@nex3 Absolutely agree with you. Yours is a perfect case where a dynamic loadable module system would be very usefull. In the meanwhile I've done some work using darttdevc but it's working only on the web side : https://www.dart-polymer.com/polymerize/polymer/mixin/dart/2.x/autonotify/dynamic/loading/2017/06/19/dynamic-loading/

[MewX]

Hoping there will be some functions allowing loading compiled dart classes, just like "loading DLL on Windows" and "dynamically loading *.dex on Android".

BTW, is is possible to let the program load dynamic module plugins for now?

[nex3]

This would also be very useful for Dart Sass, which allows users to define custom functions and importers but doesn't currently have any way of loading them from the command-line executable.

[nex3]

Note: this recently became higher-priority for Dart Sass, because Google users want to be able to easily load an importer that supports a Bazel-style virtual filesystem overlay without rewriting the wrapper executable.

[matanlurey]

I think given Dart 2, which is no longer (really) run-from-source, and has strict ahead-of-time compilation knowledge required for the language, is unlikely to add this feature anymore.

@nex3 Consider syncing with @jakemac53 or @natebosch - there was no need to dynamically add a loader for any other integration in google3 (Dart analyzer, DDC, Dart2JS, AngularDart, etc). A wrapper executable is likely to be what you'll need anyway - that's what TypeScript does internally, for example.

@leafpetersen and @lrhn do you imagine any work here?

[lrhn]

I don't see dynamic code loading into an isolate happening any time soon.

Our current compilation model does not lend itself to non-isolated code loading. Loading the code in an isolated way prevents type-compatability, and then you might as well use isolates on the VM anyway.

[miaosun009]

So has this thing progressed?

[dam0vm3nt]

No, they (dart team) has killed it (see the https://github.com/dart-lang/sdk/issues/10530#issuecomment-416782367) .

[miaosun009]

@dam0vm3nt Is there a better solution to the above problem?

[miaosun009]

@DartBot Did you give up on dart？😢

[dam0vm3nt]

@dam0vm3nt Is there a better solution to the above problem?

I've created a dart2ts compiler (https://github.com/polymer-dart/dart2ts) that can be used to produce a more "modular" code. It's usable even if it needs more testing, but you can give it a try.

[mraleph]

@miaosun009 what is your use case?

[miaosun009]

@mraleph Like most people, I look forward to this feature and am evaluating whether dart can be deployed in real projects

[miaosun009]

@mraleph If it's technically possible, why not? Dart users expect this feature

[mraleph]

@miaosun009 well, that's why I asked about the concrete use case you are evaluating Dart for.

For example, you might be using Dart on VM in JIT mode on the server. That's one situation - in this situation dynamic code loading is rather simple to implement and it would not have any negative impact on the performance of the code you are running.

But you might be also using Dart on VM in AOT mode (e.g. as part of Flutter) - in this case dynamic code loading is a much harder problem, as AOT currently assumes closed world knowledge about your program - and permitting your code to load other arbitrary code dynamically violates this assumption. This would have profound implications on performance and code size of your code. (performance would be worse and code size would go up considerably - because tree shaking becomes much harder).

[miaosun009]

@mraleph Thank you for your reply. My use cases are more about the server side, Dart vm, so will such features be implemented on vm in the future?

[mraleph]

@miaosun009 on the server side you can try using IsolateMirror.loadUri and see if it satisfies your needs.

[miaosun009]

@mraleph Thank you. seem to have found what I want

[Blquinn]

AOT dynamic class loading is required to develop a plugin system for desktop applications. This is now relevant due to flutter supporting desktop.

Currently it seems it would be impossible to implement dynamic plugins a la vs code or intellij.

[BugDiver]

I'm looking for this feature for a very long time, and it's disappointing that it's in a discussion knowing the importance of it for the community. I just don't get why the team does not think it's an important feature.

[mraleph]

I just don't get why the team does not think it's an important feature.

We understand very well that this feature might be important for some very specific niche use cases. However as explained in this comment dynamic class loading and AOT don't play well together.

So for the foreseeable future you have a choice of

• Deploying your applications in JIT, which support dynamic loading;
• Deploy your applications in AOT and isolating your dynamically loaded code inside a separate isolate or in a separate process (which is what VS Code does to their extensions).

[mraleph]

@BugDiver what is the use case you are looking at? Is it plugins or something else? Is it for Flutter or something else?

[BugDiver]

Is it plugins or something else? Is it for Flutter or something else?

It's a sort of plugin. the workflow is following (gist)

• The plugin is lanched by a Framework
• The framework and plugin communicate using RPC
• The plugin needs to load some Dart code (User code) when it's launched and perform reflection to get metadata of user code

You can think of as a test runner which needs to load the dart files (test files) and then reflect to collect the metadata

[mraleph]

@BugDiver from your description it does not sound like you actually need the same type of dynamic code loading as requested in this issue. Given that plugin and framework are already separated by RPC they are not required to exist within the same isolate or process. Furthermore, to extract metadata from Dart code you don't really need to load it, it is enough to parse it (using package analyzer or package front_end). Additionally it seems that you are targeting JIT mode specifically - so IsolateMirror.loadUri could work for you (though I would not recommend using it). Hard to say more given that I am missing a lot of details - but that's my assessment based on the information provided.

[BugDiver]

it does not sound like you actually need the same type of dynamic code loading as requested in this issue Maybe.

to extract metadata from Dart code you don't really need to load it, it is enough to parse it

Well, that won't be sufficient because I also need to get the methods and class instances. So static parsing is not what I need.

Given that plugin and framework are already separated by RPC they are not required to exist within the same isolate or process

Yes, the plugin and the framework are isolated from each other, but the plugin needs to load the User's code.

so IsolateMirror.loadUri could work for you (though I would not recommend using it)

That might work, but as you said it's not recommended.

For more details,

Things of it as a test runner which can be used for any language, given that there is a plugin written in that language. The test runner launches the plugin and later requests it to execute certain tests. The plugin's job is to load the user's code, perform reflection, collect classes, create instances, and get methods inside that.

Most of the languages provide some or other ways to do import dynamic code at runtime. Dotnet has AssembalyLoader, java has ClassLoader on the other hand dynamic languages like nodejs, python, or ruby allow you to directly import the libraries.

[mraleph]

Well, that won't be sufficient because I also need to get the methods and class instances. So static parsing is not what I need.

Ok, makes sense. I think you have three possible implementation strategies (sorted from least future proof to most future proof):

• Use IsolateMirror.loadUri to load dart code into the current isolate. Use dart:mirrors in the current isolate to reflect on it.

• Use Isolate.spawnUri to spawn dart code in a separate isolate and dart:mirrors to inspect it. You would need to create an auxiliary entry point file (you can pass it as data-uri to spawnUri), like so

  // entrypoint.dart 
  import 'actual.dart';
  import 'dart:mirrors';
  void main() { 
    // Use mirrors here to reflect on actual.dart
  }

• Combine parsing with code generation to avoid the need for dart:mirrors. Imagine for example that you have something like

   // test.dart
  
   class Smth {
     @test
     void foo() {
     }
  }

  You parse test.dart and figure out that you need to run foo on Smth because it is annotated with @test. Then you can generate something like this (I am simplifying - you can tweak code generation to suite your goals):

  // entrypoint.g.dart
  import 'test.dart';
  
  void main() {
  test("Smth.foo", () => Smth().foo());
  }

  Now you can load entrypoint.g.dart into a separate isolate and it will run the test. No runtime reflection necessary.

You can also use https://pub.dev/packages/reflectable to take care of code generation for you - this allows you to replace dart:mirrors without figuring out how to do parsing.

[BugDiver]

@mraleph Thanks for all the pointers.

Few questions

Use IsolateMirror.loadUri to load dart code into the current isolate.

Isn't IsolateMirror an abstract class? How can I create a concrete IsoalteMirror? (this might sound dumb, but If I extend I have to implement the loadURi) Or are you pointing me to use currentMirrorSystem().isolate?

• Second option is something which also came to my mind, but last time when I tried that, it was consuming quite a lot of time (proportional to the number of files I import in the auxiliary file). Will give it one more try

• the Third option seems a bit trickier and error-prone to me (Not sure I might be wrong)

[mraleph]

@BugDiver Yeah you need to use currentMirrorSystem().isolate.

For second option if you want to minimize time you can load them all into the same entry point, instead of generating multiple entrypoints per test file.

I think for the third option it is best to use something like reflectable to take care of parsing for you.

Anyway - I suggest we move this discussion to somewhere else to avoid spamming this issue further - feel free to reach me at vegorov@google.com or send mails to dart-misc mailing list.

[BugDiver]

@mraleph Thank you so much for the help.

[RaedDev]

@mraleph
But you might be also using Dart on VM in AOT mode (e.g. as part of Flutter) - in this case dynamic code loading is a much harder

As far as I can tell, this issue still lives within Flutter, even in JIT mode (debug mode), mirrors isn't an option to load code there. My use case is loading Widget class dynamically to a running Flutter app.

My proposed solution to this issue is compiling the code to dill without an entry point, which then can be loaded to the same isolate but then again this would only work inside a JIT runtime...

[mraleph]

As far as I can tell, this issue still lives within Flutter, even in JIT mode (debug mode), mirrors isn't an option to load code there.

Yes, mirrors are completely disabled in Flutter to avoid situations where people write code that works in debug builds (which use JIT), but breaks in release builds (which use AOT).

[cyberail]

@mraleph In flutter, I have following problem which I think is impossible to solve without dynamic module loading. let's say for example I have 1000 user in the app, there is a page called "my_widgets", each of these users have 10 widget on this page, (not flutter widget, just some complicated functionalities which contain flutter widgets to basically have visual) so it will be 10 000 completely different "widgets" for all of the users, if we integrate all of the 10 000 in one app, app size will be gigabytes, So we need to have them somewhere on the server, so each user will download only those "widgets" which they need, so we will maintain the app size smaller for downloading from playstore. if you can think of any way around will be great but I think it's not really possible without reflections and basically without dynamic module loading.

I often hear in github issues that dart and flutter staff really don't care about reflection and dynamic module loading(I read Hixie's comment about reflections that they were not needed at all) when I believe and I think no one will argue that, they are things which modern and powerful language should have, or otherwise it will be just another language failing to be as powerful as java or C#.

[ghost]

I did not know that. Really? Dart doesn't support reflection or dynamic module loading? Any serious language currently has these possibilities. I thought Dart would be a modern language for creating modern applications. A language in windows must be able to create dynamic modules, for example, a large application may have a manufacturing module, another accounting module, another billing module, another stock management module, .... Today's applications require modularity. If this is not possible with Dart I am seriously disappointed. I was thinking of adopting language as my preferred language, but if it does not have these possibilities, I will have to think about other languages. Please, I need to know if these features are going to be supported by Dart and Flutter to continue with them, otherwise I will have to abandon them. Thanks. Regards, Jose

[Sunbreak]

It would be possible if bytecode is still available: Remove bytecode generator

[ghost]

But bytecode is not necessary, for example the Delphi (Object Pascal) language is fully compiled and allows dynamically loading modules that can be used from within the main application as if everything were a single application. Modularization of applications is extremely important in large application environments that must load and unload memory modules depending on what the user is doing at the time.

Dart and flutter only good for small applications?

[cyberail]

@Josua2012 It supports reflection. I will be precise and say that, It does not support "static reflection" which means just to build the class by givin string as a classname to reflector. You need to have Type already defined, and visivle for reflector so it can create class from it. After that, you can do traditional reflection ops: adding methods and properties dynamically.

Other thing is that when you use reflection dart and its tree shaking freaks out and imports everything from every file, which makes executable huge in size. For that you are using reflectable package to anotate classes which you are going to reflect with @reflectable. So now it knows which class is going to be reflected so tree shaking kinda gets into shape, but I really don't think it is perfect solution.

Problem I was talking about, was about flutter, which does not support reflection in production cause it uses AOT compilation. And also it does not support dynamic module loading.

Note: Last version of flutter, specifically 2.13 was talking about deffered libraries which is supported only on android, which basically does the following, connects to playstore api and downloads needed library while you are opening your app first time. This way download time is less. To put it simply, you can have your libraries somwhere else and it will be downloaded and merged after using opens app first time, main executable or downloadable file only knows about urls that for those libs.

[cyberail]

@Josua2012 Yes they are good, language/framework for small and for enterprise if you don't need reflection. :)

[ghost]

@campfire5 I see the problem for large desktop applications on windows. Could you tell me where I can see an example that shows how a main application shows windows (Stateless and Stateful widgets) dynamically from other modules dynamically loaded in memory? The idea is to have the main application and have dynamically loaded modules that can be the warehouse management module, the accounting module, the human resources management module, ... Thank you very much for the help.

[cyberail]

@Josua2012 I don't really know examples for stateless or statefull widgets, but you can try reading Gilad Bracha's book dart programming language, one of the last chapters there are dedicated for the reflection.

Note: Some of the contents of the book is outdated but don't be discouraged because of that. I think it will be useful.

[cyberail]

@mraleph In flutter, I have following problem which I think is impossible to solve without dynamic module loading. let's say for example I have 1000 user in the app, there is a page called "my_widgets", each of these users have 10 widget on this page, (not flutter widget, just some complicated functionalities which contain flutter widgets to basically have visual) so it will be 10 000 completely different "widgets" for all of the users, if we integrate all of the 10 000 in one app, app size will be gigabytes, So we need to have them somewhere on the server, so each user will download only those "widgets" which they need, so we will maintain the app size smaller for downloading from playstore. if you can think of any way around will be great but I think it's not really possible without reflections and basically without dynamic module loading.

I often hear in github issues that dart and flutter staff really don't care about reflection and dynamic module loading(I read Hixie's comment about reflections that they were not needed at all) when I believe and I think no one will argue that, they are things which modern and powerful language should have, or otherwise it will be just another language failing to be as powerful as java or C#.

@Josua2012 I am quoting my comment, where I described problem for thousands of widgets, the best solution was with dynamic module loading, but as long as it was not possible my way around was the following:

Create a large map with all the widgets, you may need to load dynamically, but instead of literraly assigning widgets to Map's values use closures instead. So as long as closure is literal representation of the code, it does not instantiate the classes inside it until it is called. In this way memory will not be exhausted.

E.g sudo code:

Map<String, Function> wMap = {
"first_widget":(BuildContext context) {
        return Container(....)
    }
......
......
}

// And usage will be
.....

...child: wMap["first_widget"](context);
.....

in that case Container() will be instantiated after calling wMap["first_widget"](context)

More about closures: https://en.m.wikipedia.org/wiki/Closure_(computer_programming)

Also You can read more about specifically on dart's closures in Gilad Bracha's book "dart programming language"

[ghost]

@campfire5 If I have not misunderstood, with the system that you propose to me, all the application code would still have to be in the main application, although the objects are only going to be created at the time of use. If what I have understood is correct, I believe that this is not the solution, since what is sought is that the main application is not very bulky.

[cyberail]

@Josua2012 Yes you understood correctly, I mean main idea of that is that memory will not be exhausted while rapp is running.

[cyberail]

I think @mraleph 's Thoughts would be great to hear about it.

[ghost]

If the objects are handled correctly, you should not have a memory problem. What amazes me is that the Dart team did not include a good system for dynamically loading modules. All languages today contemplate this possibility, since it is fundamental for modern applications. 👎

[ghost]

Yes, it would be very interesting to know the thoughts of @mraleph :)