Dynamic module loading

[DartBot]

This issue was originally filed by @dam0vm3nt

---

This feature was initally described in

Issue #3819: Provide a Method to Dynamicaly Load Classes

But it has been closed pretending it was done. IT IS NOT as Mr. gbracha honestly admits: "... where the code being loaded is not known statically. This will require the ability advanced capabilities like mirror builders. This should be a separate issue."

So I'm posting that separate issue hoping sometime in the future will be adressed as it is the only reason stopping me to adopt dart.

In other word: We need some mechanism to dyamically load classes not know at compile time by another module. Something like java class loaders / Flex Module / JS AMD.

[madsager]

Added Area-Language, Triaged labels.

[DartBot]

This comment was originally written by @seaneagan

---

Have you seen DeferredLibrary yet:

http://api.dartlang.org/docs/bleeding_edge/dart_async/DeferredLibrary.html

is it tracked in issue #3940. it's currently limited to 2 files, but that should be changing soon.

[DartBot]

This comment was originally written by @dam0vm3nt

---

That's great. But what do you mean with 'limited to two files'?

I'll do some experiment.  Il giorno 08/mag/2013 19:34, <dart@googlecode.com> ha scritto:

[DartBot]

This comment was originally written by increo...@gmail.com

---

You can load code not known at compile time through Isolates and spawnUri().

[DartBot]

This comment was originally written by @dam0vm3nt

---

­4: I already knew you can use Isoletes and spawnUri, but this is not the solution because:

1) loading an external library dinamically doesn't necessarily implies we need to spawn a new thread 2) not needing concurrency comunication by the loading and loaded modules should be possibile in a simpler way then using Send and Receive Ports. For example after loading a module it should be possible to call a function defined in that module passing generic objects

[DartBot]

This comment was originally written by @dam0vm3nt

---

­2 : I've checked the pointer and this is noy (YET) what I mean. Infact the compiler still needs to know the library at compile time, hence the need of a constant constructor (thus not allowing for truly dynamical library loading) and declaring an import statement (that requires to know the module at compile time).

So I've proposed some changes to that feature and posted a comment on that thread.

For commodity I repeat here the example that should work to satisfy this issue:

queryServerForLibToBeLoaded().then(loadIt);

loadIt(libName) {  dynamicMod1 = DeferredLibrary(libName);  dynamicMod1.load(moduleLoaded); }

moduleLoaded(loadedLibraryMirror) {  loadedLibraryMirror.classes[libName+".Plugin"].invoke("initPlugin"); }

[DartBot]

This comment was originally written by @stevehsu77

---

i propose an easy definition of dynamic load.

======================================================================== typedef int function(String arg1,int arg2);

funntionmirror=reflect(function); functionmirror=loadsource(uri);  -or- functionmirror="int (String localarg1,int localarg2){ return int.parse(localarg1)+localarg2; }"

var result=functionmirror.invoke(arg1, arg2); =======================================================================

so that it doesn't need to load a new library and handle dynamic type processing.

[DartBot]

This comment was originally written by jirkad...@gmail.com

---

Going forward, do we really want to use this? There is much[1] talk[2] recently[3] about the security model on the web and being able to dynamically inject stuff into the code of the main app (which is easy in JS) is being frowned upon. XSS, you know. There is even the extreme case[4] of some academics trying to use the first and second W3C standard [1,3] I mentioned to implement in JS something akin to Dart isolates.

As an alternative, the current isolate model could be made more developer friendly by making some API wrapper or so. There is something in these lines in Java. It could be made with the security model of Least Privilege and Information Hiding in mind that is being proposed[2, 4, 5] for the web.

[1] The sandbox attribute http://www.whatwg.org/specs/web-apps/current-work/multipage/the-iframe-element.html#attr-iframe-sandbox [2] Douglas Crockford: Really. JavaScript. http://www.youtube.com/watch?v=lTWGoL1N-Kc , from 37:00 [3] Content Security Policy 1.0 http://www.w3.org/TR/CSP/ [4] Privilege Separation in HTML5 Applications www.cs.berkeley.edu/~devdatta/papers/LeastPrivileges.pdf [5] The Lazy Programmer's Guide to Secure Computing http://www.youtube.com/watch?v=eL5o4PFuxTY

[DartBot]

This comment was originally written by @dam0vm3nt

---

I think this is, not only desiderable, but also of extreme importance for Dart success.

Dart needs a way to create extensible and modular application. Isolates can actually be used but they are a very poor mechanism when it comes to intra-module communication.

Let's make an example. I have a project that consist in a new fantastic social network that's written in DART. I want thirdy party to contribute with they're fantastic App. But I want them to seamlessly integrate to the main framework. If I were using in any other programming language (Flex, Java or even JS) all I have to do is to provide the contract API, i.e. a set of classes to extend and interfaces to implement so that a thirdy party can implement their App. Then the thirdy party could upload their module (preferably in a "compiled" form) and their app could be installed and executed - just because it is conformant to the contract API. The framework in turns could simply load the compiled code and run it (for examble in Java we have Classloaders, in Flex URLLoader, in JS a plethora of api).

If I'm not missing something at the moment if we want to implement this simple and common pattern (i.e. a framework - plugin pattern) with dart we have only two choices:

1) use the power of OO approach and provide a contract API in terms of classes and interfaces and not using isolates. In this case we need to RECOMPILE ALL THE framework every time we have to integrate a new plugin. This also means that we cannot provide an authonomous and automatic way for a thirdy party to add their plugin but there must be a human procedure. Unfeasible.

2) specify the contract using some sort of "communication protocol" and using Isolates. We cannot provide classes to extend because there's no way an isolate could return an object to the framework implementing some sort of contract. Isolates can communicate only "simple data" and using "copy by value". This is a very poor approach, complicated, error prone. But the main defect IMHO is that we are restricted to use "copy by value" approach making difficult to pass big data structures and to share implementation across different peer plugins or between the framework and the plugin.

I hope I'm wrong and someone could teach me a third way because so far this is the only reason for people like me and my company for not to adopt DART now and until this is fixed. Don't think we are the only one expecting this.

Security should not be an excuse for not implementing important features like this. Otherwise the most secure language is the less powerfull.

[DartBot]

This comment was originally written by @dam0vm3nt

---

Maybe this example is more clear of what IMHO it is needed:

// We call our backend and ask for a module to load queryServerForLibToBeLoaded().then(loadIt);

// Backend provides us with the url of the lib to load loadIt(libName) {  dynamicMod1 = DeferredLibrary(libName);  dynamicMod1.load(moduleLoaded); }

// The module is loaded we can create object using classes defined // in the module that extends classes defined in the calling module: moduleLoaded(loadedLibraryMirror) {  var myObj: MyClass = loadedLibraryMirror.classes[libName+".Plugin"].create();  var myArg : MyOtherClass = new MyOtherClass();  // AND USE IT:   myObj.doSomething(myArg); }

[DartBot]

This comment was originally written by @stevehsu77

---

It's concerned about class run-time types while loading whole library. And another issue is refer to other libraries during compiling it. Think about this ============================================================= import 'aaa/bbb/ccc.dart'; //but we don't have ccc.dar under aaa/bbb

class a extends b {} //but b is in another dynamic library we just loaded

var myObj = loadlibrarymirror.class['lib.a'].create(); // there are two problems // 1. is an independent mirror system differ from current mirror system? if yes, how to use the class in current mirror system in dynamic load mirror system? // 2. if it's not, the current mirror system seems to have to load a chain of libraries and consider about conflict of naming, because we won't give the alias name before we use them. =================================================================================== as in my project, i want to pass a class/structure definition to the class which is dynamic loaded and return the class/structure in what i defined the other dynamic library , so, it must import my class/structure definition. it will cause many problems.

because dart is not a pre-compiled language, it does't have any header file or dependency file. and it use language-based vm, does't own any intermediate language like as bytecode or msil.

many things have been done when vm is starting. that's why i assume loading dynamic library will make vm developers to think more about security, performance, convenience, and more then more.

and that's why i propose an easy way to load dynamic functions. it does't to be considered about type and many other things in vm. and easy to modify code in vm in 3 steps. (load source string, parse to object, instance it)

---

typedef int function(String arg1,int arg2); //as a delegate, supported now

closuremirror=reflect(function); //reflect it, need a little modify closuremirror.loadsource(uri); //load realcode from web, filer, etc.  -or- closuremirror.loadsource("int (String localarg1,int localarg2){ return int.parse(localarg1)+localarg2; }"); //load realcode from string                                             //add it in mirror system var result=closuremirror.apply(arg1, arg2); //run it

don't do think too much about runtime-typed, memory heap, garbage collection. and etc. and then support dynamic function, so that we can wrap it as a module.

[DartBot]

This comment was originally written by @stevehsu77

---

believe i would so strongly except dart can support dynamic library. so i can move my c# and as3 architecture to dart. and i was trying to do this. i built everything is class in the project (https://github.com/yunist/yun). but i change my mind since dart 1.0 (there are many changes of mirrors and isolate), i think everything is object in dart as js. so i define structure use map, ans use class as interface(like go language) in the new project (https://github.com/yuner/surebet). now dynamic function is enough for me. dart is not like other language, you need redesign your architecture to suit it.

[DartBot]

This comment was originally written by @dam0vm3nt

---

Another common usecase where we absolutely want this is : a main module that loads a dynamic module that creates a web component to be returned to the main module.

This is not possible with current mirror/isolates.

Example:

the main module implements a browser of a collection. when clicking an item we want to dynamic load a module that will create an editor for the item to be displayed on the current page in a specific div.

[DartBot]

This comment was originally written by @stevehsu77

---

code as below ============================================================== class Dynamic_Class {     Name String;     Uri Source_File;     Map<String,dynamic> _inner_Mirror={};

    Dynamic_Class (this.Source_File)     {
        String source_definition=_load(Source_File);         _parse(source_definition);     }     String _load(Uri source_file) { //implement code - read json file to string}     String _parse(String source_definition} { //implement code - put to _inner_Mirror}     Function _parse_a_function(String function_source_code) { //.... }     get(String function_or_variable_name) => _inner_Mirror[function_or_variable_name];     void set(String function_or_variable_name, dynamic function_or_variable) { //...     dynamic invoke(String function_name) { //... } } ==================================================================================== json definition =================================================================================== {
   "@class_name" : "my_dynamic_class",    "_local_field" : 123,    "public_field" : "456",    "_local_method" : "dynamic (List arguments){ ... }",    "public_method" : "dynamic (List arguments){ ... }" } ================================================================================= so that we don't need to generate new structure(class type), that will be more easy for modifying vm code.

even you can implement different code for each div by special "class_name";

Map<String div_class, Dynamic_Class> div_wrapper;

is it easy? for vm developer , it's very easy to do this modification.

[DartBot]

This comment was originally written by @stevehsu77

---

and i will put dynamic_class to an independent isolate to manager all dynamic classes for security. dynamic code may cause many injection problems in web apps. for a "div" application, we often listen it's events, so pass the event to another isolate, and then process it in dynamic module, finally, use massage to transfer the result to main isolate will keep everything under control and safety. however, it will cause lots of memory-copied actions and duplicated data. but, vm use a solution like as snapshot(not sure) to solve this problem.

[DartBot]

This comment was originally written by nigel.magn...@gmail.com

---

I'm not sure Isolates, or the definition of DeferredLibrary is the answer.

For a relevant example in the JS world, look at this: https://github.com/dluksza/angular-gerrit - client-side modules, integrating into a bigger application. Comment #­9 is spot-on. Many applications - ones that dart is a good fit for - need this kind of modularisation ability.

Now, I don't really mind if the modules themselves suffer from 'poor minification'. In the (say) Java world, I might load a module into a new classloader, and be able to reference classes from my 'parent' - without having to bundle those classes all over again. I can see how that would be extremely difficult in the Dart case.

But at the moment - somewhat embarrasingly - if I want to reference dart 'module A' from 'dart application B', about the only tractable solution is to compile moduleA entirely into JS, then use the JS interop to access it (!). Yuck!

It would be nice to have even an iterim solution, otherwise dart's going to be a non-starter :-(.

[DartBot]

This comment was originally written by danil.kornishev@gmail.com

---

In Ruby/php you can import a file from anywhere in the code. Why can't I do this in dart?

I'd like to be able to scan a directory and import all dart files found, for example in my dart cucumber project. That seems not possible. Had to resort to generating a dart runner file with all the found files imported and using spawnUri.

[gbracha]

Set owner to @gbracha. Added Accepted label.

[DartBot]

This comment was originally written by imre.gabes...@gmail.com

---

I'm looking forward to this feature extension. Currently this shortage is the reason why we use JS instead of Dart. And JS is way uglier and slower than Dart but at least that's dynamic...

[trentgrover-wf]

any update on this?

[bergwerf]

+1, I really need this to be able to use Dart for my project instead of ES6/TypeScript.

[sethladd]

@hermanbergwerf thanks for the comment. Could you provide specific details on exactly what problem you need solved?

[bergwerf]

@sethladd I want to build a plugin framework where plugins can be executed during runtime so I can move features out of the core into separate modules. Additionally, third party developers will be able to develop and run plugins. In JS I could inject the plugin script which could then access a global object from the core that holds an API. It seems spawnDomURI from Dart can achieve this too (I think I could pass the API object via args), but there is no dart2js implementation yet. I suppose the compiled dart2js could also be injected and the plugin could import the core so it can use the API object (which is really just a class with a lot of functions, this should be passed by reference though).

[sethladd]

@hermanbergwerf thanks! Just for clarity, do you need this only for your web apps, or are you writing VM-based apps, too?

Also for clarity, it sounds like you need to dynamically load a library into an existing isolate, such that the loaded library has access to all the same memory as the isolate it was loaded into?

(where, isolate == app or web app, in this case)

cc @floitschG

[bergwerf]

@sethladd Initially only for the web version of my application (http://molview.org, FOSS!) but I'll also run parts on the server for pre-processing. The goal is indeed that a piece of Dart code (not sure if it should be a library) can access the same memory (mainly in order to access the DOM). The injected Dart must be able to communicate with the core. Let's say the core is a single instance of a class where the public methods are the core API. My idea was to pass a reference to this core object so the injected code can interact with the core. Perhaps this is a little to much JS-style. Some kind of messaging would also work. It makes no sense to run the injected code on another thread, so I think that means it should run inside the same isolate. Especially since the injected code should be able to alter the DOM. It would be even better if that could be sandboxed to a specified DOM node (I think the only way would be to use a kind of DOM node proxy, but that's just a wild guess).

[frank-rollpin]

Let me echo @hermanbergwerf 's comments, I am looking for exactly the same thing. The idea is that scripts (written in Dart right in the browser in my app) would be able to interact with the core, using the established API. This is pretty much the way plugins (and scripts) are usually implemented in other languages, like C# or Java. The loaded code should operate in the same thread and should have access to all DOM, so isolates are not a solution. Would be awesome to have the same capability in Dart! Plugins can actually be built with the rest of the system, but for scripts dynamic loading (in my case, this will involve dart2js transformation) is necessary.

[trentgrover-wf]

I will jump on this bandwagon while it's moving as well:

This sort of plug-in mechanism would allow us to break up deployment and iteration on complex apps in Dart by permitting us to update an isolated code module and deploy it without having to rebuild and redeploy the entire application. This also opens up some nice possibilities for A/B testing of these sorts of things.

Obviously supporting this would have some potentially large impacts on how Dart handles dependency resolution and dart2js bundling of code (deferred loading).

[bergwerf]

@sethladd Any progress on this issue?

[sethladd]

Any progress on this issue?

Nothing right now, we're getting ready to ship 1.12. You might want to reach out to web@dartlang.org mailing list , because the new dev_compiler (aka DDC) might be able to make it easier for you to dynamically load chunks of Dart code into your app.

[bergwerf]

The dev_compiler does look very interesting!

[dam0vm3nt]

Well, I think compiler_dev is the answer. Wonder if we will have to wait until ES6 becomes fully implemented or if they will make an ES5 driver too.

2015-08-24 20:25 GMT+02:00 Herman Bergwerf notifications@github.com:

The dev_compiler does look very interesting!

— Reply to this email directly or view it on GitHub https://github.com/dart-lang/sdk/issues/10530#issuecomment-134327271.

[zoechi]

You can use ES6 tools to transpile dev_compiler generated ES6 to ES5. I think Babel is such a tool there are probably more but I do not concern myself with this topic and my knowledge is sparse.

[dam0vm3nt]

Yeah I know that. At the moment this is an option but IMHO more for "peeking" what is going on. There are many ES6 -> ES5 compilers, but this kind of solution is going to be very hard to debug and also a comple workflow for building. For real use, unless there's a direct ES5 driver, I think that one should wait until ES6 become ready on major browsers.

2015-09-13 16:01 GMT+02:00 Günter Zöchbauer notifications@github.com:

You can use ES6 tools to transpile dev_compiler generated ES6 to ES5. I think Babel is such a tool there are probably more but I do not concern myself with this topic and my knowledge is sparse.

— Reply to this email directly or view it on GitHub https://github.com/dart-lang/sdk/issues/10530#issuecomment-139879198.

[fmorgado]

Hey guys,

Any news on this?

A new project just came in, a custom CMS which requires the server-side execution of user-defined functions on specific events (beforeRecordSave, aftertRecordSave, beforeRecordDelete, beforeStateTransition, custom record actions, etc). Security is currently not a problem.

I'm growing fond of my Dart CMS and would love to extend it to fit the bill. But the lack of dynamic loading is currently a no-go.

I'd love having feedback soon. We could afford to wait up to 6 months for this feature, while we work on other requirements. But if it's not coming, we'll have to start in Java or C# right now.

Thank you :)

[dam0vm3nt]

Take a look at the DCC (new developer compiler) : it will handle different styles of module loading. But at the moment it is still not ready for the web.

[fmorgado]

I need dynamic loading in the VM. I'd like to use Dart server-side, not for the browser (at least currently).

Hot-reloading is currently being implemented, it may be related. And Observatory already supports dynamic statement evaluation (not sure to what extent).

So maybe it wouldn't be that much work.

What I'd like is to evaluate a string at runtime and have it compiled to a function or class, in the VM. Or maybe load a dynamic library (similar to .dll, .so) and get a list of all top-level declarations.

[jodinathan]

I am willing to use Dart ServerSide on most of my projects and projects of companies that I can choose or suggest to. However, Dart needs to evolve faster =\

[dam0vm3nt]

Ddc can produce modules and code for node if you want to run it serverside.

But you are right. Actually the dyn mod is a feature of the build system and not the language.

[fmorgado]

I can't stand anything related to JS and the last time I tried node, it was too much of a mess for my taste.

In the VM, we could run dynamic code in an isolate, with some glue for communication. But we need to include dynamic DB access inside the caller's transaction. We could provide DB services to the dynamic code which goes back to the original isolate. But it all seems overly complex, brittle and inefficient.

Sadly, given there's little interest on this feature, we choose another language stack for the project.

[dam0vm3nt]

Js would be only the finalmente result you would see only Dart... But understand what you are saying. Sometimes I feel frustrated too

[fmorgado]

I'm just philosophizing here (I've had a glass or two) ...

Maybe we could avoid the "wild west" of dynamic loading by implementing something similar to deferred loading.

import 'package:dynamic/loading.dart' deferred as myLib; await myLib.loadLibrary(); myLib.doSomething();

Maybe something like the following would possible, without turning the world upside-down (I guess it would :) ...

// Only defines the API a dynamic library must adhere to. import 'package:dynamic/loading.dart' deferred as myLib; // Specifies the path to the (dynamically generated) library file. LibraryMirror myLoadedLib = await myLib.LoadLibrary('lib/myDynamicLib.dart'); // We need a way to cache the result (and maybe unload it). // Use the dynamically-loaded library using the mirror myLoadedLib ... // ... but it would be a lot more user-friendly to treat it just as an import prefix (or an instance ...). myLoadedLib.doSomething();

Since Dart is transitioning from scripting to an AoT-friendly language (Dart 2.0?), I think it's important to reconsider the import/export scheme and how dependencies are looked-up. (I like how Rust handles imports and lookups).

Conceptually, I don't see much of a difference between a library (or package in Java/AS3/etc) and a singleton. Libraries only differ to singletons in minor ways, mostly due to language/runtime restrictions.

// An hypothetical language ... package lib1 { class Lib2Class { void doIt() { ... } } final MyClass lib2 = const Lib2Class(); package lib3 { void doIt() { ... } } class lib4 { static void doIt() { ... } } }

I don't see much of a difference between "lib1.lib2.doIt()", "lib1.lib3.doIt() and lib1.lib4.doIt()" (except for namespaces, "this" within methods, which could resolve to the current library if static, and the AoT/runtime lookups). Basically, libraries are self-defining AoT-friendly singletons.

I think of a dynamically-loaded library as an instance of a class which implements the static API of a library (or package).

We already have the concept of static APIs with deferred loading and configurable imports. We could just "allow" instance implementations of those static APIs ... and dynamically load them ...

Of course there are corner-cases ... I believe a dynamically loaded library/instance should have access to all/some packages from the importer. We should be able to define a path where to lookup for missing dependencies. ... a lot more I'm not aware of ...

Thoughts?

[nex3]

I'd like to push for this, specifically on the VM. The test package is getting closer to the point where it needs to load user-defined plugins, and pub's isolate-based plugin loading has convinced me that that approach ultimately isn't usable. Needing to manually serialize and deserialize all APIs to let them pass through the Isolate API is a huge productivity tax and sometimes a performance tax as well.

I'm hoping the recent work on hot reloading makes intra-isolate dynamic loading more feasible. Here's an example of the sort of API I'd like to see:

/// Loads the Dart file at [url] in the current isolate, invokes the top-level
/// function named [functionName] with [argument], and returns the result.
Object loadDartFile(Uri url, Symbol functionName, [Object argument]);

This would use the current isolate's package resolution. If it's invoked from a snapshot, it would use the snapshot's location to find a .packages file, but it would load snapshotted libraries if possible.

[dam0vm3nt]

I think the best way to go with this is raising the dartdevc from "dev tool" status to fully "production oriented status". dartdevc infact already supports dynamic loading modules as it leverages js modules (AMD, ES6, ecc.) and summaries for symbol resolutions. So I hope they will follow that path.

To better understand what I'm saying you can take a look at this and this.

[bergwerf]

@dam0vm3nt this would be very useful for other purposes than web development.

[dam0vm3nt]

@bergwerf I agree 100%. For example @nex3 showed another use case. I'm trying to push for this with the dart team. If you believe that this could be an important feature you can do the same in the gitter channels or the forums.

[fmorgado]

I'm so rooting for this in the VM, even if not implemented for other targets.

@nex3, in your example, it seems it would be required to reload the URI if we plan to execute the method a second time.

I think it would make sense to load the URI first into an object, then call whatever methods whenever we need to.

var object = loadDartFile(myUri);
var result = object.invoke(myMethodSymbol, myMethodArgument);

We could try to make the module adhere to a custom interface:

T loadDartFile<T>(Uri url);

Then:

abstract class MyExternalService {
  void connect(String argument);
}
var service = loadDartFile<MyExternalService>(myUri);
service.connect("meh");

Loading from bytes would be a really nice extra:

Object loadDartBytes(List<int> bytes);

[nex3]

@fmorgado You can always have the method that's called return a function that can be called multiple times. Another option would be to allow the symbol to refer to any top-level name, so it would return a function object that you'd have to manually call.

[fmorgado]

@nex3 True. I didn't see it that way. We'd call the "module main" which would return whatever methods/interfaces we'd need. Seems cleaner.

I insist that loading dart code from bytes would be very useful. May not mean much for CLI tooling but it would on the server, where filesystems are increasingly obsolete.

[4stern]

hi,

i have another case for this feature: currently i need an migration-lib to handle database migration within my project. that means i have to load files dynamically (from a specific folder) and execute one or more functions from it ... up to now i found no mechanism to do this (in nodejs its very easy to load a file and execute stuff from it).

:(

[jodinathan]

That could be done with isolates, no?

2018-01-25 16:24 GMT-02:00 4Stern notifications@github.com:

hi,

i have another case for this feature: currently i need an migration-lib to handle database migration within my project. that means i have to load files dynamically (from a specific folder) and execute one or more functions from it ... up to now i found no mechanism to do this (in nodejs its very easy to load a file and execute stuff from it).

:(

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dart-lang/sdk/issues/10530#issuecomment-360555107, or mute the thread https://github.com/notifications/unsubscribe-auth/AF6Y3nMB1-y4O2_dqi4ReuM5a17h4JVcks5tOMbOgaJpZM4E3_KU .

-- Jonathan Rezende