Request for feedback: Discontinuing support for dart:mirrors

[mit-mit]

The Dart mirrors core library (dart:mirrors) is currently in a sub-optimal state:

• It is marked "experimental" in the list of core libraries

• It is marked "unstable" in the API docs

• It is unsupported in the Dart web platform

• Is is disabled from use in the Flutter framework (and has been so for years)

We've investigated potentially making mirrors a stable, fully supported offering, however in its current form this has a few large challenges:

1. Dart has powerful "tree shaking" for ensuring we can produce apps with small code sizes. This relies on the ability to detect unused code, which doesn't work if reflection can use any of it dynamically (it essentially makes all code implicitly used).

2. The APIs supported by mirrors are significantly behind the current Dart language, for example they do not support extension methods or null safety.

We believe that one of the core uses of mirrors is to do various forms of code generation and meta-programming. Before we potentially fully discontinue mirrors, we'd like to better understand the desired use cases here so that we could have alternative solutions and migration path ready for current users before we make any changes to dart:mirrors. If you have any such use cases, please leave a comment.

[jakemac53]

We used to have Dart Native as a platform, so that at least it would show up as a valid use case? The current thing is way over indexing to flutter to the point where it implies that Standalone Dart doesn't even exist.

[isoos]

@bradcypert: what is the package that has this report?

[jakemac53]

@isoos https://pub.dev/packages/build_runner/score is an example of this, previously we at least only got partially dinged for not supporting the platforms that made no sense to support with a command line app. Now we get 0/20 as well.

[bradcypert]

@bradcypert: what is the package that has this report?

@isoos https://pub.dev/packages/steward but I have a few other packages that use reflection, too.

[JakeSays]

@mit-mit How would adding something like a single "dart only" checkbox complicate the pub.dev ui?

[II11II]

Hi 👋, I do not have deep knowledge about dart:mirrors, but would like to know. Is it possible to create kind of interface which allows use mirrors or code gen tool depending on platform that the programmer builds his app? F.e. If a programmer used mirrors in flutter, flutter will generate code depending on usage of mirrors and will replace mirrors code by generated code F.e. or user can select tool(mirrors or code gen) while he compiles his app

dart compile myapp --code-gen dart compile myapp --mirrors

[eernstg]

generate code depending on usage of mirrors and will replace mirrors code by generated code

The package reflectable contains a code generator which will very nearly do this. However, due to the fact that it relies on code generation there are certain reflective features that aren't and can't be supported. E.g., obtaining the actual value of a type parameter in an instance of a generic class (we have a list, so we know that it has type List<T> for some T, and we want to look up the actual value of T — generated code will never be able to do that, unless Dart is extended with some new features to perform such lookups at run time). Examples here.

[gmpassos]

At runtime you can get the value of T with:

class Foo<T> {

  Type get theType {
    var t = T ;
    return t ;
  }

}

Also you can use on extensions:

extension ListExtension on List<T> {

  Type get theType {
    var t = T ;
    return t ;
  }

}

Is this that you need?

Regards.

[d3roch4]

I have a suggestion:

Reflection can be static. This discussion also exists in C++, maybe we should look at how other languages ​​face this problem, there are several proposals in C++: https://isocpp.org/files/papers/n3996.pdf https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0194r6.html https://github.com/veselink1/refl-cpp

Most suggest using metadata built at compile time to get the reflection to work, if dart going that way would be a good way to go.

[lrhn]

@gmpassos

Neither approach can actually get the runtime type parameter of a List<T>.

The getter requires the ability to add real instance methods to the class, and you can't add methods to List.

The (static!) extension method only captures the static type, not the runtime type.

There is currently no general way in the Dart language or platform libraries outside of dart:mirrors to access the runtime type arguments of an instance of a generic class, not from the outside.

[marcotas]

I also want to share my use case of dart:mirrors (although nothing is in production yet).

I'm building a server-side framework in pure dart to serve all front-end flavours (flutter, vue, react, svelt etc). So the framework trusts dart:mirrors to inject dependencies based on types.

For example, if you define a controller like the below, you will have the request and repository objects injected into your controller method automatically based on mirrors that recursively try to resolve all the dependencies if you didn't register them in the service container. This is pretty much the same as Laravel framework (my background) does with PHP.

class UserController extends Controller {
  updateAmount(int id, Request request, UserRepository repository) {
    ...
    await repository.updateBillingAmount(request.get('amount'));
    ...
  }
}

IMHO, there's no problem with removing dart:mirrors as long as an alternative with reflection capabilities is provided without code generation for the same reasons shared above by @daniel-v, @zmeggyesi, @gothamgasworks, @DavidLiedle, @rdnobrega just to name a few.

[maximveksler]

I would like to offer a step towards a discussion of what might contribute to finding a solution. Taking the example from Swift which is a statically typed, type safe language. The implementation of Json serialisation (as an example of a typical reflection use case) was achieved using Foundation SDK language support and compile time code generation.

The design of which can be reviewed here https://github.com/apple/swift-evolution/blob/main/proposals/0166-swift-archival-serialization.md

[mraleph]

@maximveksler I am pretty fond of Swift's design for this feature and have been advocating something like this for years (most recently in context of generalising some of proposed data classes features to work on normal classes: https://github.com/dart-lang/language/issues/2372). I have hopes that we will eventually converge on a set of interfaces for which compiler will autoderive method implementations.

That being said this approach is certainly less expressive than mirrors it's not really compile time code generation in the sense of macros, but rather a bunch of logic hardwired into the compiler which user can't fully customise if they need.

[bradcypert]

@maximveksler I'm under the impression that the Dart team is working on static metaprogramming which should encapsulate that scenario.

[lectrician1]

My use case for reflection is reading from and creating classes I've defined for the grammar of the SPARQL query language so that I can create visual programming SPARQL generator app.

[Silentdoer]

Mirrors can be used to create special applications, such as Java's drools rules engine framework

[ykmnkmi]

My use case for reflection is generating tools from classes.

[cheogm28]

Hi I loce Dart and honestly my favorite part of it is mirrors I have a Library that is based on shelf and mirrors, and I think that the use of mirrors improve it a lot but people are kind of afraid because it says mirrors on the dependencies , but I think that they work very well the url of the library is https://pub.dev/packages/annotated_shelf

[jakemac53]

@cheogm28 That package looks pretty doable through existing codegen or the proposed macros (it might look slightly different but not much).

[sigurdm]

Indeed, look at https://pub.dev/packages/shelf_router_generator for something similar.

[cheogm28]

@cheogm28 That package looks pretty doable through existing codegen or the proposed macros (it might look slightly different but not much).

yes macros are ok but I think not ready yet :) . I am using the reflectable to see if I can port the package. But I also think that mirrors have a lot of applications just look at other languages like Java they use it a lot and it would be a shame if we can not take advantage of it using this beautiful language. ✌🏽

Indeed, look at https://pub.dev/packages/shelf_router_generator for something similar.

Similar but not the same (specially how it uses context). It is very good package and it is amazing to see pople are trying to do something with the backend in dart which is very excited.

Also I just want to thank you contributors for make it possible (🎯 = 💙)

[jakemac53]

yes macros are ok but I think not ready yet :)

Yes, this issue exists largely to track the use cases for mirrors today, to drive the necessary features that we would need from macros/codegen/etc in order to one day drop support for mirrors (or determine it wouldn't be possible to do so).

[bradcypert]

@jakemac53

I maintain a server-side framework, and the main reason I chose reflection over code gen was to simply not force the user to think about generating code. Configuring build runner is a pain, and for newer devs it can be very confusing.

All I would need to drop reflection support would be static meta programming. I understand that there's definitely other use cases out there where reflection would be harder to leave behind though.

Edit: I've also considered maintaining my own CLI that would take care of the codegen for you -- I'm still not sure how I feel about this, but it's something I've been exploring.

[Nexushunter]

I would love to be able to retain the dart:mirrors library. I'm part of a group that is working on providing an OpenAPI generator annotation that allows for a config to be built in dart and then generate the library based on it. Currently we provide a way to pass headers into the generator for privately hosted specs. But it isn't amazing (read: very naive). We are providing a generic delegate class that can be extended and then passed in. Due to the nature of being able to provide both a positional argument and named args in a subclass constructor it becomes difficult to provide support for reviving that object. I've opened this PR within source_gen to begin allowing for that but that implementation relies on dart:mirrors to properly rebuild the object. While it is a WIP it makes sense for builders / generators to be able to use a fully qualified version of their annotation as needed.

[jakemac53]

@Nexushunter it seems to me like you could probably code generate everything you need to be able to revive those constants?

[Nexushunter]

@Nexushunter it seems to me like you could probably code generate everything you need to be able to revive those constants?

@jakemac53 I'm not looking to output it to a file 🤔 I'm looking to do an in memory revive. I think I might not be understanding what you mean. Can you expand a bit?

[jakemac53]

@Nexushunter it seems to me like you could probably code generate everything you need to be able to revive those constants?

@jakemac53 I'm not looking to output it to a file 🤔 I'm looking to do an in memory revive. I think I might not be understanding what you mean. Can you expand a bit?

For each constant you are trying to revive (each class used as an annotation), you could run a separate code generator (or eventually, macro), which would generated the code for reviving that particular class. Similar to generating a toJson/fromJson for a class.

[Nexushunter]

@jakemac53

Ahhh I see what you're saying. In this case I don't think that would be viable.

It seems that way to me because:

• consumers are potentially going to be extending our implementation and the builder shouldn't be forced to rely on externally generated code
• source code generation shouldn't be needed to restore the object to it's original form when all of the required data is present in memory already
• it feels hacky to generate source code for something we already have the source for and just want an in memory representation based on the serialized value provided.
• the solution to this problem only needs to be implemented once and should be available within the core library since the processing is being provided in the core library.

While there are lots of amazing use cases for generating source code this doesn't feel like one. We have the source and can rebuild the object in memory based on the DartObject and dart:mirrors fully enables this.

I would rather see an updated and stabilized API over the removal because there are valid use cases.

[jakemac53]

• consumers are potentially going to be extending our implementation and the builder shouldn't be forced to rely on externally generated code

If you consider a sufficiently well integrated code generation mechanism (part of the compiler) I think most of these concerns go away.

• source code generation shouldn't be needed to restore the object to it's original form when all of the required data is present in memory already

But if we were able to remove dart:mirrors, that required data would no longer necessarily be present.

• it feels hacky to generate source code for something we already have the source for and just want an in memory representation based on the serialized value provided.

The "source code" in this case is actually a very abstract representation, not really any different from JSON or some other serialization format. The types you see are not even necessarily available in the current isolate and thus impossible to rehydrate at all.

So, I would actually say it is less hacky to rely on the explicitly generated code to rehydrate very specific known types.

• the solution to this problem only needs to be implemented once and should be available within the core library since the processing is being provided in the core library.

I agree this is the advantage of mirrors in general - you can write something very general purpose and it has a decent chance of succeeding. When it fails though its going to be a lot harder to debug, and then all the other downsides of mirrors regarding tree shaking and performance (although it can be possible to get decent performance if you are very tricky, most of the time its pretty bad in practice).

I would rather see an updated and stabilized API over the removal because there are valid use cases.

Mirrors are one way to solve metaprogramming, but not the only way. So most of this issue is about understanding the use cases, and what the tradeoffs would look like if we went with a different approach.

I do agree moving your example to some form of code generation, however well integrated, would be a somewhat degraded development experience. It would be possible, and more statically safe, but would involve some extra boilerplate as well.

[Nexushunter]

@jakemac53 Thanks for the patience.

The "source code" in this case is actually a very abstract representation, not really any different from JSON or some other serialization format. The types you see are not even necessarily available in the current isolate and thus impossible to rehydrate at all.

I would be inclined to disagree. While yes the types aren't necessarily available in the current isolate, the library and it's declarations are. With knowing the library and declaration name (which we currently get from dart:mirrors) you can rehyrdate it with relative ease.

So, I would actually say it is less hacky to rely on the explicitly generated code to rehydrate very specific known types. How would the generating/builder library know about this code?

If you consider a sufficiently well integrated code generation mechanism (part of the compiler) I think most of these concerns go away. I may not have been sufficiently clear here looking at it now. To expand a bit more say we have:

/// Default [RemoteSpecHeaderDelegate] used when retrieving a remote OAS spec.
class RemoteSpecHeaderDelegate {
  const RemoteSpecHeaderDelegate();

  Map<String, String>? header() => null;
}

Defined and exposed in our library. The consumer would define a subclass like:

// within my_spec_delegate.dart
import 'package:spec_builder/spec_builder.dart';

class MyRemoteSpecHeaderDelegate extends RemoteSpecHeaderDelegate {
    const MyRemoteSpecHeaderDelegate(): super();

    @override
    Map<String, String>? header() => {'Authorization':'someAuthToken'};
}

// within spec_dfn.dart
import 'package:spec_builder/spec_builder.dart';

import 'path/to/my_spec_delegate.dart';

@SpecBuilder( // Imported from builder library
    delegate: MyRemoteSpecHeaderDelegate(),
    // some other properties
)
class SpecDef {}

In this case the consuming repo is building off of the base we delegate we provide, but they aren't implementing the builder themselves, and are having us use it in a predictable manner. I'm not sure how this would be achieved outside of dart:mirrors (obviously we don't current have another option

But if we were able to remove dart:mirrors, that required data would no longer necessarily be present.

True, but I'm providing a use case for leaving the library which this comment is ignoring.

Mirrors are one way to solve metaprogramming, but not the only way. So most of this issue is about understanding the use cases, and what the tradeoffs would look like if we went with a different approach.

Correct and that is what I'm trying to point out. This is a valid use case and is only something that really needs to be done on the developer side (builders are dev_dependencies, unless you're building; at least I would expect that to be the common case, I could be wrong here).

I would love to hear another potential solution on how that may be applied to a builder or how that might be achieved without a build (obviously nothing concrete).

(I've started to read https://github.com/dart-lang/language/issues/1482 but I'm gonna need to make a couple passes before I can sus everything out and understand it more indepth)

[jakemac53]

With knowing the library and declaration name (which we currently get from dart:mirrors) you can rehyrdate it with relative ease.

Only if the build script actually includes the library and declaration though. I don't understand how those are being included in your case, without the user going through all the other builder related config to set up their own builder?

My guess is that in your case, you are actually all within one package, and your builder_import is importing a library which also imports the my_spec_delegate.dart library? I don't understand how it would be working otherwise.

This is all mostly orthogonal though - I think the use case you are describing is well understood - it is effectively identical to the serialization/deserialization use case. With mirrors you can create a general purpose mechanism for "rehydrating" (or "dehydrating" :P) objects.

If we do add expression level macros, you could get something similar. But it would be based on the static type of an object and not its runtime type, which can be a pretty big distinction.

[ykmnkmi]

I use mirrors in utilities like walk(astNode, enter: enter, leave: leave) and serializing classes in tests.

[bobjackman]

For us, mirrors are crucial for many tests so we can inspect/manipulate values of private/protected properties.

[insinfo]

I use dart:mirrors in several backend and CLI projects. I think that even if expression level macros were implemented in Dart there are always use cases where reflection can be much more useful for the developer

[ykmnkmi]

I don't know if macros can do this? but I want to get all annotated top level members:

@Injectable(providedIn: 'root')
final class UserService {}

@RootInjector() // macro generated constant class
external Injector get rootInjector;

T inject<T>() {
  return rootInjector.get<T>();
}

final userService = inject<UserService>();

[jakemac53]

I don't know if macros can do this? but I want to get all annotated top level members:

I don't know how the providedIn: 'root' part of that is supposed to work, but generally it is possible to create a dependency injection framework similar to this. I have one example usage here https://github.com/dart-lang/language/blob/main/working/macros/example/bin/injectable_main.dart (note you can't actually run this yet).

If your example above is searching a directory for files with certain annotations that portion of it would not be possible, but you could use a slightly less magic API such as the example above.

[ykmnkmi]

@jakemac53, rootInjector is a private member of the DI package, and UserService is my class. So I need to get my user service from the DI package using the inject function.

[erickib]

I started to learn mirrors so I want to check here is really been obsolete/deprecated? I was planning to create a small MVC framework. What can I use instead for example loop a user defined 'controller' files in a directory and instantiate object calling users defined controller class/methods from those files.

[mraleph]

I started to learn mirrors so I want to check here is really been obsolete/deprecated?

Not yet, though we heavily discourage the use of mirrors as they are not really compatible with closed-world AOT compilation. They don't work on the Web or in Flutter so you can only use them in CLI in JIT mode.

We have not been updating the library with respect to new language features: e.g. you can reflect on a record but you can't get record's shape (because there is no corresponding RecordTypeMirror) which makes it somewhat useless.

So my recommendation would be: don't use mirrors for anything critical.

[natebosch]

What can I use instead for example loop a user defined 'controller' files in a directory and instantiate object calling users defined controller class/methods from those files.

If you are describing dynamically loading dart code from a directory of dart libraries, this is not a capability offered by mirrors.

Edit - I missed the obvious API for this on IsolateMirror 😳

[mraleph]

If you are describing dynamically loading dart code from a directory of dart libraries, this is not a capability offered by mirrors.

It is: IsolateMirror.loadUri is the API for that.

[bobjackman]

Just my 2 cents: I agree that use of mirrors for main application should be discouraged, due to incompatibility with AOT compilation. HOWEVER, mirrors play a critical role in our automated testing suites allowing us to verify functionality of things otherwise inaccessible (since tests are outside the test-target lib, tests can't see private members/methods, and therefore can't exercise them (or verify they were called as expected)). Deprecating mirrors would be a massive regression for us.

[francescovallone]

Good afternoon 👋!

Sorry to bother, but I'm trying to understand a little bit more about on the situation of dart:mirrors.

I understood from this issue that right now dart:mirrors shouldn't be used in any critical case. I'm really fond of using reflection both for backend and object serialization (dartson-like) but I'm a little bit concerned about the current state of the library since as stated by @mraleph there is not a way to get the shape of a record for example.

Can you give a more detailed update on what plans dart has for this specific library for example if it will be discontinued in a more permanent way in the near future or if the team is trying to provide an alternative?

[jakemac53]

I can't speak with authority, I do not control such things, but I believe it is unlikely it would be removed in a non-breaking Dart language release (so, Dart 4 at the earliest). But, it also is unlikely to get many improvements such as knowledge about record types. It has always been an experimental/unstable API and never graduated to full support. We strive mostly not to break the existing use cases, until such time as there is a viable replacement. This issue exists primarily as a way to track those use cases that exist, such that we can make decisions in the future about the support of this API, as well as the features needed to make any replacement a viable one.

[lrhn]

I wouldn't say that dart:mirrors has always been experimental. In Dart 1, it was fully supported, also on the web.

That was at a significant cost, though, which was why one was strongly discouraged from actually using it on the web, and if you did, there were annotations that allowed some tree-shaking to happen anyway. It still broke tree-shaking in general and required extra metadata, so if just one corner of your web app uses it, your deployment size could grow a lot.

Rather than worry about that, it was discontinued on the web, and never supported in AoT, and because of that dart:mirrors has not had any further development since about Dart 2.0. That's when it became "effectively unsupported". New compilers didn't need to support it, new platforms (including Flutter) didn't. It's a feature that only works on the JIT compiled VM, and only really with features that existed in Dart 2.0. That means you can use it for scripts (if you don't AoT compile them), and not much else.

Using reflection in tests means that those tests cannot be run AoT compiled, which means they are not testing the code that's actually going to run in production. (Can't do dart test -c exe.)

You can use dart:mirrors today, but you shouldn't. They only work in stand-alone programs that you can control how gets run. They should never leak into other people's code, or be run by any system that migth one day want to compile the code to exe first.

I do hope that macros can give users what they need to replace existing uses of reflcection. Like a reflectiveTest that finds all methods with a name starting with test, a macro could find the same methods, and create an allTests() method that calls them one by one.

[sigurdm]

In Dart 1, it was fully supported, also on the web.

https://codereview.chromium.org//61793006 marked it as "unstable" - that was 2013

The documentation (today) says: This library is only supported by the Dart VM and only available on some platforms.

[lrhn]

True, I guess the API was never considered stable, which makes sense if it has to match an evolving language. The phrasing there was "API might change slightly as a result of user feedback", which is far from "this doesn't work". And the comment should have been removed if such changes didn't appear soon after.

The API at the time was supported everywhere. And the API has been stable, so the comment didn't end up meaning much.

Mirrors, as such, were not considered experimental in Dart 1. The concrete API was, possibly, conisdered a little experimental, but after surviving a few releases, that disclaimer didn't mean much.

[jakemac53]

Like a reflectiveTest that finds all methods with a name starting with test, a macro could find the same methods, and create an allTests() method that calls them one by one.

Right, macros can definitely enable these class based test runners. It could pretty easily even just be a small layer on top of package:test, since tests/groups can be added programatically.

[gmpassos]

Macros will need to be able to inform the annotations of classes and methods or use them as filters and input parameters for generators.

[turbobuilt]

All right, I think I have the answer. We just need a simple way to have a class extend JsonSerializable or Mappable. (maybe both). This should be built into the compiler/interpreter directly.

Then the compiler can see this info and add custom code accordingly. I tried using Reflectable, and it works, but you have to do an extra build step that takes a long time, and it doesn't work well with hot reload, and you have to import the generated file. So this method is better because the compiler can see it and do the work for us!

In Swift, you just extend Codable and it works! If we want dart to catch on more, we need to fix this, because this is a problem almost every developer will face!

I know it may be a lot of work to implement, but I'm telling you guys, this is a super important feature, and should be prioritized for the sake of dart.