dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.07k stars 1.56k forks source link

Analyzer uses too much memory when there is many lints enabled #41793

Closed pedromassango closed 9 months ago

pedromassango commented 4 years ago

The Dart Analyzer is taking 1,811.0MB of RAM when I have 125 lint rules enabled in analysis_options.yaml. After removing the 125 lint rules the memory usage decreased to 450MB (max).

analysis_1

The lint rules that I had enabled in analysis_options.yaml file are:

analysis_options.yaml ```yaml linter: rules: - always_declare_return_types - always_put_control_body_on_new_line # conflicts with the Flutter convention of putting {Key key} first and {Widget child} last # - always_put_required_named_parameters_first - always_require_non_null_named_parameters - annotate_overrides # conflicts with type_annotate_public_apis and implicit-dynamic:false # - avoid_annotating_with_dynamic # conflicts with `implicit-dynamic: false` #- avoid_as - avoid_bool_literals_in_conditional_expressions - avoid_catches_without_on_clauses - avoid_catching_errors - avoid_classes_with_only_static_members - avoid_double_and_int_checks - avoid_empty_else - avoid_field_initializers_in_const_classes - avoid_function_literals_in_foreach_calls - avoid_implementing_value_types - avoid_init_to_null - avoid_js_rounded_ints - avoid_null_checks_in_equality_operators - avoid_positional_boolean_parameters - avoid_private_typedef_functions - avoid_relative_lib_imports - avoid_renaming_method_parameters - avoid_return_types_on_setters - avoid_returning_null - avoid_returning_null_for_void - avoid_returning_this - avoid_setters_without_getters - avoid_single_cascade_in_expression_statements - avoid_slow_async_io - avoid_types_as_parameter_names - avoid_unused_constructor_parameters - avoid_void_async - await_only_futures - camel_case_types - cancel_subscriptions #- cascade_invocations - close_sinks - comment_references - constant_identifier_names - control_flow_in_finally - curly_braces_in_flow_control_structures - directives_ordering - empty_catches - empty_constructor_bodies - empty_statements - file_names #- flutter_style_todos - hash_and_equals - implementation_imports - invariant_booleans - iterable_contains_unrelated_type - join_return_with_assignment - library_names - library_prefixes - list_remove_unrelated_type - literal_only_boolean_expressions - no_adjacent_strings_in_list - no_duplicate_case_values - non_constant_identifier_names - null_closures - omit_local_variable_types - one_member_abstracts - only_throw_errors - overridden_fields - package_api_docs - package_names - package_prefixed_library_names - parameter_assignments - prefer_adjacent_string_concatenation - prefer_asserts_in_initializer_lists - prefer_collection_literals - prefer_conditional_assignment - prefer_const_constructors - prefer_const_constructors_in_immutables - prefer_const_declarations - prefer_const_literals_to_create_immutables - prefer_constructors_over_static_methods - prefer_contains - prefer_equal_for_default_values - prefer_expression_function_bodies - prefer_final_fields - prefer_final_locals - prefer_foreach - prefer_function_declarations_over_variables - prefer_generic_function_type_aliases - prefer_initializing_formals - prefer_int_literals - prefer_interpolation_to_compose_strings - prefer_is_empty - prefer_is_not_empty - prefer_iterable_whereType - prefer_mixin - prefer_single_quotes - prefer_typing_uninitialized_variables - prefer_void_to_null - recursive_getters - slash_for_doc_comments - sort_constructors_first - sort_pub_dependencies - sort_unnamed_constructors_first - test_types_in_equals - throw_in_finally - type_annotate_public_apis - type_init_formals - unawaited_futures - unnecessary_brace_in_string_interps - unnecessary_const - unnecessary_getters_setters - unnecessary_lambdas - unnecessary_new - unnecessary_null_aware_assignments - unnecessary_null_in_if_null_operators - unnecessary_overrides - unnecessary_parenthesis - unnecessary_statements - unnecessary_this - unrelated_type_equality_checks - use_rethrow_when_possible - use_string_buffers - use_to_and_as_if_applicable - valid_regexps - void_checks ```
jeffreyconcha commented 4 years ago

Same with me I encountered this bug after I upgraded to flutter 1.17.0 which has Dart 2.8.1... I have a 16GB of RAM and whenever i compile my code, the dart.exe file on task manager uses all the available RAM of my PC....Please fix this ASAP. It uses 99 percent of my memory......

I redownloaded the Flutter v1.12.13 w/c has Dart v2.7.0.....and everything backs to normal......PLEASE FIX THIS ASAP!!!!!

pedromassango commented 4 years ago

So lets vote the issue so that it gets more atention.

easazade commented 3 years ago

The way it happens to me is when I start typing Future< in an abstract class like when I want to use an abstract class like an interface and define many methods with return type of Future it just keeps happening. I recommend to do this for reproducing this bug.

I fix it by terminating dart process from task manager and starting it again. but this is really getting annoying. since i can't type Future< because it happens and I have to just copy another method and change that.

here is an example code where this bug shows itself :

@RestApi(baseUrl: APP_BASE_URL)
abstract class RestClient {
  factory RestClient(Dio dio, {String baseUrl}) = _RestClient;

  @GET('/courses')
  Future<List<Course>> getCourses();

  @GET('/trainers')
  Future<List<Trainer>> getTrainers();

  @POST('/signup')
  Future<HttpResponse<void>> signup(@Body() Map<String, dynamic> json);

  @POST('/login')
  Future<HttpResponse<User>> login(@Body() Map<String, dynamic> json);

  @POST('/users/avatar')
  Future<User> addUserAvatar(@Body() Map<String, dynamic> json);

}

note : and I don't even to run my code it just happens when I type

flutter doctor report:

$ flutter doctor -v
[√] Flutter (Channel stable, 1.22.2, on Microsoft Windows [Version 10.0.18363.959], locale en-US)
    • Flutter version 1.22.2 at C:\flutter
    • Framework revision 84f3d28555 (6 weeks ago), 2020-10-15 16:26:19 -0700
    • Engine revision b8752bbfff
    • Dart version 2.10.2

[!] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at F:\sdk\Android.SDK
    • Platform android-30, build-tools 30.0.2
    • ANDROID_HOME = F:\sdk\Android.SDK
    • ANDROID_SDK_ROOT = F:\sdk\Android.SDK
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
    X Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/windows#android-setup for more details.

[√] Android Studio (version 4.0)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin version 50.0.1
    • Dart plugin version 193.7547
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[√] IntelliJ IDEA Ultimate Edition (version 2019.3)
    • IntelliJ at C:\Program Files\JetBrains\IntelliJ IDEA 2019.3.2
    • Flutter plugin version 45.1.2
    • Dart plugin version 193.7361

[√] VS Code, 64-bit edition (version 1.50.1)
    • VS Code at C:\Program Files\Microsoft VS Code
    • Flutter extension version 3.16.0

[!] Connected device
    ! No devices available

! Doctor found issues in 2 categories.

here is my pubspec.yaml:

name: matjoy_app
description: MatJoy, the Best Fitness app you can find
version: 0.1.4+3

environment:
  sdk: ">=2.10.2 <3.0.0"

dependencies:
  flutter:
    sdk: flutter
  flutter_localizations:
    sdk: flutter
  footprint: ^1.1.6
  intl:
  intl_translation:
  flutter_bloc: ^6.1.1
  connectivity: ^2.0.2
#  shared_preferences: ^0.5.12
  sailor: ^0.7.1
  freezed_annotation: ^0.11.0+1
  json_annotation: ^3.0.1
  fluttertoast: ^7.1.5
  injectable: ^1.0.4
  get_it: ^4.0.4
  url_launcher: ^5.7.10
  shamsi_date: ^0.9.1
  tuple: ^1.0.3
  dio: ^3.0.10
  retrofit: ^1.3.4+1
  auto_size_text: ^2.1.0
  carousel_slider: ^2.3.1
  flutter_custom_clippers: ^1.1.2
  cached_network_image: ^2.3.3
  flushbar: ^1.10.4
  skeleton_text: ^1.0.0
  badges: ^1.1.4
#  explode_view: ^1.0.4
  #  permission_handler: ^5.0.1+1
  #  moor: ^3.3.1
  #  sqlite3_flutter_libs: ^0.2.0
#  confetti: ^0.5.4+1
  chewie: ^0.12.0

#  ff_navigation_bar: ^0.1.5
#  video_player: ^1.0.0
  video_thumbnail: ^0.2.4
  flutter_group_sliver: ^0.0.2
  flutter_verification_code: ^0.2.1
  persian_datepicker: ^1.3.1
  device_preview: ^0.4.8
  file_picker: ^1.9.0+1
  #  adaptive_dialog: ^0.9.0+1
  faker: ^1.3.0
  dotted_border: ^1.0.7
  hive: ^1.4.4+1
  lottie: ^0.7.0+1
  hive_flutter: ^0.3.1
  fl_chart: ^0.12.0
  path_provider:
  path:

  ################################# icons ##########################################
  cupertino_icons: ^1.0.0
  feather_icons_flutter: ^4.7.4 # https://feathericons.com/
#  ionicons: ^0.0.3 #https://ionicons.com/
#  community_material_icon: ^5.4.55 #https://materialdesignicons.com/
#  font_awesome_flutter: ^8.5.0 # https://fontawesome.com/icons
#  line_icons: ^0.2.0 # https://icons8.com/line-awesome

dev_dependencies:
  flutter_test:
    sdk: flutter

  injectable_generator: ^1.0.4
  retrofit_generator: ^1.4.0+2
  hive_generator: ^0.8.2
  #  moor_generator: ^3.3.1
  build_runner:
  mockito: ^4.1.1
  freezed: ^0.11.5
  # preview: ^0.0.8
  json_serializable: ^3.5.0
  flutter_launcher_icons: ^0.8.1
  test: ^1.15.2
  bloc_test: ^7.1.0

dependency_overrides:
  freezed_annotation: ^0.11.0+1

 #just run -> flutter pub run flutter_launcher_icons:main
flutter_icons:
   #    image_path: "assets/images/icon-128x128.png"
  image_path_android: "assets/icons/icon_legacy.png"
  image_path_ios: "assets/icons/icon_legacy.png"
  android: true # can specify file name here e.g. "ic_launcher"
  ios: true # can specify file name here e.g. "My-Launcher-Icon"
  adaptive_icon_background: "assets/icons/icon_back.png" # only available for Android 8.0 devices and above
  adaptive_icon_foreground: "assets/icons/icon_front.png" # only available for Android 8.0 devices and above

# run -> fluttergen -c example/pubspec.yaml
flutter_gen:
  output: lib/gen/ # Optional (default: lib/gen/)
  lineLength: 120 # Optional (default: 80)

#  integrations:
#    flutter_svg: true

#  colors:
#    inputs:
#      - assets/color/colors.xml

flutter:

  uses-material-design: true

  assets:
    - assets/images/app_logo.png
    - assets/images/app_logo_orange.png
    - assets/images/arm_image.png
    - assets/images/image_placeholder.png
    - assets/images/image_placeholder_small.png
    - assets/images/sidebar_logo.png
    - assets/images/trainer.png
    - assets/images/trainer_shadow.png
    - assets/images/trainee.png
    - assets/images/trainee_shadow.png
    - assets/images/logo_and_name.png
    - assets/images/login_top_shape_green.png
    - assets/images/login_top_shape_orange.png
    - assets/images/bar_chart.png
    - assets/json/logo_lottie.json
    - assets/video/logo_trailer.mp4
    - assets/video/trailer_1.mp4

  fonts:
    - family: IranYekan
      fonts:
        - asset: assets/fonts/iranyekan/IRANYekanMobileThin.ttf
          weight: 100
        - asset: assets/fonts/iranyekan/IRANYekanMobileLight.ttf
          weight: 300
        - asset: assets/fonts/iranyekan/IRANYekanMobileRegular.ttf
          weight: 400
        - asset: assets/fonts/iranyekan/IRANYekanMobileMedium.ttf
          weight: 500
        - asset: assets/fonts/iranyekan/IRANYekanMobileBold.ttf
          weight: 700
        - asset: assets/fonts/iranyekan/IRANYekanMobileExtraBold.ttf
          weight: 800
        - asset: assets/fonts/iranyekan/IRANYekanMobileBlack.ttf
          weight: 900
    - family: IranYekan-FD
      fonts:
        - asset: assets/fonts/iranyekan/fd/IRANYekanMobileThin-FD.ttf
          weight: 100
        - asset: assets/fonts/iranyekan/fd/IRANYekanMobileLight-FD.ttf
          weight: 300
        - asset: assets/fonts/iranyekan/fd/IRANYekanMobileRegular-FD.ttf
          weight: 400
        - asset: assets/fonts/iranyekan/fd/IRANYekanMobileMedium-FD.ttf
          weight: 500
        - asset: assets/fonts/iranyekan/fd/IRANYekanMobileBold-FD.ttf
          weight: 700
        - asset: assets/fonts/iranyekan/fd/IRANYekanMobileExtraBold-FD.ttf
          weight: 800
        - asset: assets/fonts/iranyekan/fd/IRANYekanMobileBlack-FD.ttf
          weight: 900
    - family: IranSans
      fonts:
        - asset: assets/fonts/iransans/IRANSansMobile_UltraLight.ttf
          weight: 100
        - asset: assets/fonts/iransans/IRANSansMobile_Light.ttf
          weight: 300
        - asset: assets/fonts/iransans/IRANSansMobile.ttf
          weight: 400
        - asset: assets/fonts/iransans/IRANSansMobile_Medium.ttf
          weight: 500
        - asset: assets/fonts/iransans/IRANSansMobile_Bold.ttf
          weight: 700
        - asset: assets/fonts/iransans/IRANSansMobile_Black.ttf
          weight: 900
    - family: IranSans-FD
      fonts:
        - asset: assets/fonts/iransans/fd/IRANSansMobile_FaNum_UltraLight.ttf
          weight: 100
        - asset: assets/fonts/iransans/fd/IRANSansMobile_FaNum_Light.ttf
          weight: 300
        - asset: assets/fonts/iransans/fd/IRANSansMobile_FaNum.ttf
          weight: 400
        - asset: assets/fonts/iransans/fd/IRANSansMobile_FaNum_Medium.ttf
          weight: 500
        - asset: assets/fonts/iransans/fd/IRANSansMobile_FaNum_Bold.ttf
          weight: 700
        - asset: assets/fonts/iransans/fd/IRANSansMobile_FaNum_Black.ttf
          weight: 900

    - family: MatjoyIcons
      fonts:
        - asset: assets/fonts/icons/MatjoyIcons.ttf
    - family: MatjoyIcons2
      fonts:
        - asset: assets/fonts/icons/MatjoyIcons2.ttf
bernaferrari commented 3 years ago

So.. is this issue responsible for my Dart process eating over 18GB of RAM? It only happens after a few hours. https://github.com/dart-lang/sdk/issues/44792

bernaferrari commented 2 years ago

Is this related to https://github.com/flutter/flutter/issues/90868?

simc commented 2 years ago

This is still a big issue. Our project with many packages consistently uses more than 10GB of ram.

pq commented 2 years ago

Sorry this is still an issue for you @leisim.

Are you using dart_code_metrics?

simc commented 2 years ago

@pq no, just the default flutter lints. The big issue is the number of packages in the workspace. With a VSCode workspace that only includes some of the packages, the ram usage can be reduced significantly. But it's certainly not ideal 😔

aloisdeniel commented 2 years ago

Same issue here!

The codebase is quite big, but 15GB feels a lot.

image

simc commented 2 years ago

@pq any updates or workarounds here? This issue is getting worse and slowing down our entire team 😔

pq commented 2 years ago

@scheglov has been looking at element model memory usage and summary serialization but I don't know if anything conclusive has come out of it. We're also looking at leveraging some work on leak detection that @polina-c is driving but that's early yet.

TL;DR nothing substantial yet but folks are looking... thanks for your patience @leisim and sorry for the inconvenience!

simc commented 2 years ago

Let us know if you need a sample project to reproduce the issue

pq commented 2 years ago

A repro would be fantastic. Thanks @leisim!

bernaferrari commented 2 years ago

TIL Clickup uses Flutter

ezze commented 2 years ago

Is it possible to disable Dart Analysis in IDE at least?

bwilkerson commented 2 years ago

If you're asking whether the tool currently supports having different analysis options in IDEs and the command-line, the answer is 'no'.

If you're asking whether it's possible to not run the analysis server in an IDE, the answer is 'yes', but you'll loose all of the language support: semantic highlighting, code completion, hovers, searches, everything.

If you're asking whether it's possible to stop running lints, both in the IDE and on the command-line, then the answer is 'yes'. You can disable lints in your local analysis options file even if they were enabled in an included options file.

ezze commented 2 years ago

@bwilkerson Thanks for you answer. I was asking about stopping analysis in IDE but can't find the option to turn it off.

If you're asking whether it's possible to not run the analysis server in an IDE, the answer is 'yes', but you'll loose all of the language support: semantic highlighting, code completion, hovers, searches, everything.

Does it mean that I simply should disable Dart support in IDE? Seems that it's not what I want. I thought that I could turn linting off in IDE, and it would help me to save RAM on my laptop then.

bwilkerson commented 2 years ago

You can't turn linting off in the IDE without turning it off everywhere, but you can disable any (or all) of the lints that you want to through the analysis options file. See https://dart.dev/guides/language/analysis-options#disabling-individual-rules for details.

ezze commented 2 years ago

Bad news. :( I still need to lint the source code in CI... Thanks anyway.

vasilich6107 commented 2 years ago

Working in project with monorepo. Using include: package:lint/analysis_options.yaml and

plugins:
     - dart_code_metrics
image

It completely impossible to work in this case. Have to disable most of the linter rules.

pq commented 2 years ago

Sorry for the continued trouble @vasilich6107. A few things of note for anyone digging into this:

  1. the lint set is: https://github.com/passsy/dart-lint/blob/master/lib/analysis_options.yaml
  2. dart_code_metrics is enabled

I'd be curious how different the memory profile looks like w/o lints enabled. (And if you've noticed any that seem to impact memory in particular, that'd be especially interesting.) I'd also expect the biggest drop if you disabled dart_code_metrics but that would be interesting to confirm. (Plugins are currently pretty memory intensive.)

Is this an open source project?

incendial commented 1 year ago

I'd also expect the biggest drop if you disabled dart_code_metrics but that would be interesting to confirm. (Plugins are currently pretty memory intensive.)

Yeah, any chance https://github.com/flutter/flutter/issues/95092 and https://github.com/dart-lang/sdk/issues/48925 could be addressed sooner? Coz the problem is literally "by design" and not by how the plugin code is written

pq commented 1 year ago

It completely impossible to work in this case. Have to disable most of the linter rules.

I'm working on some perf improvements for a handful of lints that use a utility method that's problematic (https://github.com/dart-lang/linter/issues/3745). Hopefully this will help. In the meantime, I'd be curious to know if any lints in particular are especially impacting. One I'm particularly curious about is invariant_booleans since it's pretty slow and (needlessly) creates a lot of short-lived objects.

hongfeiyang commented 1 year ago
image

with this repo: https://github.com/kodecocodes/rwf-materials.git 16G RAM usage

mraleph commented 1 year ago

@hongfeiyang are you opening the whole folder in VS Code? If yes - is this intentional, why not open a specific subfolder instead?

/cc @jensjoha

hongfeiyang commented 1 year ago

@hongfeiyang are you opening the whole folder in VS Code? If yes - is this intentional, why not open a specific subfolder instead?

/cc @jensjoha

Oh yea, I was being lazy and should have opened a single sub directory only. Thanks for pointing that out, it only takes around 700MB RAM to analyse a single sub directory.

bernaferrari commented 1 year ago

Possibly related: https://twitter.com/OrestesGaolin/status/1623433133369737216

image
fzyzcjy commented 1 year ago

My analysis server eats ~8GB memory (when freshly opened) and even more (have observed 16GB IIRC) after playing with IDE for a while on my own project...

So, I wonder whether this is a bug or not, and will it be solved in the future? Thanks!

image

fzyzcjy commented 1 year ago

Well today it breaks the record: 19GB (for the same codebase) :(

image

anonymousbitshifter commented 1 year ago

I just wanted to chime in after coming back from the long weekend and wondering what was chomping all my memory causing instability in other apps and found this little gremlin. It is not often you come across something more resource intensive than a chromium process but this process certainly wants to make itself known to be a competitor.

$ ps -e -orss=,args= | sort -nr | head
4857356 /home/loki/projects/flutter/bin/cache/dart-sdk/bin/dart language-server --client-id=Android-Studio --client-version=AI-222.4459.24 --protocol=analyzer

image

srawlins commented 12 months ago

Sorry that everyone here is experiencing high memory usage by the Dart Analysis Server (DAS).

After reviewing all of the comments above, I don't see any evidence that enabling many linter rules results in much higher memory usage.

If you are just experiencing high memory usage by DAS, feel free to open a separate issue, if you can follow up with repro steps and more information. Some basic info we would need:

If you believe that enabling many linter rules is the cause of high memory usage, we'll need some evidence of that. In particular, if you see high memory usage, and then disable all linter rules, and then restart DAS and check the new memory usage, this is not a great test 😄 . A long-running process will likely consume more memory than a fresh one, due to caches filling with info, leaks, etc. I think a few trials, flipping linter rules on and of and on and off, would be required.

bwilkerson commented 12 months ago

I'll just note that it's also possible, given what little we know about the cause at the moment, that it isn't the number of lints, but a small number of lints that are not well behaved in terms of memory use. (We have tried to find problematic code, but haven't found anything obvious.) If you can reliably reproduce the memory problems it would be interesting to try to narrow down whether there are a small number of lints that need to be tuned or whether it really is the number of lints. And if it is the number of lints, is the memory usage linear, exponential, etc.

srawlins commented 9 months ago

I'm closing this issue as a few folks on the Dart team have landed many memory improvements in the Analyzer code bases over the last year (see below). I won't promise that your memory problems are solved, but I think at this point, it would be most effective to sort of rebase the discussion.

Anyone experiencing similar "giant memory usage" issues with the Dart Analyzer, and you'd like to document a bit your situation, also please follow these instructions:

First, be sure to bump to the latest Dart, 3.2.0, as it has some performance improvements over 3.1.0. Then, if you experience what you feel is an inordinate amount of memory being used (we don't have any SLOs yet in this space, but anything north of 5GB is probably noteworthy):

  1. We have a public document with instructions how to setup auto-snapshotting for analyzer in case of huge memory usage: https://docs.google.com/document/d/1mxCKhvyHUtGVhIH5Apx9y7V7itqCxAQPse6ghbjaDMM
  2. You can file an issue in this repo and send us some other basic info:
    • How many .dart files are under the analysis root for the IDE?
    • In the Diagnostics pages, you may want to look at Memory, Plugins, how many Analysis Contexts exist, etc.
    • How long has the process been running? If you restart your IDE or and/or DAS, does it use less memory? What about hours later? What about days later?

Recent memory-reducing changes:

Dart ~ 2.19.0, Jan 2023

Dart ~3.0.0, May 2023

Dart ~3.1.0, August 2023

Dart ~3.2.0, November 2023

(There were also speed-up improvements made in this time frame, and improvements to our analytics, logging, benchmarking, and platform coverage, but I'm not documenting those here on this memory-focused issue.)