🐛 [cloud_firestore] Get all documents in collection returning subset of data

SDGMartin commented 1 year ago

Bug report

Describe the bug Attempting to get all documents in a collection, count gives me 70k+ records, but the querysnapshot only ever returns 15k. Running a count directly on the dataset in firebase console shows the correct count as does running a count through code

Also worth noting that although the number of records returned is incorrect, the number of reads shown in firebase console usage, matches the number of documents in the collection.

Maybe related but It takes quite a long time to retrieve the data as well (around 30s-1 min) so any tips on speeding that up is also appreciated.

Steps to reproduce

See code below Steps to reproduce the behavior:

Future<List> getGamesFuture() async {

List<StatsGame> games = [];

await firestore
    .collection("steam_games")
    .get()
    .then((querySnapshot) {
  if (kDebugMode) {
    print("snapshot games = ${querySnapshot.docs.length}");
  }
  for (var result in querySnapshot.docs) {
    StatsGame game = StatsGame.fromMap(result.id, result.data());
    games.add(game);

  }
  if (kDebugMode) {
    print("firebase games " + games.length.toString());
  }
});
return games;

}

Expected behavior

Expect all records to be returned in the collection

Sample project

Flutter doctor

Run flutter doctor and paste the output below:

Click To Expand

``` Doctor summary (to see all details, run flutter doctor -v): [√] Flutter (Channel stable, 3.13.6, on Microsoft Windows [Version 10.0.22621.2361], locale en-GB) [√] Windows Version (Installed version of Windows is version 10 or higher) [√] Android toolchain - develop for Android devices (Android SDK version 34.0.0) [√] Chrome - develop for the web [!] Visual Studio - develop Windows apps (Visual Studio Community 2022 17.4.3) X Visual Studio is missing necessary components. Please re-run the Visual Studio installer for the "Desktop development with C++" workload, and include these components: MSVC v142 - VS 2019 C++ x64/x86 build tools - If there are multiple build tool versions available, install the latest C++ CMake tools for Windows Windows 10 SDK [√] Android Studio (version 2022.3) [√] Connected device (4 available) [√] Network resources ! Doctor found issues in 1 category. ```

Flutter dependencies

Run flutter pub deps -- --style=compact and paste the output below:

Click To Expand

``` Dart SDK 3.1.3 Flutter SDK 3.13.6 steam_stats 1.1.1+2 dependencies: - cloud_firestore 4.9.3 [cloud_firestore_platform_interface cloud_firestore_web collection firebase_core firebase_core_platform_interface flutter meta] - cupertino_icons 1.0.6 - firebase_analytics 10.5.1 [firebase_analytics_platform_interface firebase_analytics_web firebase_core firebase_core_platform_interface flutter] - firebase_auth 4.10.1 [firebase_auth_platform_interface firebase_auth_web firebase_core firebase_core_platform_interface flutter meta] - firebase_core 2.17.0 [firebase_core_platform_interface firebase_core_web flutter meta] - firebase_crashlytics 3.3.7 [firebase_core firebase_core_platform_interface firebase_crashlytics_platform_interface flutter stack_trace] - firebase_ui_auth 1.8.0 [email_validator firebase_auth firebase_core firebase_dynamic_links firebase_ui_localizations firebase_ui_oauth firebase_ui_shared flutter flutter_localizations] - flutter 0.0.0 [characters collection material_color_utilities meta vector_math web sky_engine] - flutter_html 3.0.0-beta.2 [html csslib collection list_counter flutter] - flutter_launcher_icons 0.13.1 [args checked_yaml cli_util image json_annotation path yaml] - google_mobile_ads 3.0.0 [meta flutter visibility_detector webview_flutter_android webview_flutter_wkwebview webview_flutter] - http 1.1.0 [async http_parser meta] - intl 0.18.1 [clock meta path] - provider 6.0.5 [collection flutter nested] - pull_to_refresh 2.0.0 [flutter] - url_launcher 6.1.14 [flutter url_launcher_android url_launcher_ios url_launcher_linux url_launcher_macos url_launcher_platform_interface url_launcher_web url_launcher_windows] dev dependencies: - flutter_lints 2.0.3 [lints] - flutter_test 0.0.0 [flutter test_api matcher path fake_async clock stack_trace vector_math async boolean_selector characters collection material_color_utilities meta source_span stream_channel string_scanner term_glyph web] transitive dependencies: - _flutterfire_internals 1.3.7 [collection firebase_core firebase_core_platform_interface flutter meta] - archive 3.4.5 [crypto path pointycastle] - args 2.4.2 - async 2.11.0 [collection meta] - boolean_selector 2.1.1 [source_span string_scanner] - characters 1.3.0 - checked_yaml 2.0.3 [json_annotation source_span yaml] - cli_util 0.4.0 [meta path] - clock 1.1.1 - cloud_firestore_platform_interface 5.16.2 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface] - cloud_firestore_web 3.7.2 [_flutterfire_internals cloud_firestore_platform_interface collection firebase_core firebase_core_web flutter flutter_web_plugins js] - collection 1.17.2 - convert 3.1.1 [typed_data] - crypto 3.0.3 [typed_data] - csslib 0.17.3 [source_span] - desktop_webview_auth 0.0.13 [crypto flutter http flutter_web_plugins plugin_platform_interface] - email_validator 2.1.17 - fake_async 1.3.1 [clock collection] - firebase_analytics_platform_interface 3.7.1 [_flutterfire_internals firebase_core flutter meta plugin_platform_interface] - firebase_analytics_web 0.5.5+1 [_flutterfire_internals firebase_analytics_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins js] - firebase_auth_platform_interface 6.19.1 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface] - firebase_auth_web 5.8.2 [firebase_auth_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins http_parser js meta] - firebase_core_platform_interface 4.8.0 [collection flutter flutter_test meta plugin_platform_interface] - firebase_core_web 2.8.0 [firebase_core_platform_interface flutter flutter_web_plugins js meta] - firebase_crashlytics_platform_interface 3.6.7 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface] - firebase_dynamic_links 5.3.7 [firebase_core firebase_core_platform_interface firebase_dynamic_links_platform_interface flutter meta plugin_platform_interface] - firebase_dynamic_links_platform_interface 0.2.6+7 [_flutterfire_internals firebase_core flutter meta plugin_platform_interface] - firebase_ui_localizations 1.6.1 [flutter flutter_localizations path] - firebase_ui_oauth 1.4.11 [desktop_webview_auth firebase_auth firebase_ui_auth firebase_ui_shared flutter_svg flutter] - firebase_ui_shared 1.3.1 [flutter] - flutter_localizations 0.0.0 [flutter intl characters clock collection material_color_utilities meta path vector_math web] - flutter_svg 2.0.7 [flutter vector_graphics vector_graphics_codec vector_graphics_compiler] - flutter_web_plugins 0.0.0 [flutter characters collection material_color_utilities meta vector_math web] - html 0.15.4 [csslib source_span] - http_parser 4.0.2 [collection source_span string_scanner typed_data] - image 4.1.3 [archive meta xml] - js 0.6.7 [meta] - json_annotation 4.8.1 [meta] - lints 2.1.1 - list_counter 1.0.2 - matcher 0.12.16 [async meta stack_trace term_glyph test_api] - material_color_utilities 0.5.0 [collection] - meta 1.9.1 - nested 1.0.0 [flutter] - path 1.8.3 - path_parsing 1.0.1 [vector_math meta] - petitparser 5.4.0 [meta] - plugin_platform_interface 2.1.6 [meta] - pointycastle 3.7.3 [collection convert js] - sky_engine 0.0.99 - source_span 1.10.0 [collection path term_glyph] - stack_trace 1.11.0 [path] - stream_channel 2.1.1 [async] - string_scanner 1.2.0 [source_span] - term_glyph 1.2.1 - test_api 0.6.0 [async boolean_selector collection meta source_span stack_trace stream_channel string_scanner term_glyph] - typed_data 1.3.2 [collection] - url_launcher_android 6.1.0 [flutter url_launcher_platform_interface] - url_launcher_ios 6.1.5 [flutter url_launcher_platform_interface] - url_launcher_linux 3.0.6 [flutter url_launcher_platform_interface] - url_launcher_macos 3.0.7 [flutter url_launcher_platform_interface] - url_launcher_platform_interface 2.1.5 [flutter plugin_platform_interface] - url_launcher_web 2.0.20 [flutter flutter_web_plugins url_launcher_platform_interface] - url_launcher_windows 3.0.8 [flutter url_launcher_platform_interface] - vector_graphics 1.1.7 [flutter vector_graphics_codec] - vector_graphics_codec 1.1.7 - vector_graphics_compiler 1.1.7 [args meta path_parsing xml vector_graphics_codec] - vector_math 2.1.4 - visibility_detector 0.3.3 [flutter] - web 0.1.4-beta - webview_flutter 4.4.1 [flutter webview_flutter_android webview_flutter_platform_interface webview_flutter_wkwebview] - webview_flutter_android 3.12.0 [flutter webview_flutter_platform_interface] - webview_flutter_platform_interface 2.6.0 [flutter meta plugin_platform_interface] - webview_flutter_wkwebview 3.9.1 [flutter path webview_flutter_platform_interface] - xml 6.3.0 [collection meta petitparser] - yaml 3.1.2 [collection source_span string_scanner] ```

darshankawar commented 1 year ago

@SDGMartin Does your collection also contains sub-collections ? If so, that might be the reason as why you are not seeing all documents. You can check https://firebase.google.com/docs/firestore/using-console and see if it helps in your case or not.

SDGMartin commented 1 year ago

@SDGMartin Does your collection also contains sub-collections ? If so, that might be the reason as why you are not seeing all documents. You can check https://firebase.google.com/docs/firestore/using-console and see if it helps in your case or not.

no sub-collections, just basic fields, I can see the count going up after adding a few more in the pulled collection, now 17k via flutter but still 76k+ in Firebase console.

russellwheatley commented 1 year ago

You ought to read query limitations. My guess is you've hit the limit for a request which is limited to 10mb.

I would start thinking about pagination. Do you really need 17 thousand documents in memory much less 76 thousand documents?

SDGMartin commented 1 year ago

You ought to read query limitations. My guess is you've hit the limit for a request which is limited to 10mb.

I would start thinking about pagination. Do you really need 17 thousand documents in memory much less 76 thousand documents?

It used to run fine a year ago last time I messed around with it all, I can get more documents from a similar collection without issue - I think I found the issue, it has something to do with some properties being absent but indexed which is messing with the data returned.

As for Do I need them all? Well, for what I'm doing yes, which is taking my dataset and checking against an external dataset to see what is missing, and then filling in the blanks, its much more efficient to do that check locally than hitting FB for every record individually to check if it exists.

Pull dataset from FB (A), pull external Dataset (B) Compare A to B and create what is missing (C) Add records in C to A

darshankawar commented 1 year ago

I think I found the issue, it has something to do with some properties being absent but indexed which is messing with the data returned.

Good to know you found the issue. Based on it, I'll go ahead and close this as not a bug in plugin code. If you disagree, write in comments and I'll reopen it.

firebase / flutterfire