Datastore query only returns 10000 items

[zamargab]

Describe the bug I have a DynamoDb with 170,000 items, when i query the database, I only get 10,000 records

To Reproduce Steps to reproduce the behavior:

1. Create new table
2. Populate with 170,000 records
3. Query database

Expected behavior I expect to retrieve all the data in my table but i only get 10,000 records

Platform Amplify Flutter current supports iOS and Android. This issue is reproducible in (check all that apply): [] Android

Output of flutter doctor -v

```Doctor summary (to see all details, run flutter doctor -v): [√] Flutter (Channel stable, 2.5.1, on Microsoft Windows [Version 10.0.15063], locale en-GB) [!] Android toolchain - develop for Android devices (Android SDK version 30.0.2) X cmdline-tools component is missing Run `path/to/sdkmanager --install "cmdline-tools;latest"` See https://developer.android.com/studio/command-line for more details. X Android license status unknown. Run `flutter doctor --android-licenses` to accept the SDK licenses. See https://flutter.dev/docs/get-started/install/windows#android-setup for more details. [√] Chrome - develop for the web [√] Android Studio (version 4.0) [√] VS Code (version 1.61.2) [√] Connected device (2 available) ! Doctor found issues in 1 category.```

Dependencies (pubspec.lock)

``` # Generated by pub # See https://dart.dev/tools/pub/glossary#lockfile packages: amplify_analytics_plugin_interface: dependency: transitive description: name: amplify_analytics_plugin_interface url: "https://pub.flutter-io.cn" source: hosted version: "0.2.5" amplify_api: dependency: "direct main" description: name: amplify_api url: "https://pub.flutter-io.cn" source: hosted version: "0.2.5" amplify_api_plugin_interface: dependency: transitive description: name: amplify_api_plugin_interface url: "https://pub.flutter-io.cn" source: hosted version: "0.2.5" amplify_auth_cognito: dependency: "direct main" description: name: amplify_auth_cognito url: "https://pub.flutter-io.cn" source: hosted version: "0.2.5" amplify_auth_plugin_interface: dependency: transitive description: name: amplify_auth_plugin_interface url: "https://pub.flutter-io.cn" source: hosted version: "0.2.5" amplify_core: dependency: transitive description: name: amplify_core url: "https://pub.flutter-io.cn" source: hosted version: "0.2.5" amplify_datastore: dependency: "direct main" description: name: amplify_datastore url: "https://pub.flutter-io.cn" source: hosted version: "0.2.5" amplify_datastore_plugin_interface: dependency: transitive description: name: amplify_datastore_plugin_interface url: "https://pub.flutter-io.cn" source: hosted version: "0.2.5" amplify_flutter: dependency: "direct main" description: name: amplify_flutter url: "https://pub.flutter-io.cn" source: hosted version: "0.2.5" amplify_storage_plugin_interface: dependency: transitive description: name: amplify_storage_plugin_interface url: "https://pub.flutter-io.cn" source: hosted version: "0.2.5" amplify_storage_s3: dependency: "direct main" description: name: amplify_storage_s3 url: "https://pub.flutter-io.cn" source: hosted version: "0.2.5" another_flushbar: dependency: transitive description: name: another_flushbar url: "https://pub.flutter-io.cn" source: hosted version: "1.10.24" another_transformer_page_view: dependency: transitive description: name: another_transformer_page_view url: "https://pub.flutter-io.cn" source: hosted version: "1.0.1" archive: dependency: transitive description: name: archive url: "https://pub.flutter-io.cn" source: hosted version: "3.1.6" async: dependency: transitive description: name: async url: "https://pub.flutter-io.cn" source: hosted version: "2.8.1" awesome_dialog: dependency: "direct main" description: name: awesome_dialog url: "https://pub.flutter-io.cn" source: hosted version: "2.1.1" boolean_selector: dependency: transitive description: name: boolean_selector url: "https://pub.flutter-io.cn" source: hosted version: "2.1.0" cached_network_image: dependency: "direct main" description: name: cached_network_image url: "https://pub.flutter-io.cn" source: hosted version: "3.1.0" cached_network_image_platform_interface: dependency: transitive description: name: cached_network_image_platform_interface url: "https://pub.flutter-io.cn" source: hosted version: "1.0.0" cached_network_image_web: dependency: transitive description: name: cached_network_image_web url: "https://pub.flutter-io.cn" source: hosted version: "1.0.1" characters: dependency: transitive description: name: characters url: "https://pub.flutter-io.cn" source: hosted version: "1.1.0" charcode: dependency: transitive description: name: charcode url: "https://pub.flutter-io.cn" source: hosted version: "1.3.1" chewie: dependency: "direct main" description: name: chewie url: "https://pub.flutter-io.cn" source: hosted version: "1.2.2" clock: dependency: transitive description: name: clock url: "https://pub.flutter-io.cn" source: hosted version: "1.1.0" collection: dependency: transitive description: name: collection url: "https://pub.flutter-io.cn" source: hosted version: "1.15.0" crypto: dependency: transitive description: name: crypto url: "https://pub.flutter-io.cn" source: hosted version: "3.0.1" csslib: dependency: transitive description: name: csslib url: "https://pub.flutter-io.cn" source: hosted version: "0.17.1" cupertino_icons: dependency: "direct main" description: name: cupertino_icons url: "https://pub.flutter-io.cn" source: hosted version: "1.0.3" date_time_format: dependency: transitive description: name: date_time_format url: "https://pub.flutter-io.cn" source: hosted version: "2.0.1" fake_async: dependency: transitive description: name: fake_async url: "https://pub.flutter-io.cn" source: hosted version: "1.2.0" ffi: dependency: transitive description: name: ffi url: "https://pub.flutter-io.cn" source: hosted version: "1.1.2" file: dependency: transitive description: name: file url: "https://pub.flutter-io.cn" source: hosted version: "6.1.2" flare_flutter: dependency: transitive description: name: flare_flutter url: "https://pub.flutter-io.cn" source: hosted version: "3.0.2" flutter: dependency: "direct main" description: flutter source: sdk version: "0.0.0" flutter_blurhash: dependency: transitive description: name: flutter_blurhash url: "https://pub.flutter-io.cn" source: hosted version: "0.6.0" flutter_cache_manager: dependency: transitive description: name: flutter_cache_manager url: "https://pub.flutter-io.cn" source: hosted version: "3.1.2" flutter_custom_clippers: dependency: "direct main" description: name: flutter_custom_clippers url: "https://pub.flutter-io.cn" source: hosted version: "1.1.2" flutter_lints: dependency: "direct dev" description: name: flutter_lints url: "https://pub.flutter-io.cn" source: hosted version: "1.0.4" flutter_login: dependency: "direct main" description: name: flutter_login url: "https://pub.flutter-io.cn" source: hosted version: "2.2.1" flutter_plugin_android_lifecycle: dependency: transitive description: name: flutter_plugin_android_lifecycle url: "https://pub.flutter-io.cn" source: hosted version: "1.0.11" flutter_spinkit: dependency: "direct main" description: name: flutter_spinkit url: "https://pub.flutter-io.cn" source: hosted version: "5.1.0" flutter_test: dependency: "direct dev" description: flutter source: sdk version: "0.0.0" flutter_web_plugins: dependency: transitive description: flutter source: sdk version: "0.0.0" fluttertoast: dependency: "direct main" description: name: fluttertoast url: "https://pub.flutter-io.cn" source: hosted version: "8.0.8" font_awesome_flutter: dependency: transitive description: name: font_awesome_flutter url: "https://pub.flutter-io.cn" source: hosted version: "9.1.0" html: dependency: transitive description: name: html url: "https://pub.flutter-io.cn" source: hosted version: "0.15.0" http: dependency: transitive description: name: http url: "https://pub.flutter-io.cn" source: hosted version: "0.13.4" http_parser: dependency: transitive description: name: http_parser url: "https://pub.flutter-io.cn" source: hosted version: "4.0.0" image: dependency: "direct main" description: name: image url: "https://pub.flutter-io.cn" source: hosted version: "3.0.8" image_picker: dependency: "direct main" description: name: image_picker url: "https://pub.flutter-io.cn" source: hosted version: "0.6.5+3" intl: dependency: "direct main" description: name: intl url: "https://pub.flutter-io.cn" source: hosted version: "0.17.0" js: dependency: transitive description: name: js url: "https://pub.flutter-io.cn" source: hosted version: "0.6.3" lints: dependency: transitive description: name: lints url: "https://pub.flutter-io.cn" source: hosted version: "1.0.1" matcher: dependency: transitive description: name: matcher url: "https://pub.flutter-io.cn" source: hosted version: "0.12.10" meta: dependency: transitive description: name: meta url: "https://pub.flutter-io.cn" source: hosted version: "1.7.0" nested: dependency: transitive description: name: nested url: "https://pub.flutter-io.cn" source: hosted version: "1.0.0" octo_image: dependency: transitive description: name: octo_image url: "https://pub.flutter-io.cn" source: hosted version: "1.0.0+1" path: dependency: transitive description: name: path url: "https://pub.flutter-io.cn" source: hosted version: "1.8.0" path_provider: dependency: transitive description: name: path_provider url: "https://pub.flutter-io.cn" source: hosted version: "2.0.5" path_provider_linux: dependency: transitive description: name: path_provider_linux url: "https://pub.flutter-io.cn" source: hosted version: "2.1.0" path_provider_macos: dependency: transitive description: name: path_provider_macos url: "https://pub.flutter-io.cn" source: hosted version: "2.0.2" path_provider_platform_interface: dependency: transitive description: name: path_provider_platform_interface url: "https://pub.flutter-io.cn" source: hosted version: "2.0.1" path_provider_windows: dependency: transitive description: name: path_provider_windows url: "https://pub.flutter-io.cn" source: hosted version: "2.0.3" pedantic: dependency: transitive description: name: pedantic url: "https://pub.flutter-io.cn" source: hosted version: "1.11.1" petitparser: dependency: transitive description: name: petitparser url: "https://pub.flutter-io.cn" source: hosted version: "4.4.0" pin_code_fields: dependency: "direct main" description: name: pin_code_fields url: "https://pub.flutter-io.cn" source: hosted version: "7.3.0" platform: dependency: transitive description: name: platform url: "https://pub.flutter-io.cn" source: hosted version: "3.0.2" plugin_platform_interface: dependency: transitive description: name: plugin_platform_interface url: "https://pub.flutter-io.cn" source: hosted version: "2.0.2" process: dependency: transitive description: name: process url: "https://pub.flutter-io.cn" source: hosted version: "4.2.3" provider: dependency: transitive description: name: provider url: "https://pub.flutter-io.cn" source: hosted version: "5.0.0" quiver: dependency: transitive description: name: quiver url: "https://pub.flutter-io.cn" source: hosted version: "3.0.1+1" rxdart: dependency: transitive description: name: rxdart url: "https://pub.flutter-io.cn" source: hosted version: "0.27.2" shared_preferences: dependency: "direct main" description: name: shared_preferences url: "https://pub.flutter-io.cn" source: hosted version: "2.0.8" shared_preferences_linux: dependency: transitive description: name: shared_preferences_linux url: "https://pub.flutter-io.cn" source: hosted version: "2.0.2" shared_preferences_macos: dependency: transitive description: name: shared_preferences_macos url: "https://pub.flutter-io.cn" source: hosted version: "2.0.2" shared_preferences_platform_interface: dependency: transitive description: name: shared_preferences_platform_interface url: "https://pub.flutter-io.cn" source: hosted version: "2.0.0" shared_preferences_web: dependency: transitive description: name: shared_preferences_web url: "https://pub.flutter-io.cn" source: hosted version: "2.0.2" shared_preferences_windows: dependency: transitive description: name: shared_preferences_windows url: "https://pub.flutter-io.cn" source: hosted version: "2.0.2" shimmer: dependency: "direct main" description: name: shimmer url: "https://pub.flutter-io.cn" source: hosted version: "2.0.0" simple_animations: dependency: transitive description: name: simple_animations url: "https://pub.flutter-io.cn" source: hosted version: "3.2.0" simple_gesture_detector: dependency: transitive description: name: simple_gesture_detector url: "https://pub.flutter-io.cn" source: hosted version: "0.2.0" sizer: dependency: "direct main" description: name: sizer url: "https://pub.flutter-io.cn" source: hosted version: "2.0.15" sky_engine: dependency: transitive description: flutter source: sdk version: "0.0.99" sleek_circular_slider: dependency: "direct main" description: name: sleek_circular_slider url: "https://pub.flutter-io.cn" source: hosted version: "2.0.1" source_span: dependency: transitive description: name: source_span url: "https://pub.flutter-io.cn" source: hosted version: "1.8.1" sqflite: dependency: transitive description: name: sqflite url: "https://pub.flutter-io.cn" source: hosted version: "2.0.0+4" sqflite_common: dependency: transitive description: name: sqflite_common url: "https://pub.flutter-io.cn" source: hosted version: "2.0.1+1" stack_trace: dependency: transitive description: name: stack_trace url: "https://pub.flutter-io.cn" source: hosted version: "1.10.0" stream_channel: dependency: transitive description: name: stream_channel url: "https://pub.flutter-io.cn" source: hosted version: "2.1.0" string_scanner: dependency: transitive description: name: string_scanner url: "https://pub.flutter-io.cn" source: hosted version: "1.1.0" supercharged: dependency: transitive description: name: supercharged url: "https://pub.flutter-io.cn" source: hosted version: "2.1.1" supercharged_dart: dependency: transitive description: name: supercharged_dart url: "https://pub.flutter-io.cn" source: hosted version: "2.1.1" synchronized: dependency: transitive description: name: synchronized url: "https://pub.flutter-io.cn" source: hosted version: "3.0.0" table_calendar: dependency: "direct main" description: name: table_calendar url: "https://pub.flutter-io.cn" source: hosted version: "3.0.2" term_glyph: dependency: transitive description: name: term_glyph url: "https://pub.flutter-io.cn" source: hosted version: "1.2.0" test_api: dependency: transitive description: name: test_api url: "https://pub.flutter-io.cn" source: hosted version: "0.4.2" typed_data: dependency: transitive description: name: typed_data url: "https://pub.flutter-io.cn" source: hosted version: "1.3.0" universal_io: dependency: transitive description: name: universal_io url: "https://pub.flutter-io.cn" source: hosted version: "2.0.4" uuid: dependency: "direct main" description: name: uuid url: "https://pub.flutter-io.cn" source: hosted version: "3.0.5" vector_math: dependency: transitive description: name: vector_math url: "https://pub.flutter-io.cn" source: hosted version: "2.1.0" video_player: dependency: "direct main" description: name: video_player url: "https://pub.flutter-io.cn" source: hosted version: "2.2.5" video_player_platform_interface: dependency: transitive description: name: video_player_platform_interface url: "https://pub.flutter-io.cn" source: hosted version: "4.2.0" video_player_web: dependency: transitive description: name: video_player_web url: "https://pub.flutter-io.cn" source: hosted version: "2.0.4" wakelock: dependency: transitive description: name: wakelock url: "https://pub.flutter-io.cn" source: hosted version: "0.5.6" wakelock_macos: dependency: transitive description: name: wakelock_macos url: "https://pub.flutter-io.cn" source: hosted version: "0.4.0" wakelock_platform_interface: dependency: transitive description: name: wakelock_platform_interface url: "https://pub.flutter-io.cn" source: hosted version: "0.3.0" wakelock_web: dependency: transitive description: name: wakelock_web url: "https://pub.flutter-io.cn" source: hosted version: "0.4.0" wakelock_windows: dependency: transitive description: name: wakelock_windows url: "https://pub.flutter-io.cn" source: hosted version: "0.2.0" win32: dependency: transitive description: name: win32 url: "https://pub.flutter-io.cn" source: hosted version: "2.2.9" xdg_directories: dependency: transitive description: name: xdg_directories url: "https://pub.flutter-io.cn" source: hosted version: "0.2.0" xml: dependency: transitive description: name: xml url: "https://pub.flutter-io.cn" source: hosted version: "5.3.1" sdks: dart: ">=2.14.0 <3.0.0" flutter: ">=2.5.0" ```

[HuiSF]

Hello @zamargab DataStore has a default limit (10,000) to determine how many data entries to be synced considering performance. This limit can be configured, please refer to the document DataStore custom configuration.

[zamargab]

What is the recommended max records? i have a table with 900,000 rows

[HuiSF]

What is the recommended max records? i have a table with 900,000 rows

It depends on your use case :) 900,000 is a big number, it may take a while to sync all data. Here are some tips may help:

• Ensure to start any DB operations until the ready event is emitted from Amplify Hub (example)
• Provide a loading screen or something similar for the long syncing process

[zamargab]

Thanks......

Quick question, I opened an issue some days ago, about not been able to view the data i entered through a Lambda function, You suggested to generate a UUID for my records and that solved the problem but i have a question, since I don not really need to use the "createdAt", "_lastChangedAt", "updatedAt" fields, I hard coded those values when creating the DynamoDB records, do you think this might cause perfromance issues?

[HuiSF]

Thanks......

Quick question, I opened an issue some days ago, about not been able to view the data i entered through a Lambda function, You suggested to generate a UUID for my records and that solved the problem but i have a question, since I don not really need to use the "createdAt", "_lastChangedAt", "updatedAt" fields, I hard coded those values when creating the DynamoDB records, do you think this might cause perfromance issues?

Hum, I think as long as the values for these fields are accurate and with correct data type, and you are not expecting to sort data based on these field, that should be fine...

[zamargab]

Awesome

[zamargab]

Like I mentioned earlier, my table has about 900,000 rows and I realized it took nearly 20 minutes to sync about 40,000 rows, that means it might take a considerable amount of time to actually sync all 900,000. This might not be good for user experience seeing they'll have to wait for a long time before they can make any database operation.. I have a question, is it possible to eliminate the local database and keep all data in the cloud and make queries to the Cloud database every time a need arises? Seeing it takes a long time to sync the cloud database and the local.

[HuiSF]

Yes @zamargab it sounds like you don't need real time data syncing for your use case.

You can stop using DataStore by remove DataStore plugin and use only the API plugin.

If you set up DataStore initially, you may need to run

amplify update api

Follow the instructions to Disable DataStore for entire API.

And then run

amplify push

To redeploy your App to AWS. This operation will remove all DataStore data attributes from your DynamoDB.

Please follow Amplify documents of how to use API plugin with GraphQL. Or you can choose to use REST instead to provide data entry from the 900,0000 rows.

If you want to use DataStore for other use case, e.g. persisting user preference you can add another API to use with.