search

aws-amplify / amplify-flutter

A declarative library with an easy-to-use interface for building Flutter applications on AWS.
https://docs.amplify.aws
Apache License 2.0
1.33k stars 248 forks source link

Api Query does not return all query results (GraphQL) #1877

Closed zamarwells closed 1 year ago

zamarwells commented 2 years ago

Description

I am accessing my database via API and I have noticed that when I query the database from my flutter app, I don't get all the desired query results. At first I thought this issue was because datastore was not syncing, I checkded the admin console, when I navigate to the content area, I can actually see all the desired results. I have also tried to access data through datastore and it comes out fine. I only have issue when I decide to use API.

P.S, I opened an issue on this sometime ago but I have been away from this project for a while and I don't have access to the GitHub account anymore, hence I had to open a new issue.

Categories

Steps to Reproduce

  1. Create a tables
  2. Add data via datastore or Api
  3. Query data using API.

below is the code I am using to query the database

try {
      final request = ModelQueries.list(IndividualRecipee.classType,
          where: IndividualRecipee.MEALTEMPLATEIDNUMBER
              .eq(activemealTemplateID)
              .and(IndividualRecipee.DAYNO.eq(dayNo))
              .and(IndividualRecipee.WEEKNO.eq(weekNo)));
      final response = await Amplify.API.query(request: request).response;

      List<IndividualRecipee?>? todos = response.data?.items;
      if (todos == null) {
        print('errors: ' + response.errors.toString());
        return;
      }
      print('Query result: ' + todos.toString());
    } on ApiException catch (e) {
      print('Query failed: $e');
    }

Screenshots

No response

Platforms

Android Device/Emulator API Level

API 31

Environment

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 2.10.4, on macOS 12.4 21F79 darwin-x64, locale en-GB)
[!] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.
[✓] Xcode - develop for iOS and macOS (Xcode 13.4.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2021.2)
[✓] VS Code (version 1.68.1)
[✓] Connected device (3 available)
[✓] HTTP Host Availability

Dependencies

Dart SDK 2.16.2
Flutter SDK 2.10.4
clientapp 1.0.0+10

dependencies:
- amplify_api 0.6.1 [amplify_api_android amplify_api_ios amplify_core amplify_flutter aws_common collection flutter meta plugin_platform_interface]
- amplify_auth_cognito 0.6.1 [amplify_auth_cognito_android amplify_auth_cognito_ios amplify_core aws_common collection flutter meta plugin_platform_interface]
- amplify_datastore 0.6.1 [flutter amplify_datastore_plugin_interface amplify_core plugin_platform_interface meta collection async]
- amplify_flutter 0.6.1 [amplify_core amplify_datastore_plugin_interface amplify_flutter_android amplify_flutter_ios aws_common collection flutter meta plugin_platform_interface]
- amplify_storage_s3 0.6.1 [amplify_storage_s3_android amplify_storage_s3_ios amplify_core aws_common flutter meta plugin_platform_interface]
- awesome_dialog 2.1.1 [flutter simple_animations flare_flutter]
- cached_network_image 3.2.0 [flutter flutter_cache_manager octo_image cached_network_image_platform_interface cached_network_image_web]
- charts_flutter 0.12.0 [charts_common collection flutter intl logging meta]
- chewie 1.3.1 [cupertino_icons flutter provider video_player wakelock very_good_analysis]
- cloud_functions 3.2.5 [cloud_functions_platform_interface cloud_functions_web firebase_core firebase_core_platform_interface flutter]
- country_code_picker 2.0.2 [flutter modal_bottom_sheet collection universal_platform]
- cupertino_icons 1.0.4
- emoji_picker_flutter 1.0.8 [flutter shared_preferences]
- firebase_auth 3.3.5 [firebase_auth_platform_interface firebase_auth_web firebase_core firebase_core_platform_interface flutter meta]
- firebase_core 1.11.0 [firebase_core_platform_interface firebase_core_web flutter meta]
- firebase_messaging 10.0.9 [firebase_core firebase_core_platform_interface firebase_messaging_platform_interface firebase_messaging_web flutter meta]
- fl_chart 0.40.6 [flutter equatable]
- flutter 0.0.0 [characters collection material_color_utilities meta typed_data vector_math sky_engine]
- flutter_blue 0.8.0 [flutter convert protobuf rxdart collection meta]
- flutter_chat_bubble 2.0.0 [flutter]
- flutter_launcher_icons 0.9.2 [args image path yaml]
- flutter_spinkit 5.1.0 [flutter]
- flutter_svg 1.0.3 [flutter meta path_drawing vector_math xml]
- fluttertoast 8.0.8 [flutter flutter_web_plugins]
- http 0.13.4 [async http_parser meta path]
- image 3.1.0 [archive meta xml]
- image_picker 0.7.5+4 [flutter flutter_plugin_android_lifecycle image_picker_platform_interface image_picker_for_web]
- in_app_purchase 3.0.5 [flutter in_app_purchase_android in_app_purchase_platform_interface in_app_purchase_storekit]
- intl 0.17.0 [clock path]
- is_first_run 1.0.0 [flutter shared_preferences]
- path_provider 2.0.8 [flutter path_provider_android path_provider_ios path_provider_linux path_provider_macos path_provider_platform_interface path_provider_windows]
- pay 1.0.8 [flutter flutter_localizations pay_android pay_ios pay_platform_interface meta]
- pin_code_fields 7.3.0 [flutter]
- provider 6.0.2 [collection flutter nested]
- purchases_flutter 3.10.0 [flutter freezed_annotation json_annotation]
- shared_preferences 2.0.9 [flutter meta shared_preferences_android shared_preferences_ios shared_preferences_linux shared_preferences_macos shared_preferences_platform_interface shared_preferences_web shared_preferences_windows]
- shimmer 2.0.0 [flutter]
- sizer 2.0.15 [universal_io flutter]
- sleek_circular_slider 2.0.1 [flutter]
- table_calendar 3.0.3 [flutter intl simple_gesture_detector]
- url_launcher 6.1.1 [flutter url_launcher_android url_launcher_ios url_launcher_linux url_launcher_macos url_launcher_platform_interface url_launcher_web url_launcher_windows]
- uuid 3.0.6 [crypto]
- video_player 2.2.7 [flutter meta video_player_platform_interface video_player_web html]
- wakelock 0.6.1+2 [flutter meta wakelock_macos wakelock_platform_interface wakelock_web wakelock_windows]
- webview_flutter 3.0.0 [flutter webview_flutter_android webview_flutter_platform_interface webview_flutter_wkwebview]
- xfile 1.1.0

transitive dependencies:
- amplify_api_android 0.6.1 [flutter]
- amplify_api_ios 0.6.1 [amplify_core flutter]
- amplify_auth_cognito_android 0.6.1 [flutter]
- amplify_auth_cognito_ios 0.6.1 [amplify_core flutter]
- amplify_core 0.6.1 [aws_common collection flutter intl json_annotation meta plugin_platform_interface uuid]
- amplify_datastore_plugin_interface 0.6.1 [amplify_core collection flutter meta]
- amplify_flutter_android 0.6.1 [flutter]
- amplify_flutter_ios 0.6.1 [amplify_core flutter]
- amplify_storage_s3_android 0.6.1 [flutter]
- amplify_storage_s3_ios 0.6.1 [flutter]
- archive 3.1.6 [crypto path]
- args 2.3.0
- async 2.8.2 [collection meta]
- aws_common 0.1.1 [async collection http meta stream_transform uuid]
- boolean_selector 2.1.0 [source_span string_scanner]
- cached_network_image_platform_interface 1.0.0 [flutter flutter_cache_manager]
- cached_network_image_web 1.0.1 [flutter flutter_cache_manager cached_network_image_platform_interface]
- characters 1.2.0
- charcode 1.3.1
- charts_common 0.12.0 [collection intl logging meta vector_math]
- clock 1.1.0
- cloud_functions_platform_interface 5.0.20 [firebase_core flutter meta plugin_platform_interface]
- cloud_functions_web 4.2.6 [cloud_functions_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins js]
- collection 1.15.0
- convert 3.0.1 [typed_data]
- cross_file 0.3.2 [flutter js meta]
- crypto 3.0.1 [collection typed_data]
- csslib 0.17.1 [source_span]
- equatable 2.0.3 [collection meta]
- fake_async 1.2.0 [clock collection]
- ffi 1.1.2
- file 6.1.2 [meta path]
- firebase_auth_platform_interface 6.1.10 [firebase_core flutter meta plugin_platform_interface]
- firebase_auth_web 3.3.6 [firebase_auth_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins http_parser intl js meta]
- firebase_core_platform_interface 4.2.3 [collection flutter meta plugin_platform_interface]
- firebase_core_web 1.5.3 [firebase_core_platform_interface flutter flutter_web_plugins js meta]
- firebase_messaging_platform_interface 3.1.6 [firebase_core flutter meta plugin_platform_interface]
- firebase_messaging_web 2.2.6 [firebase_core firebase_core_web firebase_messaging_platform_interface flutter flutter_web_plugins js meta]
- fixnum 1.0.0
- flare_flutter 3.0.2 [collection flutter meta]
- flutter_blurhash 0.6.0 [flutter meta pedantic]
- flutter_cache_manager 3.3.0 [clock collection file flutter http path path_provider pedantic rxdart sqflite uuid]
- flutter_lints 1.0.4 [lints]
- flutter_localizations 0.0.0 [flutter intl characters clock collection material_color_utilities meta path typed_data vector_math]
- flutter_plugin_android_lifecycle 2.0.5 [flutter]
- flutter_test 0.0.0 [flutter test_api path fake_async clock stack_trace vector_math async boolean_selector characters charcode collection matcher material_color_utilities meta source_span stream_channel string_scanner term_glyph typed_data]
- flutter_web_plugins 0.0.0 [flutter js characters collection material_color_utilities meta typed_data vector_math]
- freezed_annotation 1.1.0 [collection json_annotation meta]
- html 0.15.0 [csslib source_span]
- http_parser 4.0.0 [charcode collection source_span string_scanner typed_data]
- image_picker_for_web 2.1.4 [flutter flutter_web_plugins image_picker_platform_interface meta]
- image_picker_platform_interface 2.4.1 [flutter http meta plugin_platform_interface cross_file]
- in_app_purchase_android 0.2.2+7 [collection flutter in_app_purchase_platform_interface json_annotation]
- in_app_purchase_platform_interface 1.3.0 [flutter plugin_platform_interface]
- in_app_purchase_storekit 0.3.0+9 [collection flutter in_app_purchase_platform_interface json_annotation]
- js 0.6.3
- json_annotation 4.4.0 [meta]
- lints 1.0.1
- logging 1.0.2
- matcher 0.12.11 [stack_trace]
- material_color_utilities 0.1.3
- meta 1.7.0
- modal_bottom_sheet 2.0.0 [flutter]
- nested 1.0.0 [flutter]
- octo_image 1.0.1 [flutter flutter_blurhash]
- path 1.8.0
- path_drawing 1.0.0 [vector_math meta path_parsing flutter]
- path_parsing 1.0.0 [vector_math meta]
- path_provider_android 2.0.9 [flutter path_provider_platform_interface]
- path_provider_ios 2.0.7 [flutter path_provider_platform_interface]
- path_provider_linux 2.1.2 [flutter path path_provider_platform_interface xdg_directories]
- path_provider_macos 2.0.4 [flutter meta path_provider_platform_interface]
- path_provider_platform_interface 2.0.1 [flutter meta platform plugin_platform_interface]
- path_provider_windows 2.0.4 [ffi flutter meta path path_provider_platform_interface win32]
- pay_android 1.0.6 [flutter flutter_localizations pay_platform_interface flutter_svg]
- pay_ios 1.0.6 [flutter pay_platform_interface]
- pay_platform_interface 1.0.2 [flutter yaml]
- pedantic 1.11.1
- petitparser 4.4.0 [meta]
- platform 3.1.0
- plugin_platform_interface 2.0.2 [meta]
- process 4.2.4 [file path platform]
- protobuf 2.0.1 [fixnum collection]
- rxdart 0.26.0
- shared_preferences_android 2.0.9 [flutter meta shared_preferences_platform_interface]
- shared_preferences_ios 2.0.8 [flutter meta shared_preferences_platform_interface]
- shared_preferences_linux 2.0.3 [file flutter meta path path_provider_linux shared_preferences_platform_interface]
- shared_preferences_macos 2.0.2 [flutter shared_preferences_platform_interface]
- shared_preferences_platform_interface 2.0.0 [flutter]
- shared_preferences_web 2.0.2 [flutter flutter_web_plugins meta shared_preferences_platform_interface]
- shared_preferences_windows 2.0.3 [file flutter meta path path_provider_platform_interface path_provider_windows shared_preferences_platform_interface]
- simple_animations 3.2.0 [flutter supercharged flutter_lints]
- simple_gesture_detector 0.2.0 [flutter]
- sky_engine 0.0.99
- source_span 1.8.1 [collection path term_glyph]
- sqflite 2.0.1 [flutter sqflite_common path]
- sqflite_common 2.0.1+1 [synchronized path meta]
- stack_trace 1.10.0 [path]
- stream_channel 2.1.0 [async]
- stream_transform 2.0.0
- string_scanner 1.1.0 [charcode source_span]
- supercharged 2.1.1 [supercharged_dart flutter]
- supercharged_dart 2.1.1
- synchronized 3.0.0
- term_glyph 1.2.0
- test_api 0.4.8 [async boolean_selector collection meta source_span stack_trace stream_channel string_scanner term_glyph matcher]
- typed_data 1.3.0 [collection]
- universal_io 2.0.4 [collection crypto meta typed_data]
- universal_platform 1.0.0+1
- url_launcher_android 6.0.17 [flutter url_launcher_platform_interface]
- url_launcher_ios 6.0.16 [flutter url_launcher_platform_interface]
- url_launcher_linux 3.0.1 [flutter url_launcher_platform_interface]
- url_launcher_macos 3.0.1 [flutter url_launcher_platform_interface]
- url_launcher_platform_interface 2.0.4 [flutter plugin_platform_interface]
- url_launcher_web 2.0.10 [flutter flutter_web_plugins url_launcher_platform_interface]
- url_launcher_windows 3.0.1 [flutter url_launcher_platform_interface]
- vector_math 2.1.1
- very_good_analysis 2.4.0
- video_player_platform_interface 4.2.0 [flutter flutter_test meta]
- video_player_web 2.0.4 [flutter flutter_web_plugins meta video_player_platform_interface]
- wakelock_macos 0.4.0 [flutter flutter_web_plugins wakelock_platform_interface]
- wakelock_platform_interface 0.3.0 [flutter meta]
- wakelock_web 0.4.0 [flutter flutter_web_plugins js wakelock_platform_interface]
- wakelock_windows 0.2.0 [flutter wakelock_platform_interface win32]
- webview_flutter_android 2.8.2 [flutter webview_flutter_platform_interface]
- webview_flutter_platform_interface 1.8.0 [flutter plugin_platform_interface]
- webview_flutter_wkwebview 2.7.1 [flutter webview_flutter_platform_interface]
- win32 2.3.1 [ffi]
- xdg_directories 0.2.0 [meta path process]
- xml 5.3.1 [collection meta petitparser]
- yaml 3.1.0 [collection source_span string_scanner]

Device

Iphone 12

OS

IOS 15.5

CLI Version

8.4.0

Additional Context

No response

Lorenzohidalgo commented 2 years ago

I've not yet worked with the premade requests ModelQueries but a common error/issue is expecting DynamoDB/AppSync to work as a relational database when querying.

In this case, you are filtering the results of a list request, something you should have in mind is that the filters are applied after the resolver has fetched the first n items from DynamoDB (being n the limit applied to the number of items to compare, usually 100).

It works in the following order:

  1. n items are fetched from DynamoDB
  2. The filter is applied to the fetched items
  3. You'll get the items that match your filter

If none of the fetched items match your filter, you'll receive an empty list as a response with a non-null nextToken. You should re-send the same query with the received nextToken value until nextToken == null or you've received the desired number of items.

Hope this helps.

zamarwells commented 2 years ago 
try {
      final request = ModelQueries.list(IndividualRecipee.classType,
          where: IndividualRecipee.MEALTEMPLATEIDNUMBER
              .eq(activemealTemplateID));
      final response = await Amplify.API.query(request: request).response;

      List<IndividualRecipee?>? todos = response.data?.items;
      if (todos == null) {
        print('errors: ' + response.errors.toString());
        return;
      }
      print('Query result: ' + todos.toString());
    } on ApiException catch (e) {
      print('Query failed: $e');
    }

I'm sorry, above comment did not help me at all, take a look at the query above, even after removing two query conditions, it still does not return the right number of results.

spbarber commented 2 years ago

Hi @zamarwells,

I get the same issue when using Swift on iOS.

If I use the Datastore it works fine, but querying using the API with a custom query doesn't return the results shown in the Amplify Studio Content section.

Even if I go to AppSync's Query section and run the GraphQL query, it returns the wrong results.

It would be great if we could get to the bottom of this issue soonest!

There seems to be a difference in how the Datastore queries the data compared to querying the API directly.

Lorenzohidalgo commented 2 years ago

@zamarwells maybe this part in the docs will help. As mentioned in my previous comment, you need to fetch all the results not just the first page, maybe that is the issue you are facing.

@spbarber did you check in the AppSync's Query section if the returned nextToken is set to null? Otherwise, it could be the issue that I previously mentioned and you would need to fetch all paginated results.

spbarber commented 2 years ago

Hi @Lorenzohidalgo

Ah ha, if I query with the limit of 10000 then I get all the results I see in the Amplify Studio portal and on local storage.

query MyQuery { listInboxs(filter: {propertyId: {eq: 186}}, limit: 10000) { items { id } } }

That is strange, a better way to do that?

Lorenzohidalgo commented 2 years ago

I've not yet worked with the premade requests ModelQueries but a common error/issue is expecting DynamoDB/AppSync to work as a relational database when querying.

In this case, you are filtering the results of a list request, something you should have in mind is that the filters are applied after the resolver has fetched the first n items from DynamoDB (being n the limit applied to the number of items to compare, usually 100).

It works in the following order:

  1. n items are fetched from DynamoDB
  2. The filter is applied to the fetched items
  3. You'll get the items that match your filter

If none of the fetched items match your filter, you'll receive an empty list as a response with a non-null nextToken. You should re-send the same query with the received nextToken value until nextToken == null or you've received the desired number of items.

Hope this helps.

Hey @spbarber, as per my first comment, the limit applies to the number of items that will be scanned in DynamoDB, not the number of items to be returned.

A DynamoDB Scan is also a fairly expensive operation, take a look at the following post. I would recommend you to create GSIs to avoid that.

For example, if you create a GSI on propertyId you will be able to query by the GSI and not need to scan the whole table.

Another options to kind of force DynamoDB to work as a relational DB would be to use the @searchable directive, take a look at the docs for more info.

The differences you might be seeing between querying DataStore and AppSync might be caused because DataStore transforms the data into a local SQLlite DB (or at least that was the case the last time I worked with DataStore)

dnys1 commented 2 years ago

Thank you @Lorenzohidalgo for all the incredible resources and insights.

@zamarwells @spbarber - do you feel like you have enough info to mitigate the issue your seeing with the list query result set? If you feel like the experience of DataStore is correct, then the pagination docs should help you replicate that and/or increasing the search limit (noting the considerations above).

If you feel like neither is giving the results you want, you may also wish to explore custom indexes, as Lorenzo mentioned, or custom resolvers which allow for attaching arbitrary logic to your GraphQL API.

Mozart-Alkhateeb commented 1 year ago

Indexes helped me out in my use-case, it will also save you from looping over unnecessary number of rows on the server. https://docs.amplify.aws/cli/graphql/data-modeling/#configure-a-secondary-index I tried with a single index and it returns up to 100 items based on the specified filter in the above example it is accountRepresentativeID.

I don't know if it will work when filtering by multiple fields.

Jordan-Nelson commented 1 year ago

Hey Folks - I am going to close this out since I think the original issue/question was resolved. If you are facing an further issues, please open a new issue.