search

aws-amplify / amplify-category-api

The AWS Amplify CLI is a toolchain for simplifying serverless web and mobile development. This plugin provides functionality for the API category, allowing for the creation and management of GraphQL and REST based backends for your amplify project.
https://docs.amplify.aws/
Apache License 2.0
88 stars 76 forks source link

OpenSearch: @searchable - wildcard, regexp matching returns 0 results #1038

Open majirosstefan opened 2 years ago

majirosstefan commented 2 years ago

Before opening, please confirm:

JavaScript Framework

React, React Native

Amplify APIs

GraphQL API

Amplify Categories

api

Environment information

``` # Put output below this line System: OS: macOS 12.1 CPU: (8) x64 Apple M1 Pro Memory: 145.91 MB / 16.00 GB Shell: 5.8 - /bin/zsh Binaries: Node: 12.22.10 - ~/.nvm/versions/node/v12.22.10/bin/node Yarn: 1.22.17 - ~/.nvm/versions/node/v12.22.10/bin/yarn npm: 6.14.16 - ~/.nvm/versions/node/v12.22.10/bin/npm Watchman: 2022.03.21.00 - /opt/homebrew/bin/watchman Browsers: Brave Browser: 103.1.40.107 Chrome: 103.0.5060.114 Safari: 15.2 npmPackages: @alfonmga/react-lottie-light-ts: ^0.0.1 => 0.0.1 @aws-amplify/pushnotification: ^4.3.8 => 4.3.8 @babel/core: ^7.8.6 => 7.14.8 (7.17.5, 7.9.0, 7.16.0) @cometchat-pro/chat: ^3.0.4 => 3.0.6 @cometchat-pro/react-native-chat: 3.0.0 => 3.0.0 @expo-google-fonts/inter: ^0.2.0 => 0.2.0 @expo/match-media: ^0.3.0 => 0.3.0 @expo/webpack-config: ^0.16.6 => 0.16.16 @react-native-async-storage/async-storage: ^1.12.1 => 1.15.9 @react-native-clipboard/clipboard: ^1.9.0 => 1.9.0 @react-native-community/cameraroll: ^4.1.2 => 4.1.2 @react-native-community/checkbox: ^0.5.1 => 0.5.8 @react-native-community/masked-view: 0.1.10 => 0.1.10 @react-native-community/netinfo: 6.0.2 => 6.0.2 @react-native-community/push-notification-ios: ^1.8.0 => 1.8.0 (1.0.3) @react-native-community/slider: 3.0.3 => 3.0.3 @react-native-picker/picker: ^2.1.0 => 2.2.1 @react-navigation/bottom-tabs: ^6.0.5 => 6.0.5 @react-navigation/drawer: ^6.3.1 => 6.3.1 @react-navigation/native: ^6.0.2 => 6.0.2 @react-navigation/stack: ^6.0.7 => 6.0.7 @sentry/react-native: ^3.2.4 => 3.2.5 @storybook/addon-actions: ^6.0.21 => 6.3.6 @storybook/addon-knobs: ^6.0.21 => 6.3.0 @storybook/addon-ondevice-actions: ^5.3.21 => 5.3.23 @storybook/addon-ondevice-knobs: ^5.3.21 => 5.3.25 @storybook/react-native: ^5.3.21 => 5.3.25 @stripe/react-stripe-js: ^1.7.0 => 1.7.0 @stripe/stripe-js: ^1.23.0 => 1.23.0 @svgr/webpack: ^5.4.0 => 5.5.0 @types/howler: ^2.2.4 => 2.2.4 @types/lodash: ^4.14.161 => 4.14.172 @types/react: ~16.9.35 => 16.9.56 (17.0.15) @types/react-native: ~0.63.2 => 0.63.53 (0.65.1) @types/uuid: ^8.3.0 => 8.3.1 HelloWorld: 0.0.1 amazon-cognito-identity-js: ^4.4.0 => 4.6.3 (5.2.3) aws-amplify: ^4.3.11 => 4.3.11 aws-amplify-react-native: ^6.0.2 => 6.0.2 aws-sdk: ^2.1056.0 => 2.1057.0 babel-jest: ~25.2.6 => 25.2.6 (25.5.1) babel-loader: ^8.2.3 => 8.2.3 (8.1.0) babel-plugin-react-native-web: ^0.17.5 => 0.17.5 (0.13.18) buffer: ^6.0.3 => 6.0.3 (4.9.2, 5.7.1) dayjs: ^1.9.4 => 1.10.6 dotenv-webpack: ^7.1.0 => 7.1.0 emoji-mart-native: ^0.6.2-beta => 0.6.2-beta expo: ^39.0.0 => 39.0.5 expo-application: ~2.3.0 => 2.3.0 expo-av: ^9.2.3 => 9.2.3 expo-document-picker: ^9.2.3 => 9.2.4 expo-font: ^9.2.1 => 9.2.1 (8.3.0) expo-image-picker: ^10.2.2 => 10.2.2 expo-linking: 1.0.7 => 1.0.7 expo-permissions: ^12.1.1 => 12.1.1 (9.3.0) expo-splash-screen: ~0.6.2 => 0.6.2 expo-status-bar: ~1.0.2 => 1.0.4 expo-updates: ~0.3.2 => 0.3.5 faker: 5.5.3 => 5.5.3 file-loader: ^6.2.0 => 6.2.0 (6.0.0) hermes-inspector-msggen: 1.0.0 howler: ^2.2.3 => 2.2.3 install: ^0.13.0 => 0.13.0 jest: ~25.2.6 => 25.2.7 lodash: ^4.17.20 => 4.17.21 lottie-react-native: ~2.6.1 => 2.6.1 numeral: ^2.0.6 => 2.0.6 patch-package: ^6.4.7 => 6.4.7 plaid: ^10.1.0 => 10.1.0 postinstall-postinstall: ^2.1.0 => 2.1.0 prettier: 2.5.1 => 2.5.1 query-string: ^7.0.1 => 7.0.1 react: 16.13.1 => 16.13.1 (16.14.0) react-animated: 0.1.0 react-dom: 16.13.1 => 16.13.1 react-image-file-resizer: ^0.4.7 => 0.4.7 react-is: ^16.13.1 => 16.13.1 react-lottie: ^1.2.3 => 1.2.3 react-native: ~0.63.4 => 0.63.4 react-native-android-keyboard-adjust: ^1.2.0 => 1.2.0 react-native-autolink: ^4.0.0 => 4.0.0 react-native-blob-courier: 3.0.2 => 3.0.2 react-native-config: ^1.4.5 => 1.4.5 react-native-device-info: ^8.3.0 => 8.3.1 react-native-elements: ^3.4.2 => 3.4.2 react-native-fs: ^2.18.0 => 2.18.0 react-native-gesture-handler: ^1.10.3 => 1.10.3 react-native-get-random-values: ^1.5.0 => 1.7.0 react-native-hyperlink: ^0.0.19 => 0.0.19 react-native-image-resizer: ^1.4.5 => 1.4.5 react-native-keep-awake: ^4.0.0 => 4.0.0 react-native-keyboard-aware-scroll-view: https://github.com/APSL/react-native-keyboard-aware-scroll-view#fb40649a9c84aa23d6a55355e9e0e432795967a6 => 0.9.2 react-native-mov-to-mp4: ^0.2.2 => 0.2.2 react-native-plaid-link-sdk: ^7.2.1 => 7.2.1 react-native-portalize: ^1.0.7 => 1.0.7 react-native-reanimated: 1.13.2 => 1.13.2 react-native-safe-area-context: ^3.3.2 => 3.3.2 (3.1.4) react-native-screens: ^3.7.2 => 3.7.2 react-native-sound: ^0.11.1 => 0.11.1 react-native-svg: 12.1.0 => 12.1.0 react-native-svg-transformer: ^0.14.3 => 0.14.3 react-native-swipe-list-view: ^3.2.9 => 3.2.9 react-native-unimodules: ^0.14.6 => 0.14.6 react-native-vector-icons: ^8.1.0 => 8.1.0 react-native-web: ~0.17.5 => 0.17.5 react-native-webview: ^11.13.0 => 11.13.0 react-plaid-link: 3.2.2 => 3.2.2 react-responsive: ^9.0.0-beta.6 => 9.0.0-beta.6 react-test-renderer: ~16.13.1 => 16.13.1 read-excel-file: ^5.3.5 => 5.3.5 read-excel-file/commonjs: undefined () read-excel-file/node: 1.0.0 read-excel-file/schema: 1.0.0 read-excel-file/web-worker: 1.0.0 reanimated-bottom-sheet: ^1.0.0-alpha.22 => 1.0.0-alpha.22 rn-fetch-blob: ^0.12.0 => 0.12.0 stripe: ^8.135.0 => 8.209.0 tipsi-stripe: ^8.0.2 => 8.0.2 typescript: ^4.3.2 => 4.5.5 url-loader: ^4.1.1 => 4.1.1 uuid: ^8.3.2 => 8.3.2 (3.4.0, 3.3.2, 7.0.2, 7.0.3) webpack: ^4.43.0 => 4.46.0 (4.43.0) npmGlobalPackages: @aws-amplify/cli: 5.1.0 expo-cli: 5.3.0 npm: 6.14.16 yarn: 1.22.17 ```

Describe the bug

Running this code gives me zero results (Index contains data).

I am running app on iOS & Android simulator. I get the same response even when I am running react-native in web environment.

I am using "ElasticSearchInstanceType": "t2.small.elasticsearch".

I would expect it returned something back.

 const variables: SearchSchoolsQueryVariables = {
      filter:  {
        schoolName: {            
          wildcard: "*Lo*"
        },
      },
    };

 const response = (await API.graphql(
      graphqlOperation(searchSchools, variables)
    )) as GraphQLResult<SearchSchoolsQuery>;

schema.graphql: 

type School
  @model(subscriptions: null)
  @auth(
    rules: [
      { allow: private, operations: [read] }
      { allow: public, operations: [read] }
    ]
  )
  @key(
    name: "schoolsByName"
    fields: ["schoolName"]
    queryField: "allSchoolsByName"
  )
  @searchable
  {
  id: ID!  
  schoolName: String!
  createdAt: AWSDateTime!
  updatedAt: AWSDateTime!
}

Expected behavior

It would return non-empty response.

Reproduction steps

Run the search & add dummy data.

Code Snippet

// Put your code below this line.

Log output

``` // Put your logs below this line ```

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

Web, iOS, Android

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

chrisbonifacio commented 2 years ago

Hi @majirosstefan 👋 thank you for raising this issue. Are you experiencing the same result when attempting the query in the AppSync console?

majirosstefan commented 2 years ago

Yes, @chrisbonifacio:

Screen Shot 2022-07-08 at 4 46 45 PM

Also tried with nested "or": Screen Shot 2022-07-08 at 4 49 08 PM

majirosstefan commented 2 years ago

If it helps, we are using OpenSearch / ElasticSearch engine with version 6.2:

Screen Shot 2022-07-08 at 4 50 12 PM

chrisbonifacio commented 2 years ago

@majirosstefan thank you for the info, we will be trying to reproduce this issue and get some feedback for you from the Amplify data team

chrisbonifacio commented 2 years ago

@majirosstefan would you happen to know if you might've created the records you're querying for before or after you added @searchable to the model? If you added them before, you might have to backfill the data using a script:

https://docs.amplify.aws/cli/graphql/troubleshooting/#backfill-opensearch-index-from-dynamodb-table

majirosstefan commented 2 years ago

@chrisbonifacio Although we were using searchable for another model before, this time we only needed to add/create another new model that is also searchable.

So I created the model that you could see above, and then imported data to DynamoDB // manually using nodeJS script that uses putOperation and Dynamo's Document Client. Not 100% sure, but Ddb uses streaming or something like that to propagate changes to Opensearch, no ?

Also, if the data simply was not present in OpenSearch so backfilling would be needed, how could it be possible that search with other conditions is working but it's only ignoring regexp and wildcard ?

chrisbonifacio commented 2 years ago

So I created the model that you could see above, and then imported data to DynamoDB // manually using nodeJS script that uses putOperation and Dynamo's Document Client. Not 100% sure, but Ddb uses streaming or something like that to propagate changes to Opensearch, no ?

Yes, any records created after opensearch has been added to the model will be streamed from DDB to OpenSearch and should be available via AppSync query.

Also, if the data simply was not present in OpenSearch so backfilling would be needed, how could it be possible that search with other conditions is working but it's only ignoring regexp and wildcard ?

Can you share the filtered queries that are working and which are not? If the records don't exist in OpenSearch but are still exist in the DDB table, they could still be queried by standard filters.

I noticed you are on version 5.1.0 of the Amplify CLI. Can you upgrade to the latest version, redeploy your schema and see if that makes a difference? Our attempts to reproduce the issue have been using version 9.1.0, we're just forcing the use of graphql transformer V1 by changing some settings in the cli.json file so we can deploy the schema you provided.

nadetastic commented 2 years ago

Hi @majirosstefan I attempted to reproduce this issue however, I am able to get the expected response:

Seeing the use of @key we assume you are using the v1 transformer.

CLI.json

  "features": {
    "graphqltransformer": {
      ...
      "useexperimentalpipelinedtransformer": false,
      ...
      "transformerversion": 1,
      ...
    },
    ...
  }

schema.graphl (copied from above)

type School
  @model(subscriptions: null)
  @auth(
    rules: [
      { allow: private, operations: [read] }
      { allow: public, operations: [read] }
    ]
  )
  @key(
    name: "schoolsByName"
    fields: ["schoolName"]
    queryField: "allSchoolsByName"
  )
  @searchable
  {
  id: ID!  
  schoolName: String!
  createdAt: AWSDateTime!
  updatedAt: AWSDateTime!
}

AppSync response opensearchissue

majirosstefan commented 2 years ago
  1. Yes, we are using transformer v1.
  2. I tried to find CLI.json in the project, but seems there is none (I have added all version of all dependencies when creating this ticket )
  3. Only matchPhrasePrefix, and match are working:

Screen Shot 2022-07-14 at 8 21 41 PM

Screen Shot 2022-07-14 at 8 22 01 PM

nadetastic commented 2 years ago

@majirosstefan Thanks for the confirmation.

Out of curiosity, have you tried running the queries in the OpenSearch generated Kibana URL to see if there's a response? Hoping to isolate the root of your issue.

If you haven't done this before here are some tips:

majirosstefan commented 2 years ago

will do, but in next few days, not today

Btw, should I take a look also on logs somewhere there ? Or just see if it returns anything (or an error maybe)

nadetastic commented 2 years ago

@majirosstefan yes. You may need to enable logging on the AppSync API if you haven't, then check the associated CloudWatch logs after running a new Query.

Also it can be helpful to check/share the AppSync Resolver mapping template for the searchSchools query as well.

majirosstefan commented 2 years ago

Btw, seems like "match" filter was removed from the docs:

https://docs.amplify.aws/cli-legacy/graphql-transformer/searchable/#usage

EDIT (no, it's still there // long hours in the work, sorry)

chrisbonifacio commented 2 years ago

@majirosstefan just want to follow up with you and see if you're still running into this issue. Also, have you tried upgrading your Amplify CLI version to the latest and see if that helps after redeploying your schema?

Also, when you can, can you please confirm that you are not able to query even newly created records that should fulfill those regex and wildcard filter expressions?

majirosstefan commented 2 years ago

yeah, still running into this. Did not have time to upgrade CLI or look into OpenSearch yet.

"Also, when you can, can you please confirm that you are not able to query even newly created records that should fulfill those regex and wildcard filter expressions?" == yes

chrisbonifacio commented 2 years ago

@majirosstefan No worries, let us know when you get a chance to test if upgrading makes a difference!

internetammo commented 2 years ago

same issue here I just created my search with amplify looks like it provisions Elasticsearch 6.2 by default and only match and matchPhrasePrefix are working even though I didnt see them in the docs wildcard and regex dont seem to work

elorzafe commented 1 year ago

I will transfer this issue to Amplify CLI repo, they will handle this better than me

AnilMaktala commented 1 year ago

Hi @majirosstefan, Have you had an opportunity to upgrade to the latest version and confirm?

@internetammo, Are you still encountering the issue? If so, Would you be able to provide us with screenshots of the request and response, please?

AnilMaktala commented 1 year ago

Hi @majirosstefan, I am checking to see if you are still experiencing this issue? If not, please let us know we will close this bug.

majirosstefan commented 1 year ago

Hello Anil, I am sorry, but currently I have no bandwidth to upgrade the app, or to provide more information.

I do not know if this can be closed - based on my experience bugs in the software will not be resolved on their own (but if it's your policy, feel free to close this issue, even if it was not resolved or triaged properly).

dral3x commented 1 year ago

I got the same issue while upgrading some OpenSearch clusters in different environments. When the version is "OpenSearch 2.5 (latest)", wildcard does NOT work. When the version is "Elasticsearch 7.10", wildcard does work.

Any suggestion on how to revert clusters upgraded to the nonworking version is appreciated.

Adding more details

I'm using

AnilMaktala commented 1 year ago

Hi @dral3x, Kindly follow the instructions provided here to revert back to the previous version of Elasticsearch.

dral3x commented 1 year ago

That works. Thanks, @AnilMaktala

majirosstefan commented 1 year ago

Just small update from me:

I created new project (independent of the one that has issues above) with amplify-cli (11.0.3), and "aws-amplify": "^5.3.8" and it seems to work. Elasticsearch version is 7.10 ( In the AWS Console).