DataStore sync fail on startup when model has custom primary key

ritxweb commented 1 year ago

Before opening, please confirm:

[X] I have searched for duplicate or closed issues and discussions.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

Angular

Amplify APIs

GraphQL API, DataStore

Amplify Categories

api

Environment information

``` # Put output below this line System: OS: Windows 10 10.0.19044 CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz Memory: 2.76 GB / 15.92 GB Binaries: Node: 16.16.0 - C:\Program Files\nodejs\node.EXE npm: 8.18.0 - C:\Program Files\nodejs\npm.CMD Browsers: Chrome: 105.0.5195.127 Edge: Spartan (44.19041.1266.0), Chromium (105.0.1343.53) Internet Explorer: 11.0.19041.1566 npmPackages: @angular-devkit/build-angular: ~13.0.1 => 13.0.2 @angular/animations: ~13.0.0 => 13.0.1 @angular/animations/browser: undefined () @angular/animations/browser/testing: undefined () @angular/cdk: ^13.0.1 => 13.0.1 @angular/cdk/a11y: undefined () @angular/cdk/accordion: undefined () @angular/cdk/bidi: undefined () @angular/cdk/clipboard: undefined () @angular/cdk/coercion: undefined () @angular/cdk/collections: undefined () @angular/cdk/drag-drop: undefined () @angular/cdk/keycodes: undefined () @angular/cdk/layout: undefined () @angular/cdk/observers: undefined () @angular/cdk/overlay: undefined () @angular/cdk/platform: undefined () @angular/cdk/portal: undefined () @angular/cdk/scrolling: undefined () @angular/cdk/stepper: undefined () @angular/cdk/table: undefined () @angular/cdk/testing: undefined () @angular/cdk/testing/protractor: undefined () @angular/cdk/testing/selenium-webdriver: undefined () @angular/cdk/testing/testbed: undefined () @angular/cdk/text-field: undefined () @angular/cdk/tree: undefined () @angular/cli: ~13.0.1 => 13.0.2 @angular/common: ~13.0.0 => 13.0.1 @angular/common/http: undefined () @angular/common/http/testing: undefined () @angular/common/testing: undefined () @angular/common/upgrade: undefined () @angular/compiler: ~13.0.0 => 13.0.1 @angular/compiler-cli: ~13.0.0 => 13.0.1 @angular/compiler/testing: undefined () @angular/core: ~13.0.0 => 13.0.1 @angular/core/testing: undefined () @angular/forms: ~13.0.0 => 13.0.1 @angular/localize: ^13.0.1 => 13.0.1 @angular/localize/init: undefined () @angular/material: ^13.0.1 => 13.0.1 @angular/material/autocomplete: undefined () @angular/material/autocomplete/testing: undefined () @angular/material/badge: undefined () @angular/material/badge/testing: undefined () @angular/material/bottom-sheet: undefined () @angular/material/bottom-sheet/testing: undefined () @angular/material/button: undefined () @angular/material/button-toggle: undefined () @angular/material/button-toggle/testing: undefined () @angular/material/button/testing: undefined () @angular/material/card: undefined () @angular/material/card/testing: undefined () @angular/material/checkbox: undefined () @angular/material/checkbox/testing: undefined () @angular/material/chips: undefined () @angular/material/chips/testing: undefined () @angular/material/core: undefined () @angular/material/core/testing: undefined () @angular/material/datepicker: undefined () @angular/material/datepicker/testing: undefined () @angular/material/dialog: undefined () @angular/material/dialog/testing: undefined () @angular/material/divider: undefined () @angular/material/divider/testing: undefined () @angular/material/expansion: undefined () @angular/material/expansion/testing: undefined () @angular/material/form-field: undefined () @angular/material/form-field/testing: undefined () @angular/material/form-field/testing/control: undefined () @angular/material/grid-list: undefined () @angular/material/grid-list/testing: undefined () @angular/material/icon: undefined () @angular/material/icon/testing: undefined () @angular/material/input: undefined () @angular/material/input/testing: undefined () @angular/material/list: undefined () @angular/material/list/testing: undefined () @angular/material/menu: undefined () @angular/material/menu/testing: undefined () @angular/material/paginator: undefined () @angular/material/paginator/testing: undefined () @angular/material/progress-bar: undefined () @angular/material/progress-bar/testing: undefined () @angular/material/progress-spinner: undefined () @angular/material/progress-spinner/testing: undefined () @angular/material/radio: undefined () @angular/material/radio/testing: undefined () @angular/material/select: undefined () @angular/material/select/testing: undefined () @angular/material/sidenav: undefined () @angular/material/sidenav/testing: undefined () @angular/material/slide-toggle: undefined () @angular/material/slide-toggle/testing: undefined () @angular/material/slider: undefined () @angular/material/slider/testing: undefined () @angular/material/snack-bar: undefined () @angular/material/snack-bar/testing: undefined () @angular/material/sort: undefined () @angular/material/sort/testing: undefined () @angular/material/stepper: undefined () @angular/material/stepper/testing: undefined () @angular/material/table: undefined () @angular/material/table/testing: undefined () @angular/material/tabs: undefined () @angular/material/tabs/testing: undefined () @angular/material/toolbar: undefined () @angular/material/toolbar/testing: undefined () @angular/material/tooltip: undefined () @angular/material/tooltip/testing: undefined () @angular/material/tree: undefined () @angular/material/tree/testing: undefined () @angular/platform-browser: ~13.0.0 => 13.0.1 @angular/platform-browser-dynamic: ~13.0.0 => 13.0.1 @angular/platform-browser-dynamic/testing: undefined () @angular/platform-browser/animations: undefined () @angular/platform-browser/testing: undefined () @angular/platform-server: ~13.0.0 => 13.0.1 @angular/platform-server/init: undefined () @angular/platform-server/testing: undefined () @angular/router: ~13.0.0 => 13.0.1 @angular/router/testing: undefined () @angular/router/upgrade: undefined () @nguniversal/builders: ^13.0.1 => 13.0.1 @nguniversal/express-engine: ^13.0.1 => 13.0.1 @nguniversal/express-engine/tokens: undefined () @types/express: ^4.17.0 => 4.17.13 @types/jasmine: ~3.10.0 => 3.10.2 @types/node: ^12.11.1 => 12.20.37 aws-amplify: ^4.3.36 => 4.3.36 express: ^4.15.2 => 4.17.1 ini: ^1.3.5 => 1.3.8 (2.0.0) inquirer: ^6.5.1 => 6.5.2 (8.2.0) jasmine-core: ~3.10.0 => 3.10.1 karma: ~6.3.0 => 6.3.9 karma-chrome-launcher: ~3.1.0 => 3.1.0 karma-coverage: ~2.0.3 => 2.0.3 karma-coverage-coffee-example: 1.0.0 karma-jasmine: ~4.0.0 => 4.0.1 karma-jasmine-html-reporter: ~1.7.0 => 1.7.0 rxjs: ~7.4.0 => 7.4.0 (6.6.7, 7.5.6, 5.5.12) rxjs/ajax: undefined () rxjs/fetch: undefined () rxjs/internal-compatibility: undefined () rxjs/operators: undefined () rxjs/testing: undefined () rxjs/webSocket: undefined () tslib: ^2.3.0 => 2.3.1 (1.14.1, 2.1.0) typescript: ~4.4.3 => 4.4.4 ulid: ^2.3.0 => 2.3.0 zone-mix: undefined () zone-node: undefined () zone-testing: undefined () zone.js: ~0.11.4 => 0.11.4 zone.js/async-test: undefined () zone.js/async-test.min: undefined () zone.js/fake-async-test: undefined () zone.js/fake-async-test.min: undefined () zone.js/jasmine-patch: undefined () zone.js/jasmine-patch.min: undefined () zone.js/long-stack-trace-zone: undefined () zone.js/long-stack-trace-zone.min: undefined () zone.js/mocha-patch: undefined () zone.js/mocha-patch.min: undefined () zone.js/proxy: undefined () zone.js/proxy.min: undefined () zone.js/sync-test: undefined () zone.js/sync-test.min: undefined () zone.js/task-tracking: undefined () zone.js/task-tracking.min: undefined () zone.js/webapis-media-query: undefined () zone.js/webapis-media-query.min: undefined () zone.js/webapis-notification: undefined () zone.js/webapis-notification.min: undefined () zone.js/webapis-rtc-peer-connection: undefined () zone.js/webapis-rtc-peer-connection.min: undefined () zone.js/webapis-shadydom: undefined () zone.js/webapis-shadydom.min: undefined () zone.js/wtf: undefined () zone.js/wtf.min: undefined () zone.js/zone-bluebird: undefined () zone.js/zone-bluebird.min: undefined () zone.js/zone-error: undefined () zone.js/zone-error.min: undefined () zone.js/zone-legacy: undefined () zone.js/zone-legacy.min: undefined () zone.js/zone-patch-canvas: undefined () zone.js/zone-patch-canvas.min: undefined () zone.js/zone-patch-cordova: undefined () zone.js/zone-patch-cordova.min: undefined () zone.js/zone-patch-electron: undefined () zone.js/zone-patch-electron.min: undefined () zone.js/zone-patch-fetch: undefined () zone.js/zone-patch-fetch.min: undefined () zone.js/zone-patch-jsonp: undefined () zone.js/zone-patch-jsonp.min: undefined () zone.js/zone-patch-message-port: undefined () zone.js/zone-patch-message-port.min: undefined () zone.js/zone-patch-promise-test: undefined () zone.js/zone-patch-promise-test.min: undefined () zone.js/zone-patch-resize-observer: undefined () zone.js/zone-patch-resize-observer.min: undefined () zone.js/zone-patch-rxjs: undefined () zone.js/zone-patch-rxjs-fake-async: undefined () zone.js/zone-patch-rxjs-fake-async.min: undefined () zone.js/zone-patch-rxjs.min: undefined () zone.js/zone-patch-socket-io: undefined () zone.js/zone-patch-socket-io.min: undefined () zone.js/zone-patch-user-media: undefined () zone.js/zone-patch-user-media.min: undefined () npmGlobalPackages: @angular-devkit/schematics-cli: 0.801.1 @angular/cli: 14.1.3 @aws-amplify/cli: 10.0.0 @ionic/cli: 6.6.0 angular: 1.7.8 ng2-custom-carousel: 0.0.30 npm: 8.18.0 ```

Describe the bug

System fails to sync on startup when creating a model providing a custom primary key composed by partition key and sort key. From this point none of the sync subscriptions works and updates DataStore. It looks like, on startup, system tries to initialize the subscriptions using the default "id" field as primary key but the model is using as primary key a composite key composed by the fields "pk" and "sk", as specified in the schema. It is weird because I can still see socket messages from the server of a new instance created in the backend but DataStore is not actualized with this new instances. The Api subscriptions also work and react to onCreate events.

Expected behavior

Model sync and reacting to new instances created on the backend.

Reproduction steps

PS C:> amplify add api
? Select from one of the below mentioned services: GraphQL ? Here is the GraphQL API that we will create. Select a setting to edit or continue Name: **** ? Provide API name: **** ? Here is the GraphQL API that we will create. Select a setting to edit or continue Authorization modes: API key (default, expiration time: 7 days from now) ? Choose the default authorization type for the API Amazon Cognito User Pool Use a Cognito user pool configured as a part of this project. ? Configure additional auth types? No ? Here is the GraphQL API that we will create. Select a setting to edit or continue Conflict detection (required for DataStore): Disabled ? Enable conflict detection? Yes ? Select the default resolution strategy Auto Merge ? Here is the GraphQL API that we will create. Select a setting to edit or continue Continue ? Choose a schema template: Single object with fields (e.g., “Todo” with ID, name, description)

Learn more about "@auth" authorization rules here: https://docs.amplify.aws/cli/graphql/authorization-rules ✅ GraphQL schema compiled successfully.

Edit your schema at ...\amplify\backend\api******\schema.graphql or place .graphql files in a directory at ...\amplify\backend\api******\schema √ Do you want to edit the schema now? (Y/n) · no ✅ Successfully added resource **** locally

✅ Some next steps: "amplify push" will build all your local backend resources and provision it in the cloud "amplify publish" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud

PS C:> amplify push
/ Fetching updates to backend environment: dev from the cloud.✅ GraphQL schema compiled successfully.

Edit your schema at ...\amplify\backend\api*\schema.graphql or place .graphql files in a directory at ...\amplify\backend\api\\schema √ Successfully pulled backend environment dev from the cloud. \ Building resource api/****✅ GraphQL schema compiled successfully.

Edit your schema at ...\amplify\backend\api****\schema.graphql or place .graphql files in a directory at ...\amplify\backend\api****\schema

Current Environment: dev

┌──────────┬──────────────────┬───────────┬───────────────────┐ │ Category │ Resource name │ Operation │ Provider plugin │ ├──────────┼──────────────────┼───────────┼───────────────────┤ │ Api │ *│ Create │ awscloudformation │ ├──────────┼──────────────────┼───────────┼───────────────────┤ │ Auth │ *** │ No Change │ awscloudformation │ ├──────────┼──────────────────┼───────────┼───────────────────┤ │ Storage │ ****│ No Change │ awscloudformation │ ├──────────┼──────────────────┼───────────┼───────────────────┤ │ Hosting │ amplifyhosting │ No Change │ │ └──────────┴──────────────────┴───────────┴───────────────────┘ ? Are you sure you want to continue? Yes ✅ GraphQL schema compiled successfully.

Edit your schema at ......\amplify\backend\api**\schema.graphql or place .graphql files in a directory at .....\amplify\backend\api***\schema

Building resource api/****✅ GraphQL schema compiled successfully.

Edit your schema at .....\amplify\backend\api***\schema.graphql or place .graphql files in a directory at ...\amplify\backend\api****\schema ? Do you want to generate code for your newly created GraphQL API Yes ? Choose the code generation language target angular ? Enter the file name pattern of graphql queries, mutations and subscriptions src\graphql*\.graphql ? Do you want to generate/update all possible GraphQL operations - queries, mutations and subscriptions Yes ? Enter maximum statement depth [increase from default if your schema is deeply nested] 2 ? Enter the file name for the generated code src\app\API.service.ts

Deploying resources into dev environment. This will take a few minutes. ⠹ Deployment completed. Deploying root stack ****[ ================================-------- ] 4/5 amplify-****-dev-65545 AWS::CloudFormation::Stack UPDATE_COMPLETE Wed Sep 28 2022 20:32:24…
storagel AWS::CloudFormation::Stack UPDATE_COMPLETE Wed Sep 28 2022 20:27:41…
auth AWS::CloudFormation::Stack UPDATE_COMPLETE Wed Sep 28 2022 20:27:41…
api* AWS::CloudFormation::Stack CREATE_COMPLETE Wed Sep 28 2022 20:32:20…
Deployed api ****[ ======================================== ] 9/9 GraphQLAPI AWS::AppSync::GraphQLApi CREATE_COMPLETE Wed Sep 28 2022 20:27:54…
GraphQLAPI**** AWS::AppSync::DataSource CREATE_COMPLETE Wed Sep 28 2022 20:27:58…
AmplifyDataStore***… AWS::IAM::Role CREATE_COMPLETE Wed Sep 28 2022 20:28:19…
DataStore AWS::DynamoDB::Table CREATE_COMPLETE Wed Sep 28 2022 20:28:14…
GraphQLAPITransformerSchema3C… AWS::AppSync::GraphQLSchema CREATE_COMPLETE Wed Sep 28 2022 20:28:59…
DynamoDBAccess**** AWS::IAM::Policy CREATE_COMPLETE Wed Sep 28 2022 20:31:21…

Edit your schema at ...\amplify\backend\api****\schema.graphql or place .graphql files in a directory at ...\amplify\backend\api****\schema Successfully generated models. Generated models can be found in ...\src √ Code generated successfully and saved in file src\app\API.service.ts √ Successfully generated models in the cloud.

GraphQL endpoint: https://************************************************************.amazonaws.com/graphql

GraphQL transformer version: 2

PS C:>

Code Snippet

// Put your code below this line.

import { Component, OnInit, OnDestroy } from '@angular/core';

import { DataStore } from 'aws-amplify';
import { Subscription } from 'rxjs';
import { Item } from 'src/models';

@Component({
  selector: 'app-user',
  templateUrl: './user.component.html',
  styleUrls: ['./user.component.sass']
})
export class UserComponent implements OnInit, OnDestroy {

  private newItemSubscription: Subscription | null = null;

  constructor() { }

  ngOnInit(): void {
    this.newItemSubscription = <Subscription>(
      DataStore.observeQuery(Item).subscribe((msg: any) => {
        console.log('msg Item:', msg);
      })
    );
  }

  ngOnDestroy(): void {
    if (this.newItemSubscription) {
      this.newItemSubscription.unsubscribe();
    }
    this.newItemSubscription = null;
  }

}

Log output

``` // Put your logs below this line [WARN] 54:58.801 DataStore - subscriptionError Connection failed: {"errors":[{"message":"Validation error of type FieldUndefined: Field 'id' in type 'Item' is undefined @ 'onCreateItem/id'"}]} [WARN] 54:58.803 DataStore { "recoverySuggestion": "Ensure app code is up to date, auth directives exist and are correct on each model, and that server-side data has not been invalidated by a schema change. If the problem persists, search for or create an issue: https://github.com/aws-amplify/amplify-js/issues", "localModel": null, "message": "Connection failed: {\"errors\":[{\"message\":\"Validation error of type FieldUndefined: Field 'id' in type 'Item' is undefined @ 'onCreateItem/id'\"}]}", "model": "Item", "operation": "Create", "errorType": "Unknown", "process": "subscribe", "remoteModel": null, "cause": { "provider": { "_config": { "aws_project_region": "eu-central-1", "aws_cognito_identity_pool_id": "**********************************************************", "aws_cognito_region": "eu-central-1", "aws_user_pools_id": "*************************", "aws_user_pools_web_client_id": "*********************************", "oauth": {}, "aws_cognito_username_attributes": [ "EMAIL" ], "aws_cognito_social_providers": [], "aws_cognito_signup_attributes": [ "EMAIL" ], "aws_cognito_mfa_configuration": "OFF", "aws_cognito_mfa_types": [ "SMS" ], "aws_cognito_password_protection_settings": { "passwordPolicyMinLength": 8, "passwordPolicyCharacters": [] }, "aws_cognito_verification_mechanisms": [ "EMAIL" ], "aws_user_files_s3_bucket": "***************************", "aws_user_files_s3_bucket_region": "eu-central-1", "aws_appsync_graphqlEndpoint": "https://********************************.appsync-api.eu-central-1.amazonaws.com/graphql", "aws_appsync_region": "eu-central-1", "aws_appsync_authenticationType": "AMAZON_COGNITO_USER_POOLS" }, "socketStatus": 0, "keepAliveTimeout": 300000, "subscriptionObserverMap": {}, "promiseArray": [], "connectionStateMonitor": { "_linkedConnectionState": { "networkState": "connected", "connectionState": "disconnected", "intendedConnectionState": "disconnected", "keepAliveState": "healthy" }, "_linkedConnectionStateObservable": {}, "_linkedConnectionStateObserver": { "_subscription": { "_observer": {}, "_state": "ready" } } }, "keepAliveTimeoutId": 129, "keepAliveAlertTimeoutId": 130 }, "error": { "errors": [ { "message": "Connection failed: {\"errors\":[{\"message\":\"Validation error of type FieldUndefined: Field 'id' in type 'Item' is undefined @ 'onCreateItem/id'\"}]}" } ] } } } ERROR Error: Uncaught (in promise): Object: {"data":null,"errors":[{"path":null,"locations":[{"line":9,"column":7,"sourceName":null}],"message":"Validation error of type FieldUndefined: Field 'id' in type 'Item' is undefined @ 'syncItems/items/id'"}]} ```

aws-exports.js

No response

Manual configuration

No response

Additional configuration


type Item @model @auth(rules: [{allow: private}]) {
  pk: ID! @primaryKey(sortKeyFields: ["sk"])
  sk: ID!
}

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

AnilMaktala commented 1 year ago

Hi @ritxweb 👋 thanks for raising this issue. Can you share your GraphQL schema, or at least the models you believe might be part of the issue?

ritxweb commented 1 year ago

Hi Anil, The graphql schema is the one described before:

type Item @model @auth(rules: [{allow: private}]) { pk: ID! @primaryKey(sortKeyFields: ["sk"]) sk: ID! }

My model now is different to bypass this issue so I don't have the completed one generated by Amplify after the transformer applies. Do you need this too?

chrisbonifacio commented 1 year ago

DataStore does not yet support custom primary and composite keys. This feature will be released in the near future and will potentially resolve this issue.

AnilMaktala commented 1 year ago

pull request changes has been merged and waiting for the release. Will close the ticket once released.

dpilch commented 1 year ago

Custom primary key support is now released in aws-amplify@4.3.38/@aws-amplify/datastore@3.13.0. Let us know if you have an issue after upgrading.

mjmostachetti commented 1 year ago

@dpilch I'm running into an issue where the 'ready' event never fires when we do the initial full sync.

DataStore - subscriptionError Connection failed: {"errors":[{"message":"Validation error of type FieldUndefined: Field 'id' in type 'OurModel' is undefined @ 'onDeleteOurModel/id'"}]}

I'm running "aws-amplify": "^5.0.4" and the model has Id: ID! @primaryKey

mjmostachetti commented 1 year ago

I have verified that when I revert the primary key back to the default id: ID!, the sync completes and the 'ready' event fires.

chrisbonifacio commented 1 year ago

@mjmostachetti

I have verified that when I revert the primary key back to the default id: ID!, the sync completes and the 'ready' event fires. What do you mean by "revert the primary key"? Changing the primary key on a model is a destructive action and will probably result in invalid data.

Are you using the latest version of the Amplify CLI? If not, please upgrade and regenerate your models with amplify codegen models.

Can you share your schema and give us reproduction steps like your schema before & after the primary key change so that we can reproduce your issue?

mjmostachetti commented 1 year ago

Yeah. I'm using the latest cli. Unfortunately I'm not going to share my schema. All our models are using the default id except for this one, so the repro steps would be create a few models with the default "id: ID!" and one model with "Id: ID! @PrimaryKey". Run an app w/ the hub events and see that you get the 'ready' event for datastore when you pull. When I reverted that one model back to "id: ID!" it worked again.

mjmostachetti commented 1 year ago

This is also in a fresh environment with no data - re: your comment about a destructive action.

dpilch commented 1 year ago

In amplify/cli.json is respectprimarykeyattributesonconnectionfield set to true?

mjmostachetti commented 1 year ago

At the moment respectprimarykeyattributesonconnectionfield is set to false - I'd assume it was also set to false previously. I don't recall setting this field or modifying it.

chrisbonifacio commented 1 year ago

At the moment respectprimarykeyattributesonconnectionfield is set to false - I'd assume it was also set to false previously. I don't recall setting this field or modifying it.

Flip it to true, regenerate your models, and let us know if the issue persists!

This is required for custom primary keys in DataStore to work

More info: https://docs.amplify.aws/cli/reference/feature-flags/#respectPrimaryKeyAttributesOnConnectionField

mjmostachetti commented 1 year ago

Good to know. Thanks for the info.

aws-amplify / amplify-js