search

aws / aws-sdk-js-v3

Modularized AWS SDK for JavaScript.
Apache License 2.0
2.96k stars 557 forks source link

TransactWrite does not include Item on ConditionalCheckFailed #6237

Open dil-mvallone opened 4 days ago

dil-mvallone commented 4 days ago

Checkboxes for prior research

Describe the bug

I am trying to write a write transaction that has condition expressions. I am including ReturnValuesOnConditionCheckFailure: "ALL_OLD" in my request. The condition fails, but the response does not include the Item property. I am also using electrodb, but I grabbed the command and used the dynamo client directly to test without the library. The Item property is still missing.

I am also using the NodejsFunction CDK construct.

SDK version number

aws-sdk 3.552.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

Nodejs 20 Lambda runtime.

Reproduction Steps

Here is the dynamodb command and how I execute it:

const transactCommand: TransactWriteCommandInput = {
  TransactItems: [
    {
      Update: {
        UpdateExpression:
          'SET #deleteAt = :deleteAt_u0, #orgId = :orgId_u0, #userUid = :userUid_u0, #licenseTypeUid = :licenseTypeUid_u0, #__edb_e__ = :__edb_e___u0, #__edb_v__ = :__edb_v___u0 ADD #entityVersion :entityVersion_u0',
        ExpressionAttributeNames: {
          '#PK': 'PK',
          '#SK': 'SK',
          '#deleteAt': 'deleteAt',
          '#entityVersion': 'entityVersion',
          '#orgId': 'orgId',
          '#userUid': 'userUid',
          '#licenseTypeUid': 'licenseTypeUid',
          '#__edb_e__': '__edb_e__',
          '#__edb_v__': '__edb_v__',
        },
        ExpressionAttributeValues: {
          ':deleteAt_u0': 1753710692,
          ':entityVersion_u0': 1,
          ':orgId_u0': 3,
          ':userUid_u0': 'useruid',
          ':licenseTypeUid_u0': 'professional',
          ':__edb_e___u0': 'directAssignment',
          ':__edb_v___u0': '1',
        },
        TableName: 'LicenseManagementTable',
        Key: { PK: 'direct_assignments#3', SK: 'useruid#professional' },
        ReturnValuesOnConditionCheckFailure: 'ALL_OLD',
        ConditionExpression: 'attribute_exists(#PK) AND attribute_exists(#SK)',
      },
    },
    {
      Update: {
        UpdateExpression:
          'SET #deleteAt = :deleteAt_u0, #orgId = :orgId_u0, #userUid = :userUid_u0, #licenseTypeUid = :licenseTypeUid_u0, #__edb_e__ = :__edb_e___u0, #__edb_v__ = :__edb_v___u0 ADD #entityVersion :entityVersion_u0',
        ExpressionAttributeNames: {
          '#PK': 'PK',
          '#SK': 'SK',
          '#deleteAt': 'deleteAt',
          '#entityVersion': 'entityVersion',
          '#orgId': 'orgId',
          '#userUid': 'userUid',
          '#licenseTypeUid': 'licenseTypeUid',
          '#__edb_e__': '__edb_e__',
          '#__edb_v__': '__edb_v__',
        },
        ExpressionAttributeValues: {
          ':deleteAt_u0': 1753710692,
          ':entityVersion_u0': 1,
          ':orgId_u0': 3,
          ':userUid_u0': 'useruid',
          ':licenseTypeUid_u0': 'professional',
          ':__edb_e___u0': 'license',
          ':__edb_v___u0': '1',
        },
        TableName: 'LicenseManagementTable',
        Key: { PK: 'licenses#3', SK: 'useruid#professional' },
        ReturnValuesOnConditionCheckFailure: 'ALL_OLD',
        ConditionExpression: 'attribute_exists(#PK) AND attribute_exists(#SK)',
      },
    },
    {
      Update: {
        UpdateExpression:
          'SET #assignedCount = (if_not_exists(#assignedCount, :assignedCount_default_value_u0) - :assignedCount_u0), #orgId = :orgId_u0, #licenseTypeUid = :licenseTypeUid_u0, #__edb_e__ = :__edb_e___u0, #__edb_v__ = :__edb_v___u0',
        ExpressionAttributeNames: {
          '#PK': 'PK',
          '#SK': 'SK',
          '#assignedCount': 'assignedCount',
          '#orgId': 'orgId',
          '#licenseTypeUid': 'licenseTypeUid',
          '#__edb_e__': '__edb_e__',
          '#__edb_v__': '__edb_v__',
        },
        ExpressionAttributeValues: {
          ':assignedCount_u0': 1,
          ':assignedCount_default_value_u0': 0,
          ':orgId_u0': 3,
          ':licenseTypeUid_u0': 'professional',
          ':__edb_e___u0': 'orgLicenses',
          ':__edb_v___u0': '1',
        },
        TableName: 'LicenseManagementTable',
        Key: { PK: 'org_licenses#3', SK: 'professional' },
        ReturnValuesOnConditionCheckFailure: 'ALL_OLD',
        ConditionExpression: 'attribute_exists(#PK) AND attribute_exists(#SK)',
      },
    },
  ],
};

    import { DynamoDBDocument, TransactWriteCommand, TransactWriteCommandInput } from '@aws-sdk/lib-dynamodb';
    import { DynamoDB } from '@aws-sdk/client-dynamodb';

    const client = DynamoDBDocument.from(new DynamoDB({ endpoint: process.env.DYNAMO_ENDPOINT }));
    const command: TransactWriteCommand = new TransactWriteCommand(transactCommand);
    const update = await client.send(command).catch((e) => {
      console.log(JSON.stringify(e));
    });

Observed Behavior

The error I get is the following:

2024-06-28T20:10:22.583Z e6d0e31f-0634-4c36-b7b2-c5d9dbdca4c9 INFO TransactionCanceledException: Transaction cancelled, please refer cancellation reasons for specific reasons [None, ConditionalCheckFailed, None] at de_TransactionCanceledExceptionRes (/var/runtime/node_modules/@aws-sdk/client-dynamodb/dist-cjs/index.js:2511:21) at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-dynamodb/dist-cjs/index.js:2198:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20 at async /var/runtime/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/index.js:163:30 at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18 at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38 at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22 at deleteUser (/src/actions/license-management/delete-user/delete-user.ts:35:20) at Runtime.handler (/src/handlers/test-version/test-version.ts:12:3) { '$fault': 'client', '$metadata': { httpStatusCode: 400, requestId: 'ITGRHBDO9JN6SHNSP3ICSB4U4NVV4KQNSO5AEMVJF66Q9ASUAAJG', extendedRequestId: undefined, cfId: undefined, attempts: 1, totalRetryDelay: 0 }, CancellationReasons: [ { Code: 'None' }, { Code: 'ConditionalCheckFailed', Message: 'The conditional request failed' }, { Code: 'None' } ], __type: 'com.amazonaws.dynamodb.v20120810#TransactionCanceledException' }

The cancellationReasons correctly marks the 2nd Update as conditionalCheckFailed, but the Item property is missing.

Expected Behavior

The Item property should be present.

Possible Solution

No response

Additional Information/Context

No response

RanVaknin commented 4 days ago

Hi @dil-mvallone ,

I've seen this issue manifest in lambda a few times before, mainly because the Lambda provided SDK is provided at an older version that doesn't have this feature. Are you positive the SDK version you are using is in fact 3.552.0?

Can you please double check by following this https://docs.aws.amazon.com/lambda/latest/dg/lambda-nodejs.html#nodejs-sdk-included

Another way to rule out any SDK specific behavior is to log the raw request as it is returned from the dynamodb server itself (before it gets deserialized) :

client.middlewareStack.add(next => async (args) => {
 console.log(args.request)
 const response = await next(args);
 console.log(response);
 return response;
}, {step: 'finalizeRequest'})

Can you please add this snippet and see if the response contains the item that failed the check condition?

Thanks, Ran~

dil-mvallone commented 1 day ago

Hi @RanVaknin, thanks for you reply. Yes I grabbed the version by adding that code snippet found in the aws docs you linked. Also this behavior is replicated locally running unit tests and local dynamodb (v2.5.2 - aws sdk v3.592.0).

Here is the output of the middlewareStack snippet:

HttpRequest { method: 'POST', hostname: 'dynamodb.us-west-2.amazonaws.com', port: undefined, query: {}, headers: { 'content-type': 'application/x-amz-json-1.0', 'x-amz-target': 'DynamoDB_20120810.TransactWriteItems', 'content-length': '2915', host: 'dynamodb.us-west-2.amazonaws.com', 'X-Amzn-Trace-Id': 'Root=1-6682add3-10a5fe____3c96d8bc;Parent=11754f580babd82f;Sampled=1;Lineage=b08cd040:0', 'x-amz-user-agent': 'aws-sdk-js/3.552.0', 'user-agent': 'aws-sdk-js/3.552.0 ua/2.0 os/linux#5.10.216-225.855.amzn2.aarch64 lang/js md/nodejs#20.14.0 api/dynamodb#3.552.0 exec-env/AWS_Lambda_nodejs20.x', 'amz-sdk-invocation-id': '9f37a3b7-ebcd-4964-938f-fd422a570acb', 'amz-sdk-request': 'attempt=1; max=3', 'x-amz-date': '20240701T132335Z', 'x-amz-security-token': 'IQ_K___=', 'x-amz-content-sha256': 'fec6__2589d014a466867fcf5', authorization: 'AWS4-HMAC-SHA256 Credential=ASIARPXGKZ6D2IXW3K6S/20240701/us-west-2/dynamodb/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-target;x-amz-user-agent, Signature=bed8a3a7d5c24276a2cf58c4443070548c9b14471520b55a6128f253c3a309b8' }, body: '{"ClientRequestToken":"2436c395-ac66-428a-92d5-f9bab0ee1af7","TransactItems":[{"Update":{"ConditionExpression":"attribute_exists(#PK) AND attribute_exists(#SK)","ExpressionAttributeNames":{"#PK":"PK","#SK":"SK","#deleteAt":"deleteAt","#entityVersion":"entityVersion","#orgId":"orgId","#userUid":"userUid","#licenseTypeUid":"licenseTypeUid","#edb_e":"edb_e","#edb_v":"edb_v"},"ExpressionAttributeValues":{":deleteAt_u0":{"N":"1753710692"},":entityVersion_u0":{"N":"2"},":orgId_u0":{"N":"3"},":userUid_u0":{"S":"useruid"},":licenseTypeUid_u0":{"S":"professional"},":edbeu0":{"S":"directAssignment"},":edbvu0":{"S":"1"}},"Key":{"PK":{"S":"direct_assignments#3"},"SK":{"S":"useruid#professional"}},"ReturnValuesOnConditionCheckFailure":"ALL_OLD","TableName":"LicenseManagementTable","UpdateExpression":"SET #deleteAt = :deleteAt_u0, #orgId = :orgId_u0, #userUid = :userUid_u0, #licenseTypeUid = :licenseTypeUid_u0, #edb_e = :edb_e_u0, #edb_v = :edbvu0 ADD #entityVersion :entityVersion_u0"}},{"Update":{"ConditionExpression":"attribute_exists(#PK) AND attribute_exists(#SK)","ExpressionAttributeNames":{"#PK":"PK","#SK":"SK","#deleteAt":"deleteAt","#entityVersion":"entityVersion","#orgId":"orgId","#userUid":"userUid","#licenseTypeUid":"licenseTypeUid","#edb_e":"edb_e","#edb_v":"edb_v"},"ExpressionAttributeValues":{":deleteAt_u0":{"N":"1753710692"},":entityVersion_u0":{"N":"2"},":orgId_u0":{"N":"3"},":userUid_u0":{"S":"useruid"},":licenseTypeUid_u0":{"S":"professional"},":edbeu0":{"S":"license"},":edbvu0":{"S":"1"}},"Key":{"PK":{"S":"licenses#3"},"SK":{"S":"useruid#professional"}},"ReturnValuesOnConditionCheckFailure":"ALL_OLD","TableName":"LicenseManagementTable","UpdateExpression":"SET #deleteAt = :deleteAt_u0, #orgId = :orgId_u0, #userUid = :userUid_u0, #licenseTypeUid = :licenseTypeUid_u0, #edb_e = :edb_e_u0, #edb_v = :edbvu0 ADD #entityVersion :entityVersion_u0"}},{"Update":{"ConditionExpression":"attribute_exists(#PK) AND attribute_exists(#SK)","ExpressionAttributeNames":{"#PK":"PK","#SK":"SK","#assignedCount":"assignedCount","#orgId":"orgId","#licenseTypeUid":"licenseTypeUid","#edb_e":"edb_e","#edb_v":"edb_v"},"ExpressionAttributeValues":{":assignedCount_u0":{"N":"1"},":assignedCount_default_value_u0":{"N":"0"},":orgId_u0":{"N":"3"},":licenseTypeUid_u0":{"S":"professional"},":edbeu0":{"S":"orgLicenses"},":edbvu0":{"S":"1"}},"Key":{"PK":{"S":"org_licenses#3"},"SK":{"S":"professional"}},"ReturnValuesOnConditionCheckFailure":"ALL_OLD","TableName":"LicenseManagementTable","UpdateExpression":"SET #assignedCount = (if_not_exists(#assignedCount, :assignedCount_default_value_u0) - :assignedCount_u0), #orgId = :orgId_u0, #licenseTypeUid = :licenseTypeUid_u0, #edb_e = :edb_e_u0, #edb_v = :edbvu0"}}]}', protocol: 'https:', path: '/', username: undefined, password: undefined, fragment: undefined }

Looks like sdk version is in fact 3.552.0.

The 2nd console.log(response) is not logged because a TransactionCanceledException is thrown. The output is the same as the my first post.

RanVaknin commented 1 day ago

HI @dil-mvallone ,

Thanks for providing the info. From the raw response I can see that the dynamodb server itself did not respond with the failed item. This rules out an SDK issue and suggests two potential explanations: Either the item data in your DynamoDB table does not meet the conditions required to trigger this exception (we would need to review the specific item attributes to assess this properly), or there may be a service-side limitation affecting the operation.

Based on this blogpost from Dynamodb this feature is supported only for single item operations, but you are using TransactWriteItems which does not seem to support this feature.

By incorporating the ReturnValuesOnConditionCheckFailure parameter, you can reduce additional read operations and simplify error handling. You can now retrieve detailed information directly from the server side when a ConditionalCheckFailedException occurs, providing you with increased efficiency and improved decision-making. To get started, add the new parameter to your PutItem, UpdateItem, or DeleteItem operations and set the value to ALL_OLD. You can use your favorite coding language in our getting started guide.

Can you try to make a single PutItem and see if indeed the dynamodb server returns the desired failed item?

Thanks, Ran~

dil-mvallone commented 1 day ago

Is it really not supported for TransactWriteItems? This example from aws docs (in Java) it makes use of it inside a transaction. It sets the flag, but the example does not retrieve the item on the exception thought.

It is also part of the Java SDK.

With a single PutItem I am able to retrieve the Item upon ConditionCheckFailure.

Is there some docs that clearly mentions this feature is not implemented in TransactWriteItems ? I've been searching but was unable to get a definite answer.

Thanks!

RanVaknin commented 3 hours ago

Hey @dil-mvallone ,

Thanks for doing some more investigation. I did my own reproduction and found that TransactWriteItems can and does indeed return the desired ConditionCheckFailure and I'm able to see the items in the error response:

import { DynamoDBClient, TransactWriteItemsCommand } from "@aws-sdk/client-dynamodb";

const client = new DynamoDBClient({ region: "us-east-1" });

const params = {
    TransactItems: [
        {
            Put: {
                TableName: "foo-bar-table",
                Item: {
                    PK: { S: "testPK2" },
                    SK: { S: "testSK2" },
                    Attribute: { S: "value2" }
                },
                ConditionExpression: "attribute_not_exists(PK) AND attribute_not_exists(SK)",
                ReturnValuesOnConditionCheckFailure: "ALL_OLD"
            }
        },
        {
            Put: {
                TableName: "foo-bar-table",
                Item: {
                    PK: { S: "testPK3" },
                    SK: { S: "testSK3" }, 
                    Attribute: { S: "value3" }
                },
                ConditionExpression: "attribute_not_exists(PK) AND attribute_not_exists(SK)",
                ReturnValuesOnConditionCheckFailure: "ALL_OLD"
            }
        }
    ]
};
try {
    await client.send(new TransactWriteItemsCommand(params));
    console.log("success");
} catch (error) {
    console.error(JSON.stringify(error,null, 2));
}

Prints:

$ node sample.mjs
{
  "name": "TransactionCanceledException",
  "$fault": "client",
  "$metadata": {
    "httpStatusCode": 400,
    "requestId": "REDACTED",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "CancellationReasons": [
    {
      "Code": "ConditionalCheckFailed",
      "Item": {
        "Attribute": {
          "S": "value2"
        },
        "PK": {
          "S": "testPK2"
        },
        "SK": {
          "S": "testSK2"
        }
      },
      "Message": "The conditional request failed"
    },
    {
      "Code": "ConditionalCheckFailed",
      "Item": {
        "Attribute": {
          "S": "value3"
        },
        "PK": {
          "S": "testPK3"
        },
        "SK": {
          "S": "testSK3"
        }
      },
      "Message": "The conditional request failed"
    }
  ],
  "__type": "com.amazonaws.dynamodb.v20120810#TransactionCanceledException",
  "message": "Transaction cancelled, please refer cancellation reasons for specific reasons [ConditionalCheckFailed, ConditionalCheckFailed]"
}

Looking at your initial error stack trace, the individual cancellationReasons is { Code: 'None' } which means the error you are trying to get back is not ConditionalCheckFailed and therefore would not contain the item in the error response. This is interesting because you said when using only a single PutItem this did work.

With a single PutItem I am able to retrieve the Item upon ConditionCheckFailure.

Are you using the same exact same item structure here as in the TransactWriteItems? Can you provide us with some more details on the two invocations? Perhaps a side-by-side snippets of both? This might be some server side issue. But without knowing you table structure and attributes it would be quite difficult to determine definitively.

Thanks again! Ran~