[BUG] Cannot add GSI sortkey column to multiple indexes

[hellohellosharp]

Summary:

My table has two GSI's: userId-unix and date-unix. There is no way to decorate the unix column to allow both secondary indexes.

Code sample:

@attribute.gsi.sortKey.number({indexName: 'date-unix'})
@attribute.gsi.sortKey.number({indexName: 'userId-unix'})
unix: number;

DynamodeStorageError: Attribute "unix" was already decorated in entity "TokenBalanceChangeTable"

Expected result:

Should decorate the column with both indexes

Actual result:

Throws error: DynamodeStorageError: Attribute "unix" was already decorated in entity "TokenBalanceChangeTable"

Environment:

Windows Node v20

Other:

• [X ] I have read through the Dynamode documentation before posting this issue
• [X ] I have searched through the GitHub issues (including closed issues) and pull requests to ensure this issue has not already been raised before
• [X ] I have searched the internet to ensure this issue hasn't been raised or answered before

[blazejkustra]

Hey @hellohellosharp, thanks for posting this issue! It is indeed not possible to have an attribute with multiple indexes in dynamode – which is a bummer... I went ahead and adjusted the code, turns out it required a lot of adjustments as logic in some places relied on the fact that an attribute had only one index.

Try installing the release candidate and let me know if it works well for you 🚀

npm i dynamode@1.3.0-rc.0

[blazejkustra]

Also, taking this opportunity, how do you like dynamode so far? @hellohellosharp

[hellohellosharp]

Thanks for the fast reply!

I tried installing your release-candidate but it caused a bunch of errors in my typescript project.

First it made me change all my imports from dynamode to dynamode/lib.

For example: import TableManager from 'dynamode/lib/table';

But now its throwing tons of typing errors. About ~50 total. Examples:

node_modules/dynamode/lib/table/helpers/indexes.ts:12:29 - error TS18046: 'index' is of type 'unknown'.

12     .filter(([, index]) => !index.partitionKey && index.sortKey)
                               ~~~~~

node_modules/dynamode/lib/table/helpers/indexes.ts:12:51 - error TS18046: 'index' is of type 'unknown'.

12     .filter(([, index]) => !index.partitionKey && index.sortKey)
                                                     ~~~~~

node_modules/dynamode/lib/table/helpers/indexes.ts:15:45 - error TS18046: 'index' is of type 'unknown'.

15       KeySchema: getKeySchema(partitionKey, index.sortKey),
                                               ~~~~~

node_modules/dynamode/lib/table/helpers/indexes.ts:26:30 - error TS18046: 'index' is of type 'unknown'.

26     .filter(([, index]) => !!index.partitionKey)
                                ~~~~~

node_modules/dynamode/lib/table/helpers/indexes.ts:30:31 - error TS18046: 'index' is of type 'unknown'.

30       KeySchema: getKeySchema(index.partitionKey!, index.sortKey),
                                 ~~~~~

node_modules/dynamode/lib/table/helpers/indexes.ts:30:52 - error TS18046: 'index' is of type 'unknown'.

30       KeySchema: getKeySchema(index.partitionKey!, index.sortKey),
                                                      ~~~~~

node_modules/dynamode/lib/table/index.ts:11:22 - error TS2307: Cannot find module '@lib/dynamode/index' or its corresponding type declarations.

11 import Dynamode from '@lib/dynamode/index';
                        ~~~~~~~~~~~~~~~~~~~~~

node_modules/dynamode/lib/table/index.ts:12:20 - error TS2307: Cannot find module '@lib/entity' or its corresponding type declarations.

12 import Entity from '@lib/entity';
                      ~~~~~~~~~~~~~

node_modules/dynamode/lib/table/index.ts:13:27 - error TS2307: Cannot find module '@lib/entity/entityManager' or its corresponding type declarations.

13 import EntityManager from '@lib/entity/entityManager';
                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/dynamode/lib/table/index.ts:14:52 - error TS2307: Cannot find module '@lib/table/helpers/builders' or its corresponding type declarations.

14 import { buildIndexCreate, buildIndexDelete } from '@lib/table/helpers/builders';
                                                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/dynamode/lib/table/index.ts:15:36 - error TS2307: Cannot find module '@lib/table/helpers/converters' or its corresponding type declarations.

15 import { convertToTableData } from '@lib/table/helpers/converters';
                                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/dynamode/lib/table/index.ts:16:46 - error TS2307: Cannot find module '@lib/table/helpers/definitions' or its corresponding type declarations.

16 import { getTableAttributeDefinitions } from '@lib/table/helpers/definitions';
                                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/dynamode/lib/table/index.ts:17:30 - error TS2307: Cannot find module '@lib/table/helpers/schema' or its corresponding type declarations.

17 import { getKeySchema } from '@lib/table/helpers/schema';
                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/dynamode/lib/table/index.ts:18:31 - error TS2307: Cannot find module '@lib/table/helpers/validator' or its corresponding type declarations.

18 import { validateTable } from '@lib/table/helpers/validator';
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/dynamode/lib/table/index.ts:28:8 - error TS2307: Cannot find module '@lib/table/types' or its corresponding type declarations.

28 } from '@lib/table/types';
          ~~~~~~~~~~~~~~~~~~

node_modules/dynamode/lib/table/index.ts:29:58 - error TS2307: Cannot find module '@lib/utils' or its corresponding type declarations.

29 import { isNotEmptyArray, Narrow, ValidationError } from '@lib/utils';
                                                            ~~~~~~~~~~~~

node_modules/dynamode/lib/transactionWrite/index.ts:2:22 - error TS2307: Cannot find module '@lib/dynamode/index' or its corresponding type declarations.

2 import Dynamode from '@lib/dynamode/index';
                       ~~~~~~~~~~~~~~~~~~~~~

node_modules/dynamode/lib/transactionWrite/index.ts:3:20 - error TS2307: Cannot find module '@lib/entity' or its corresponding type declarations.

3 import Entity from '@lib/entity';
                     ~~~~~~~~~~~~~

Let me know if I did something wrong.

Your package is fantastic. I've worked with many dynamodb projects and honestly I hate working with it directly. With your wrapper my code is so much cleaner and easier to follow. 5 stars from me!

[hellohellosharp]

Here's my tsconfig:

{
  "include": [
    "src/"
  ],
  "exclude": [
    "tests/*"
  ],
  "compilerOptions": {
    /* Visit https://aka.ms/tsconfig to read more about this file */

    /* Projects */
    // "incremental": true,                              /* Save .tsbuildinfo files to allow for incremental compilation of projects. */
    // "composite": true,                                /* Enable constraints that allow a TypeScript project to be used with project references. */
    // "tsBuildInfoFile": "./.tsbuildinfo",              /* Specify the path to .tsbuildinfo incremental compilation file. */
    // "disableSourceOfProjectReferenceRedirect": true,  /* Disable preferring source files instead of declaration files when referencing composite projects. */
    // "disableSolutionSearching": true,                 /* Opt a project out of multi-project reference checking when editing. */
    // "disableReferencedProjectLoad": true,             /* Reduce the number of projects loaded automatically by TypeScript. */

    /* Language and Environment */
    "target": "es2016",
    /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
    // "lib": [],                                        /* Specify a set of bundled library declaration files that describe the target runtime environment. */
    // "jsx": "preserve",                                /* Specify what JSX code is generated. */
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    /* Enable experimental support for legacy experimental decorators. */
    // "emitDecoratorMetadata": true,                    /* Emit design-type metadata for decorated declarations in source files. */
    // "jsxFactory": "",                                 /* Specify the JSX factory function used when targeting React JSX emit, e.g. 'React.createElement' or 'h'. */
    // "jsxFragmentFactory": "",                         /* Specify the JSX Fragment reference used for fragments when targeting React JSX emit e.g. 'React.Fragment' or 'Fragment'. */
    // "jsxImportSource": "",                            /* Specify module specifier used to import the JSX factory functions when using 'jsx: react-jsx*'. */
    // "reactNamespace": "",                             /* Specify the object invoked for 'createElement'. This only applies when targeting 'react' JSX emit. */
    // "noLib": true,                                    /* Disable including any library files, including the default lib.d.ts. */
    // "useDefineForClassFields": true,                  /* Emit ECMAScript-standard-compliant class fields. */
    // "moduleDetection": "auto",                        /* Control what method is used to detect module-format JS files. */

    /* Modules */
    "module": "commonjs",
    /* Specify what module code is generated. */
    // "rootDir": "./",                                  /* Specify the root folder within your source files. */
    // "moduleResolution": "node10",                     /* Specify how TypeScript looks up a file from a given module specifier. */
    // "baseUrl": "./",                                  /* Specify the base directory to resolve non-relative module names. */
    // "paths": {},                                      /* Specify a set of entries that re-map imports to additional lookup locations. */
    // "rootDirs": [],                                   /* Allow multiple folders to be treated as one when resolving modules. */
    // "typeRoots": [],                                  /* Specify multiple folders that act like './node_modules/@types'. */
    // "types": [],                                      /* Specify type package names to be included without being referenced in a source file. */
    // "allowUmdGlobalAccess": true,                     /* Allow accessing UMD globals from modules. */
    // "moduleSuffixes": [],                             /* List of file name suffixes to search when resolving a module. */
    // "allowImportingTsExtensions": true,               /* Allow imports to include TypeScript file extensions. Requires '--moduleResolution bundler' and either '--noEmit' or '--emitDeclarationOnly' to be set. */
    // "resolvePackageJsonExports": true,                /* Use the package.json 'exports' field when resolving package imports. */
    // "resolvePackageJsonImports": true,                /* Use the package.json 'imports' field when resolving imports. */
    // "customConditions": [],                           /* Conditions to set in addition to the resolver-specific defaults when resolving imports. */
    // "resolveJsonModule": true,                        /* Enable importing .json files. */
    // "allowArbitraryExtensions": true,                 /* Enable importing files with any extension, provided a declaration file is present. */
    // "noResolve": true,                                /* Disallow 'import's, 'require's or '<reference>'s from expanding the number of files TypeScript should add to a project. */

    /* JavaScript Support */
    // "allowJs": true,                                  /* Allow JavaScript files to be a part of your program. Use the 'checkJS' option to get errors from these files. */
    // "checkJs": true,                                  /* Enable error reporting in type-checked JavaScript files. */
    // "maxNodeModuleJsDepth": 1,                        /* Specify the maximum folder depth used for checking JavaScript files from 'node_modules'. Only applicable with 'allowJs'. */

    /* Emit */
    // "declaration": true,                              /* Generate .d.ts files from TypeScript and JavaScript files in your project. */
    // "declarationMap": true,                           /* Create sourcemaps for d.ts files. */
    // "emitDeclarationOnly": true,                      /* Only output d.ts files and not JavaScript files. */
    // "sourceMap": true,                                /* Create source map files for emitted JavaScript files. */
    // "inlineSourceMap": true,                          /* Include sourcemap files inside the emitted JavaScript. */
    // "outFile": "./",                                  /* Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output. */
    "outDir": "./dist",
    /* Specify an output folder for all emitted files. */
    // "removeComments": true,                           /* Disable emitting comments. */
    // "noEmit": true,                                   /* Disable emitting files from a compilation. */
    // "importHelpers": true,                            /* Allow importing helper functions from tslib once per project, instead of including them per-file. */
    // "importsNotUsedAsValues": "remove",               /* Specify emit/checking behavior for imports that are only used for types. */
    // "downlevelIteration": true,                       /* Emit more compliant, but verbose and less performant JavaScript for iteration. */
    // "sourceRoot": "",                                 /* Specify the root path for debuggers to find the reference source code. */
    // "mapRoot": "",                                    /* Specify the location where debugger should locate map files instead of generated locations. */
    // "inlineSources": true,                            /* Include source code in the sourcemaps inside the emitted JavaScript. */
    // "emitBOM": true,                                  /* Emit a UTF-8 Byte Order Mark (BOM) in the beginning of output files. */
    // "newLine": "crlf",                                /* Set the newline character for emitting files. */
    // "stripInternal": true,                            /* Disable emitting declarations that have '@internal' in their JSDoc comments. */
    // "noEmitHelpers": true,                            /* Disable generating custom helper functions like '__extends' in compiled output. */
    // "noEmitOnError": true,                            /* Disable emitting files if any type checking errors are reported. */
    // "preserveConstEnums": true,                       /* Disable erasing 'const enum' declarations in generated code. */
    // "declarationDir": "./",                           /* Specify the output directory for generated declaration files. */
    // "preserveValueImports": true,                     /* Preserve unused imported values in the JavaScript output that would otherwise be removed. */

    /* Interop Constraints */
    // "isolatedModules": true,                          /* Ensure that each file can be safely transpiled without relying on other imports. */
    // "verbatimModuleSyntax": true,                     /* Do not transform or elide any imports or exports not marked as type-only, ensuring they are written in the output file's format based on the 'module' setting. */
    // "allowSyntheticDefaultImports": true,             /* Allow 'import x from y' when a module doesn't have a default export. */
    "esModuleInterop": true,
    /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */
    // "preserveSymlinks": true,                         /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */
    "forceConsistentCasingInFileNames": true,
    /* Ensure that casing is correct in imports. */

    /* Type Checking */
    "strict": true,
    /* Enable all strict type-checking options. */
    // "noImplicitAny": true,                            /* Enable error reporting for expressions and declarations with an implied 'any' type. */
    // "strictNullChecks": true,                         /* When type checking, take into account 'null' and 'undefined'. */
    // "strictFunctionTypes": true,                      /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */
    // "strictBindCallApply": true,                      /* Check that the arguments for 'bind', 'call', and 'apply' methods match the original function. */
    // "strictPropertyInitialization": true,             /* Check for class properties that are declared but not set in the constructor. */
    // "noImplicitThis": true,                           /* Enable error reporting when 'this' is given the type 'any'. */
    // "useUnknownInCatchVariables": true,               /* Default catch clause variables as 'unknown' instead of 'any'. */
    // "alwaysStrict": true,                             /* Ensure 'use strict' is always emitted. */
    // "noUnusedLocals": true,                           /* Enable error reporting when local variables aren't read. */
    // "noUnusedParameters": true,                       /* Raise an error when a function parameter isn't read. */
    // "exactOptionalPropertyTypes": true,               /* Interpret optional property types as written, rather than adding 'undefined'. */
    // "noImplicitReturns": true,                        /* Enable error reporting for codepaths that do not explicitly return in a function. */
    // "noFallthroughCasesInSwitch": true,               /* Enable error reporting for fallthrough cases in switch statements. */
    // "noUncheckedIndexedAccess": true,                 /* Add 'undefined' to a type when accessed using an index. */
    // "noImplicitOverride": true,                       /* Ensure overriding members in derived classes are marked with an override modifier. */
    // "noPropertyAccessFromIndexSignature": true,       /* Enforces using indexed accessors for keys declared using an indexed type. */
    // "allowUnusedLabels": true,                        /* Disable error reporting for unused labels. */
    // "allowUnreachableCode": true,                     /* Disable error reporting for unreachable code. */

    /* Completeness */
    // "skipDefaultLibCheck": true,                      /* Skip type checking .d.ts files that are included with TypeScript. */
    "skipLibCheck": true,
    /* Skip type checking all .d.ts files. */
  }
}

[blazejkustra]

It was my error, try installing this version 😅

npm i dynamode@1.3.0-rc.1

[blazejkustra]

Your package is fantastic. I've worked with many dynamodb projects and honestly I hate working with it directly. With your wrapper my code is so much cleaner and easier to follow. 5 stars from me!

Thank you! Consider staring the repo if you haven't already. Also if you have any suggestions how to improve the library, happy to hear your suggestions 😄

[hellohellosharp]

It was my error, try installing this version 😅

npm i dynamode@1.3.0-rc.1

Seems to be working great! I haven't tried queries on it yet but the following compiled fine:

export default class TokenBalanceChangeTable extends Entity {
    @attribute.partitionKey.string()
    id: string;

    @attribute.gsi.partitionKey.string({indexName: 'date-unix'})
    date: string;

    @attribute.gsi.partitionKey.string({indexName: 'userId-unix'})
    userId: string;

    @attribute.gsi.sortKey.number({indexName: 'userId-unix',})
    @attribute.gsi.sortKey.number({indexName: 'date-unix',})
    unix: number;

...

export const TokenBalanceChangeTableManager = new TableManager(TokenBalanceChangeTable, {
    tableName: TABLE_NAME,
    partitionKey: 'id',
    createdAt: 'createdAt',
    updatedAt: 'updatedAt',
    indexes: {
        'userId-unix': {partitionKey: 'userId', sortKey: 'unix'},
        'date-unix': {partitionKey: 'date', sortKey: 'unix'}
    }
});

[hellohellosharp]

Your package is fantastic. I've worked with many dynamodb projects and honestly I hate working with it directly. With your wrapper my code is so much cleaner and easier to follow. 5 stars from me!

Thank you! Consider staring the repo if you haven't already. Also if you have any suggestions how to improve the library, happy to hear your suggestions 😄

Will do!

Only thing I was trying and having a bit of trouble with was extending the Entity class. Something like:

export class EntityExtended extends Entity {
  @attribute.date.string()
    createdAt: Date;

    @attribute.date.string()
    updatedAt: Date;

    constructor(props: any) {
        super(props);
        this.createdAt = props.createdAt || new Date();
        this.updatedAt = props.updatedAt || new Date();
    }

}

export class UserTable extends EntityExtended {
...

So I didn't have to define createdAt and updatedAt in every table.

Only problem with this is it caused the dynamodeEntity column to stop being set for some reason.

Not really a necessary feature though.

[blazejkustra]

I think you were doing something wrong, following code works well for me and both UserTable and User entities have dynamodeEntity property:

import attribute from '../dist/decorators';
import Entity from '../dist/entity';
import TableManager from '../dist/table';

export class EntityWithTimestamps extends Entity {
  @attribute.date.string()
  createdAt: Date;

  @attribute.date.string()
  updatedAt: Date;

  constructor(props: any) {
    super(props);
    this.createdAt = props.createdAt || new Date();
    this.updatedAt = props.updatedAt || new Date();
  }
}
export class UserTable extends EntityWithTimestamps {
  @attribute.partitionKey.string()
  partitionKey: string;

  @attribute.sortKey.string()
  sortKey: string;

  constructor(props: any) {
    super(props);

    this.partitionKey = props.partitionKey;
    this.sortKey = props.sortKey;
  }
}

export const UserTableManager = new TableManager(UserTable, {
  tableName: 'test-user-table',
  partitionKey: 'partitionKey',
  sortKey: 'sortKey',
  createdAt: 'createdAt',
  updatedAt: 'updatedAt',
});

export class User extends UserTable {
  @attribute.string()
  username: string;

  constructor(props: any) {
    super(props);
    this.username = props.username;
  }
}

const UserTableEntityManager = UserTableManager.entityManager();
const UserManager = UserTableManager.entityManager(User);

async function test() {
  const user = await UserManager.put(
    new User({
      partitionKey: 'user',
      sortKey: 'user#123',
      username: 'test',
    }),
  );
  console.log(user);

  const userTableEntity = await UserTableEntityManager.put(new UserTable({ partitionKey: 'user', sortKey: 'user' }));
  console.log(userTableEntity);
}

async function createTable() {
  const table = await UserTableManager.createTable();
  console.log(table);
}

createTable();
test();

Try running it locally with ts-node file.ts. Let me know if it works 😄

[blazejkustra]

I haven't tried queries on it yet but the following compiled fine:

Once you confirm that query works well for you, also let me know, thanks!

[hellohellosharp]

Hi sorry for the late reply. A query that used to work fine no longer works (it's on a different table).

const queryResult = await APIOutgoingJobManager.query()
                .partitionKey('date')
                .eq(dateString)
                .sort('descending')
                .sortKey('unix').lt(unixBefore)
                .limit(30)
                .run();

export default class APIOutgoingJobTable extends EntityExtended {
    @attribute.partitionKey.string()
    id: string;
    @attribute.gsi.partitionKey.string({indexName: 'date-unix'})
    date: string;
    @attribute.gsi.sortKey.number({indexName: 'date-unix'})
    unix: number;

...

Error:

 Error on APIOutgoingJobs ValidationException: Query key condition not supported
[1]     at throwDefaultError (C:\Repos\sam-ai-backend\ts-app\node_modules\@smithy\smithy-client\dist-cjs\index.js:838:20)
[1]     at C:\Repos\sam-ai-backend\ts-app\node_modules\@smithy\smithy-client\dist-cjs\index.js:847:5
[1]     at de_CommandError (C:\Repos\sam-ai-backend\ts-app\node_modules\@aws-sdk\client-dynamodb\dist-cjs\index.js:2231:14)

I have confirmed that I can run this query fine from the AWS console. It's just triggering a validation error for some reason in your package.

Again, this is a different table with only one GSI date-unix. The query used to run fine before the rc version.

[blazejkustra]

Following code works well for me, can you try running it?

import attribute from '../dist/decorators';
import Entity from '../dist/entity';
import TableManager from '../dist/table';

export class EntityWithTimestamps extends Entity {
  @attribute.date.string()
  createdAt: Date;

  @attribute.date.string()
  updatedAt: Date;

  constructor(props: any) {
    super(props);
    this.createdAt = props.createdAt || new Date();
    this.updatedAt = props.updatedAt || new Date();
  }
}
export class APIOutgoingJobTable extends EntityWithTimestamps {
  @attribute.partitionKey.string()
  id: string;

  @attribute.gsi.partitionKey.string({ indexName: 'date-unix' })
  date: string;

  @attribute.gsi.sortKey.number({ indexName: 'date-unix' })
  unix: number;

  constructor(props: { id: string; date: string; unix: number }) {
    super(props);

    this.id = props.id;
    this.date = props.date;
    this.unix = props.unix;
  }
}

export const Table = new TableManager(APIOutgoingJobTable, {
  tableName: 'test-table',
  partitionKey: 'id',
  createdAt: 'createdAt',
  updatedAt: 'updatedAt',
  indexes: {
    'date-unix': {
      partitionKey: 'date',
      sortKey: 'unix',
    },
  },
});

const APIOutgoingJobManager = Table.entityManager();

async function test() {
  const entity = await APIOutgoingJobManager.put(
    new APIOutgoingJobTable({
      id: 'user',
      date: '01.01.2021',
      unix: 999,
    }),
  );
  console.log(entity);

  const queryResult = await APIOutgoingJobManager.query()
    .partitionKey('date')
    .eq('01.01.2021')
    .sort('descending')
    .sortKey('unix')
    .lt(1000)
    .limit(30)
    .run();
  console.log(queryResult);
}

async function createTable() {
  const table = await Table.createTable();
  console.log(table);
}

// createTable();
test();

[blazejkustra]

Also try running the query with return: 'input' option:

  const queryResult = await APIOutgoingJobManager.query()
    .partitionKey('date')
    .eq('01.01.2021')
    .sort('descending')
    .sortKey('unix')
    .lt(1000)
    .limit(30)
    .run({ return: 'input' });

This is the input I got from the previous example:

{
  TableName: 'test-table',
  ScanIndexForward: false,
  Limit: 30,
  IndexName: 'date-unix',
  KeyConditionExpression: '#date = :date AND unix < :unix',
  FilterExpression: undefined,
  ExpressionAttributeNames: { '#date': 'date' },
  ExpressionAttributeValues: { ':date': { S: '01.01.2021' }, ':unix': { N: '1000' } }
}

[hellohellosharp]

My fault. I had defined unix attribute in both APIOutgoingJob and APIOutgoingJobTable which was causing the issue.

From what I can see, queries are working perfectly :)

[blazejkustra]

Awesome! I'll update the PR with unit/e2e tests and let you know once v1.3.0 is released. Thanks 😄

[blazejkustra]

@hellohellosharp v1.3.0 was just released, please check it out!