Feature Request: GraphQL query with sorting by date with secondary index

MarlonJD commented 3 months ago

Description

Amplify supports query with secondary index, it's required custom document, but it's really important when pagination and blank results starting to be problem. Dynamodb scans by page by page, if first page don't have all results, we have to get next page, until there won't be any nextToken. It will take time any additional costs.

ie: You got 200 comments in Comments table, you'll want specific post's comments. Your GraphQL query limit is 100. You got 3 different post ids in 200 comments randomly. If you want first 100 comments for specific post, DynamoDB won't bring you to 100 comments, you'll probably get 20 first next page 40, next page 10, next page 30. It is even possible that the first two pages will be blank.

Let's look into possible solution.

I created secondary index according to gen 1 document, you can also create secondary index in gen 2, this won't change in gen 2. So if we'll create this new ModelQuery.listByIndex() it will work on gen 1 or gen 2 at the same time.

Let's look into normal query document in GraphQL, it can be created automatically in dart with ModelQueries.list( TaskComment.classType, where: TaskComment.TASK.eq(taskId), ); :

query listTaskComments($filter: ModelTaskCommentFilterInput, $limit: Int, $nextToken: String) {
  listTaskComments(filter: $filter, limit: $limit, nextToken: $nextToken) {
    items {
      id
      content
      post {
        id
        content
        ...
      }
      taskCommentsId
      createdAt
      updatedAt
    }
    nextToken
  }

now let's looking into taskCommentsByDate query document:

query taskCommentsByDate($filter: ModelTaskCommentFilterInput, $limit: Int, $nextToken: String, @queryField: ID) {
  taskCommentsByDate(filter: $filter, limit: $limit, nextToken: $nextToken, sortDirection: DESC, taskCommentsId: $queryField) {
    items {
      id
      content
      post {
        id
        content
        ...
      }
      taskCommentsId
      createdAt
      updatedAt
    }
    nextToken
  }

This can be create modified document, secondary indexes can be found easily on models/Comment.dart:

It will looks like this:

modelSchemaDefinition.indexes = [
  amplify_core.ModelIndex(fields: const ["taskCommentsId", "createdAt"], name: "taskCommentsByDate")
];

It's our custom document to query with secondaryIndex, my proposal is using this like:

ModelQueries.listByIndex(
  TaskComment.classType,
  queryField: taskId, <--- this will be required field
  sortDirection: DESC, <--- this will be optional, default is DESC
  customQueryName: "listTasksCommentsByDate" <--- It's optional. On Gen1 user provide query name, on gen 2 it's generating automatically.
  where: TaskComment.TASK.eq(taskId),
);

Edit: I added customQueryName for gen1 and gen2 support same time.

I'll try to create this, what's your opinion about this?

Steps to Reproduce

-

Screenshots

No response

Platforms

[X] iOS
[X] Android
[ ] Web
[ ] macOS
[ ] Windows
[ ] Linux

Flutter Version

3.22.0

Amplify Flutter Version

1.8.0

Deployment Method

Amplify CLI

Schema

type Post @model {
  id: ID!
  content: String!
  comments: [PostComment] @hasMany
}

type Comment @model {
  id: ID!
  createdAt: AWSDateTime!
  content: String!
  taskCommentsId: ID! @index(name: "taskCommentsByDate", queryField: "taskCommentsByDate", sortKeyFields: ["createdAt"])
  post: Post! @belongsTo(fields: ["taskCommentsId"])
}

Jordan-Nelson commented 3 months ago

Hello @MarlonJD - Thanks for taking the time to open the issue. I want to make sure I understand the proposal. It sounds like you have been able to achieve the desired behavior with custom GraphQL queries. Your proposal is for this to be supported out of the box without the use of customer queries. Is that correct?

MarlonJD commented 3 months ago

Hello @Jordan-Nelson. Yes custom GraphQL is possible and it's easier with copyWith after #4365. I want to use secondary index query without custom query and variable otherwise it will waste of time. It's all there we can easily create this document automatically,

Without this proposal, it can be query with these codes:

ie:

final request = ModelQueries.list(
  Comment.classType,
  // where: TaskComment.TASK.eq(taskId), <--- same field filter shouldn't be there
);

const modifiedDoc = r"""query taskCommentsByDate($filter: ModelTaskCommentFilterInput, $limit: Int, $nextToken: String, @queryField: ID!) {
  taskCommentsByDate(filter: $filter, limit: $limit, nextToken: $nextToken, sortDirection: DESC, taskCommentsId: $queryField) {
    items {
      id
      content
      post {
        id
        content
        ...
      }
      taskCommentsId
      createdAt
      updatedAt
    }
    nextToken
  }""";

// ignore: prefer_final_locals, omit_local_variable_types
var modifiedVar = request.variables;

modifiedVar["queryField"] = taskId;

final response = await Amplify.API
        .query(
          request: request.copyWith(
            document: modifiedDocument,
            variables: modifiedVar,
            decodePath: "taskCommentsByDate",
          ),
        )
        .response;

Instead of these modifications, we can just type:

ModelQueries.listByIndex(
  TaskComment.classType,
  queryField: taskId, <--- this will be required field
  sortDirection: DESC, <--- this will be optional, default is DESC
  // where: TaskComment.TASK.eq(taskId), <-- where shouldn't be same filter, but it can be other filters
);

It will save a lot of time.

Jordan-Nelson commented 3 months ago

@MarlonJD Thanks for the clarification. We will track this as a feature request.

MarlonJD commented 3 months ago

@Jordan-Nelson I created #4945 PR, for this issue.

MarlonJD commented 3 months ago

@Equartey Hey there 👋. I raised PR for this proposal, I hope you can review.

Equartey commented 2 months ago

Hi @MarlonJD, as usual, thank you for opening the issue and the accompanying PR.

First, we will need to investigate how to best address this for all our customers before we move forward with your proposal.

This process takes time, but we will provide updates here as we have them. Thanks again.

MarlonJD commented 2 months ago

Hi @Equartey. Thank you very much for your reply. I'll be waiting for updates.

hangoocn commented 2 months ago

I have the same issue and want to sort the values by createdAt. imo, if we define the secondary index like this: .secondaryIndexes(index => [index('xxxId').sortKeys(['createdAt']).queryField('listByDate')]) the cli will generate an api like ModelQueries.listByDate({sortDirection: 'DESC'}) which is same as the amplify js behavior

MarlonJD commented 2 months ago

Hey @hangoocn, I already created PR for this and waiting to merge, if you don't want to wait you can create CustomModelQueries class and use this, you can copy from the PR #4945.

Here is the example of how you can use this:

Create folder and create 3 new files called model_queries.dart, model_queries_factory.dart and graphql_request_factory:

model_queries.dart:

// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: Apache-2.0

import "package:amplify_core/amplify_core.dart";
import "model_queries_factory.dart";

/// Static helpers to generate query `GraphQLRequest` instances from models generated by Amplify codegen.

class CustomModelQueries {
  CustomModelQueries._(); // only static methods here, prevent calling constructor
  /// Generates a request for a list of model instances from
  /// secondary index. The `modelType` must have a secondary index defined.
  /// Check out the Amplify documentation for more information on how to define
  /// secondary indexes. https://docs.amplify.aws/gen1/react/build-a-backend/graphqlapi/best-practice/query-with-sorting/
  ///
  /// ```dart
  /// final request = ModelQueries.listByIndex(Todo.classType, queryField: parentId);
  /// ```
  /// or optional parameters:
  /// ```dart
  /// final request = ModelQueries.listByIndex(
  ///   Todo.classType,
  ///   queryField: parentId,
  ///   sortDirection: "DESC", // ASC or DESC, default is DESC
  ///   indexName: "todosByDate", // default is the first secondary index
  ///   overrideQueryFieldType: "ID!", // override the query field type, default is "ID!", e.g. "String!"
  ///   where: Todo.TASK.eq(taskId), // If your query parameter is taskId, you cannot use in where at the same time
  /// );
  /// ```

  static GraphQLRequest<PaginatedResult<T>> listByIndex<T extends Model>(
    ModelType<T> modelType, {
    int? limit,
    String? queryField,
    String? sortDirection,
    String? indexName,
    String? overrideQueryFieldType,
    String? customQueryName,
    QueryPredicate? where,
    String? apiName,
    APIAuthorizationType? authorizationMode,
    Map<String, String>? headers,
  }) {
    return ModelQueriesFactory.instance.listByIndex<T>(
      modelType,
      limit: limit,
      where: where,
      apiName: apiName,
      authorizationMode: authorizationMode,
      headers: headers,
      queryField: queryField,
      sortDirection: sortDirection ?? "DESC",
      indexName: indexName,
      overrideQueryFieldType: overrideQueryFieldType,
      customQueryName: customQueryName,
    );
  }
}

// TODO(ragingsquirrel3): remove when https://github.com/dart-lang/sdk/issues/50748 addressed
// ignore_for_file: flutter_style_todos

model_queries_factory.dart:

// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: Apache-2.0

// ignore_for_file: public_member_api_docs

import "package:amplify_core/amplify_core.dart";
import "graphql_request_factory.dart";

class ModelQueriesFactory {
  // Singleton methods/properties
  // usage: ModelQueriesFactory.instance;
  ModelQueriesFactory._();

  static final ModelQueriesFactory _instance = ModelQueriesFactory._();

  static ModelQueriesFactory get instance => _instance;

  GraphQLRequest<T> get<T extends Model>(
    ModelType<T> modelType,
    ModelIdentifier<T> modelIdentifier, {
    String? apiName,
    APIAuthorizationType? authorizationMode,
    Map<String, String>? headers,
  }) {
    final variables = modelIdentifier.serializeAsMap();
    return GraphQLRequestFactory.instance.buildRequest<T>(
      modelType: modelType,
      modelIdentifier: modelIdentifier,
      variables: variables,
      requestType: GraphQLRequestType.query,
      requestOperation: GraphQLRequestOperation.get,
      apiName: apiName,
      authorizationMode: authorizationMode,
      headers: headers,
    );
  }

  GraphQLRequest<PaginatedResult<T>> list<T extends Model>(
    ModelType<T> modelType, {
    int? limit,
    QueryPredicate? where,
    String? apiName,
    APIAuthorizationType? authorizationMode,
    Map<String, String>? headers,
  }) {
    final filter = GraphQLRequestFactory.instance
        .queryPredicateToGraphQLFilter(where, modelType);
    final variables = GraphQLRequestFactory.instance
        .buildVariablesForListRequest(limit: limit, filter: filter);

    return GraphQLRequestFactory.instance.buildRequest<PaginatedResult<T>>(
      modelType: PaginatedModelType(modelType),
      variables: variables,
      requestType: GraphQLRequestType.query,
      requestOperation: GraphQLRequestOperation.list,
      apiName: apiName,
      authorizationMode: authorizationMode,
      headers: headers,
    );
  }

  GraphQLRequest<PaginatedResult<T>> listByIndex<T extends Model>(
    ModelType<T> modelType, {
    int? limit,
    QueryPredicate? where,
    String? apiName,
    APIAuthorizationType? authorizationMode,
    Map<String, String>? headers,
    String? queryField,
    String? sortDirection,
    String? indexName,
    String? overrideQueryFieldType,
    String? customQueryName,
  }) {
    final filter = GraphQLRequestFactory.instance
        .queryPredicateToGraphQLFilter(where, modelType);
    final variables = GraphQLRequestFactory.instance
        .buildVariablesForListRequest(limit: limit, filter: filter);

    return GraphQLRequestFactory.instance
        .buildRequestForSecondaryIndex<PaginatedResult<T>>(
      modelType: PaginatedModelType(modelType),
      variables: variables,
      requestType: GraphQLRequestType.query,
      requestOperation: "listWithIndex",
      apiName: apiName,
      authorizationMode: authorizationMode,
      headers: headers,
      queryField: queryField,
      sortDirection: sortDirection,
      indexName: indexName,
      overrideQueryFieldType: overrideQueryFieldType,
      customQueryName: customQueryName,
    );
  }
}

and graphql_request_factory.dart:

// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: Apache-2.0

import "package:amplify_api_dart/src/graphql/utils.dart";
import "package:amplify_core/amplify_core.dart";
import "package:collection/collection.dart";

// ignore_for_file: public_member_api_docs, deprecated_member_use

/// `"id"`, the name of the id field when a primary key not specified in schema
/// with `@primaryKey` annotation.
const _defaultIdFieldName = "id";
// other string constants
const _commaSpace = ", ";
const _openParen = "(";
const _closeParen = ")";

/// Contains expressions for mapping request variables to expected places in the
/// request.
///
/// Some examples include ids or indexes for simple queries, filters, and input
/// variables/conditions for mutations.
class DocumentInputs {
  const DocumentInputs(this.upper, this.lower);

  // Upper document input: ($id: ID!)
  final String upper;
  // Lower document input: (id: $id)
  final String lower;
}

class GraphQLRequestFactory {
  GraphQLRequestFactory._();

  static final GraphQLRequestFactory _instance = GraphQLRequestFactory._();

  static GraphQLRequestFactory get instance => _instance;

  String _getName(ModelSchema schema, GraphQLRequestOperation operation) {
    // schema has been validated & schema.pluralName is non-nullable
    return operation == GraphQLRequestOperation.list
        ? schema.pluralName!
        : schema.name;
  }

  String _getSelectionSetFromModelSchema(
    ModelSchema schema,
    GraphQLRequestOperation operation, {
    bool ignoreParents = false,
  }) {
    // Schema has been validated & schema.fields is non-nullable.
    // Get a list of field names to include in the request body.
    final fields = schema.fields!.entries
        .where(
      (entry) => entry.value.association == null,
    ) // ignore related model fields
        .map((entry) {
      if (entry.value.type.fieldType == ModelFieldTypeEnum.embedded ||
          entry.value.type.fieldType == ModelFieldTypeEnum.embeddedCollection) {
        final embeddedSchema =
            getModelSchemaByModelName(entry.value.type.ofCustomTypeName!, null);
        final embeddedSelectionSet = _getSelectionSetFromModelSchema(
          embeddedSchema,
          GraphQLRequestOperation.get,
        );
        return "${entry.key} { $embeddedSelectionSet }";
      }
      return entry.key;
    }).toList(); // e.g. ["id", "name", "createdAt"]

    // If belongsTo, also add selection set of parent.
    final allBelongsTo = getBelongsToFieldsFromModelSchema(schema);
    for (final belongsTo in allBelongsTo) {
      final belongsToModelName = belongsTo.type.ofModelName;
      if (belongsToModelName != null && !ignoreParents) {
        final parentSchema =
            getModelSchemaByModelName(belongsToModelName, null);
        final parentSelectionSet = _getSelectionSetFromModelSchema(
          parentSchema,
          GraphQLRequestOperation.get,
          ignoreParents: true,
        ); // always format like a get, stop traversing parents
        fields.add("${belongsTo.name} { $parentSelectionSet }");

        // Include the ID key(s) itself in selection set. This is ignored while
        // deserializing response because model will use ID nested in `parentSelectionSet`.
        // However, including the ID in selection set allows subscriptions that
        // filter by these ID to be triggered by these mutations.
        final belongsToKeys = belongsTo.association?.targetNames;
        if (belongsToKeys != null) {
          fields.addAll(belongsToKeys);
        }
      }
    }

    // Get owner fields if present in auth rules
    final ownerFields = (schema.authRules ?? [])
        .map((authRule) => authRule.ownerField)
        .whereNotNull()
        .toList();

    final fieldSelection =
        [...fields, ...ownerFields].join(" "); // e.g. "id name createdAt"

    if (operation == GraphQLRequestOperation.list) {
      return "$items { $fieldSelection } nextToken";
    }

    return fieldSelection;
  }

  String _capitalize(String s) => s[0].toUpperCase() + s.substring(1);

  String _lowerCaseFirstCharacter(String s) =>
      s[0].toLowerCase() + s.substring(1);

  DocumentInputs _buildDocumentInputs(
    ModelSchema schema,
    dynamic operation,
    ModelIdentifier? modelIdentifier,
    Map<String, dynamic> variables, {
    String? overrideQueryFieldType,
    String? indexName,
  }) {
    var upperOutput = "";
    var lowerOutput = "";
    final modelName = _capitalize(schema.name);

    // build inputs based on request operation
    switch (operation) {
      case GraphQLRequestOperation.get:
        final indexes = schema.indexes;
        final modelIndex =
            indexes?.firstWhereOrNull((index) => index.name == null);
        if (modelIndex != null) {
          // custom index(es), e.g.
          // upperOutput: no change (empty string)
          // lowerOutput: (customId: 'abc-123, name: 'lorem')

          // Do not reference variables because scalar types not available in
          // codegen. Instead, just write the value to the document.
          final bLowerOutput = StringBuffer(_openParen);
          for (final field in modelIndex.fields) {
            var value = modelIdentifier!.serializeAsMap()[field];
            if (value is String) {
              value = '"$value"';
            }
            bLowerOutput.write("$field: $value");
            if (field != modelIndex.fields.last) {
              bLowerOutput.write(_commaSpace);
            }
          }
          bLowerOutput.write(_closeParen);
          lowerOutput = bLowerOutput.toString();
        } else {
          // simple, single identifier
          upperOutput = r"($id: ID!)";
          lowerOutput = r"(id: $id)";
        }
      case GraphQLRequestOperation.list:
        upperOutput =
            "(\$filter: Model${modelName}FilterInput, \$limit: Int, \$nextToken: String)";
        lowerOutput =
            r"(filter: $filter, limit: $limit, nextToken: $nextToken)";
      case "listWithIndex":
        // get secondary index query field
        String? queryFieldProp;

        try {
          if (indexName == null) {
            if (((schema.indexes ?? []).first.props).length >= 2) {
              queryFieldProp =
                  (schema.indexes?.first.props[1] as List<String>).first;
            }
          } else {
            queryFieldProp = (schema.indexes
                    ?.firstWhere((index) => index.name == indexName)
                    .props[1] as List<String>)
                .first;
          }
          // ignore: avoid_catches_without_on_clauses
        } catch (e) {
          throw const ApiOperationException(
            "Unable to get query field property from schema",
            recoverySuggestion: "please provide a valid query field",
          );
        }

        upperOutput =
            '(\$filter: Model${modelName}FilterInput, \$limit: Int, \$nextToken: String, \$queryField: ${overrideQueryFieldType ?? 'ID!'}, \$sortDirection: ModelSortDirection)';
        lowerOutput =
            r"(filter: $filter, limit: $limit, nextToken: $nextToken, " +
                (queryFieldProp ?? "") +
                r": $queryField, sortDirection: $sortDirection)";
      case GraphQLRequestOperation.create:
      case GraphQLRequestOperation.update:
      case GraphQLRequestOperation.delete:
        final operationValue = _capitalize(operation.name);

        upperOutput =
            '(\$input: $operationValue${modelName}Input!, \$condition:  Model${modelName}ConditionInput)';
        lowerOutput = r"(input: $input, condition: $condition)";
      case GraphQLRequestOperation.onCreate:
      case GraphQLRequestOperation.onUpdate:
      case GraphQLRequestOperation.onDelete:
        // Only add filter variable when present to support older backends that do not support filtering.
        if (variables.containsKey("filter")) {
          upperOutput = "(\$filter: ModelSubscription${modelName}FilterInput)";
          lowerOutput = r"(filter: $filter)";
        }
      default:
        throw const ApiOperationException(
          "GraphQL Request Operation is currently unsupported",
          recoverySuggestion: "please use a supported GraphQL operation",
        );
    }

    return DocumentInputs(upperOutput, lowerOutput);
  }

  /// Example:
  ///   query getBlog($id: ID!, $content: String) { getBlog(id: $id, content: $content) { id name createdAt } }
  GraphQLRequest<T> buildRequest<T extends Model>({
    required ModelType modelType,
    Model? model,
    ModelIdentifier? modelIdentifier,
    required GraphQLRequestType requestType,
    required GraphQLRequestOperation requestOperation,
    required Map<String, dynamic> variables,
    String? apiName,
    APIAuthorizationType? authorizationMode,
    Map<String, String>? headers,
    int depth = 0,
  }) {
    // retrieve schema from ModelType and validate required properties
    final schema =
        getModelSchemaByModelName(modelType.modelName(), requestOperation);

    // e.g. "Blog" or "Blogs"
    final name = _capitalize(_getName(schema, requestOperation));
    // e.g. "query" or "mutation"
    final requestTypeVal = requestType.name;
    // e.g. "get" or "list"
    final requestOperationVal = requestOperation.name;
    // e.g. "{upper: "($id: ID!)", lower: "(id: $id)"}"
    final documentInputs = _buildDocumentInputs(
      schema,
      requestOperation,
      modelIdentifier,
      variables,
    );
    // e.g. "id name createdAt" - fields to retrieve
    final fields = _getSelectionSetFromModelSchema(schema, requestOperation);
    // e.g. "getBlog"
    final requestName = '$requestOperationVal$name';
    // e.g. query getBlog($id: ID!, $content: String) { getBlog(id: $id, content: $content) { id name createdAt } }
    final document =
        '''$requestTypeVal $requestName${documentInputs.upper} { $requestName${documentInputs.lower} { $fields } }''';
    // e.g "listBlogs"
    final decodePath = requestName;

    return GraphQLRequest<T>(
      document: document,
      variables: variables,
      modelType: modelType,
      decodePath: decodePath,
      apiName: apiName,
      authorizationMode: authorizationMode,
      headers: headers,
    );
  }

  GraphQLRequest<T> buildRequestForSecondaryIndex<T extends Model>({
    required ModelType modelType,
    Model? model,
    ModelIdentifier? modelIdentifier,
    required GraphQLRequestType requestType,
    required dynamic requestOperation,
    required Map<String, dynamic> variables,
    String? apiName,
    APIAuthorizationType? authorizationMode,
    Map<String, String>? headers,
    int depth = 0,
    String? queryField,
    String? sortDirection,
    String? indexName,
    String? overrideQueryFieldType,
    String? customQueryName,
  }) {
    // retrieve schema from ModelT"ype and validate required properties
    final schema = getModelSchemaByModelName(
      modelType.modelName(),
      GraphQLRequestOperation.list,
    );

    // Get secondary index name
    String? secondaryIndexName;

    if (indexName == null) {
      try {
        secondaryIndexName = (schema.indexes ?? []).first.name;
        // ignore: avoid_catches_without_on_clauses
      } catch (e) {
        throw const ApiOperationException(
          "Unable to get secondary index name from schema",
          recoverySuggestion: "please provide a valid index name",
        );
      }
    } else {
      // Search for the index with the given name
      (schema.indexes ?? [])
              .where(
                (index) => index.name == indexName,
              )
              .isEmpty
          ? throw const ApiOperationException(
              "Given secondary index name not found in schema",
              recoverySuggestion: "please provide a valid index name",
            )
          : null;
    }

    // e.g. "query" or "mutation"
    final requestTypeVal = requestType.name;

    // ignore: prefer_single_quotes
    variables["queryField"] = queryField;

    // ignore: prefer_single_quotes
    variables["sortDirection"] = sortDirection;

    // e.g. "{upper: "($id: ID!)", lower: "(id: $id)"}"
    final documentInputs = _buildDocumentInputs(
      schema,
      requestOperation,
      modelIdentifier,
      variables,
      overrideQueryFieldType: overrideQueryFieldType,
      indexName: indexName,
    );

    // e.g. "id name createdAt" - fields to retrieve
    final fields =
        _getSelectionSetFromModelSchema(schema, GraphQLRequestOperation.list);

    // e.g. "getBlog"
    indexName ??= secondaryIndexName;

    if (indexName == null) {
      throw const ApiOperationException(
        "Unable to get secondary index name from schema",
        recoverySuggestion: "please provide a valid index name",
      );
    }

    indexName = "${schema.name}By${indexName.split("By").last}";

    var requestName = "list${_capitalize(indexName)}";

    if (customQueryName != null) {
      requestName = customQueryName;
    }

    // e.g. query getBlog($id: ID!, $content: String) { getBlog(id: $id, content: $content) { id name createdAt } }
    final document =
        """$requestTypeVal $requestName${documentInputs.upper} { $requestName${documentInputs.lower} { $fields } }""";
    // e.g "listBlogs"
    final decodePath = requestName;

    return GraphQLRequest<T>(
      document: document,
      variables: variables,
      modelType: modelType,
      decodePath: decodePath,
      apiName: apiName,
      authorizationMode: authorizationMode,
      headers: headers,
    );
  }

  Map<String, dynamic> buildVariablesForListRequest({
    int? limit,
    String? nextToken,
    Map<String, dynamic>? filter,
  }) {
    return <String, dynamic>{
      "filter": filter,
      "limit": limit,
      "nextToken": nextToken,
    };
  }

  Map<String, dynamic> buildVariablesForMutationRequest({
    required Map<String, dynamic> input,
    Map<String, dynamic>? condition,
  }) {
    return <String, dynamic>{
      "input": input,
      "condition": condition,
    };
  }

  Map<String, dynamic> buildVariablesForSubscriptionRequest({
    required ModelType modelType,
    QueryPredicate? where,
  }) {
    if (where == null) {
      return {};
    }
    final filter = GraphQLRequestFactory.instance
        .queryPredicateToGraphQLFilter(where, modelType);
    return <String, dynamic>{
      "filter": filter,
    };
  }

  /// Translates a `QueryPredicate` to a map representing a GraphQL filter
  /// which AppSync will accept. Exame:
  /// `queryPredicateToGraphQLFilter(Blog.NAME.eq('foo'));` // =>
  /// `{'name': {'eq': 'foo'}}`. In the case of a mutation, it will apply to
  /// the "condition" field rather than "filter."
  Map<String, dynamic>? queryPredicateToGraphQLFilter(
    QueryPredicate? queryPredicate,
    ModelType modelType,
  ) {
    if (queryPredicate == null) {
      return null;
    }
    final schema = getModelSchemaByModelName(modelType.modelName(), null);

    // e.g. { 'name': { 'eq': 'foo }}
    if (queryPredicate is QueryPredicateOperation) {
      final association = schema.fields?[queryPredicate.field]?.association;
      final associatedTargetName = association?.targetNames?.first;
      var fieldName = queryPredicate.field;
      if (queryPredicate.field ==
          "${_lowerCaseFirstCharacter(schema.name)}.$_defaultIdFieldName") {
        // very old schemas where fieldName is "blog.id"
        fieldName = _defaultIdFieldName;
      } else if (associatedTargetName != null) {
        // most schemas from more recent CLI codegen versions

        // when querying for the ID of another model, use the targetName from the schema
        fieldName = associatedTargetName;
      }

      return <String, dynamic>{
        fieldName: _queryFieldOperatorToPartialGraphQLFilter(
          queryPredicate.queryFieldOperator,
        ),
      };
    }

    // and, or, not
    if (queryPredicate is QueryPredicateGroup) {
      // .toShortString() is the same as expected graphql filter strings for all these,
      // so no translation helper required.
      final typeExpression = queryPredicate.type.name;

      // not
      if (queryPredicate.type == QueryPredicateGroupType.not) {
        if (queryPredicate.predicates.length == 1) {
          return <String, dynamic>{
            typeExpression: queryPredicateToGraphQLFilter(
              queryPredicate.predicates[0],
              modelType,
            ),
          };
        }
        // Public not() API only allows 1 condition but QueryPredicateGroup
        // technically allows multiple conditions so explicitly disallow multiple.
        throw const ApiOperationException(
          "Unable to translate not() with multiple conditions.",
        );
      }

      // and, or
      return <String, List<Map<String, dynamic>>>{
        typeExpression: queryPredicate.predicates
            .map(
              (predicate) =>
                  queryPredicateToGraphQLFilter(predicate, modelType)!,
            )
            .toList(),
      };
    }

    throw ApiOperationException(
      "Unable to translate the QueryPredicate $queryPredicate to a GraphQL filter.",
    );
  }

  /// Calls `toJson` on the model and removes key/value pairs that refer to
  /// children when that field is null. The structure of provisioned AppSync `input` type,
  /// such as `CreateBlogInput` does not include nested types, so they will get
  /// an error from AppSync if they are included in mutations.
  ///
  /// When the model has a parent via a belongsTo, the id from the parent is added
  /// as a field similar to "blogID" where the value is `post.blog.id`.
  Map<String, dynamic> buildInputVariableForMutations(
    Model model, {
    required GraphQLRequestOperation operation,
  }) {
    final schema =
        getModelSchemaByModelName(model.getInstanceType().modelName(), null);
    final modelJson = model.toJson();

    // Get the primary key field name from parent schema(s) so it can be taken from
    // JSON. For schemas with `@primaryKey` defined, it will be the first key in
    // the index where name is null. If no such definition in schema, use
    // default primary key "id."
    final allBelongsTo = getBelongsToFieldsFromModelSchema(schema);
    for (final belongsTo in allBelongsTo) {
      final belongsToModelName = belongsTo.name;
      final parentSchema = getModelSchemaByModelName(
        belongsTo.type.ofModelName!,
        operation,
      );
      final primaryKeyIndex = parentSchema.indexes?.firstWhereOrNull(
        (modelIndex) => modelIndex.name == null,
      );

      // Traverse parent schema to get expected JSON strings and remove from JSON.
      // A parent with complex identifier may have multiple keys.
      var belongsToKeys = belongsTo.association?.targetNames;
      if (belongsToKeys == null) {
        // no way to resolve key to write to JSON
        continue;
      }
      belongsToKeys = belongsToKeys;
      var i = 0; // needed to track corresponding index field name
      for (final belongsToKey in belongsToKeys) {
        final parentIdFieldName =
            primaryKeyIndex?.fields[i] ?? _defaultIdFieldName;
        final belongsToValue = (modelJson[belongsToModelName]
            as Map?)?[parentIdFieldName] as String?;

        // Assign the parent ID(s) if the model has a parent.
        if (belongsToValue != null) {
          modelJson[belongsToKey] = belongsToValue;
        }
        i++;
      }
      modelJson.remove(belongsToModelName);
    }

    final ownerFieldNames = (schema.authRules ?? [])
        .map((authRule) => authRule.ownerField)
        .whereNotNull()
        .toSet();
    // Remove some fields from input.
    final fieldsToRemove = schema.fields!.entries
        .where(
          (entry) =>
              // relational fields
              entry.value.association != null ||
              // read-only
              entry.value.isReadOnly ||
              // null values for owner fields on create operations
              (operation == GraphQLRequestOperation.create &&
                  ownerFieldNames.contains(entry.value.name) &&
                  modelJson[entry.value.name] == null),
        )
        .map((entry) => entry.key)
        .toSet();
    modelJson.removeWhere((key, dynamic value) => fieldsToRemove.contains(key));

    return modelJson;
  }
}

/// e.g. `.eq('foo')` => `{ 'eq': 'foo' }`
Map<String, dynamic> _queryFieldOperatorToPartialGraphQLFilter(
  QueryFieldOperator<dynamic> queryFieldOperator,
) {
  final filterExpression = _getGraphQLFilterExpression(queryFieldOperator.type);
  if (queryFieldOperator is QueryFieldOperatorSingleValue) {
    return <String, dynamic>{
      filterExpression: _getSerializedValue(queryFieldOperator.value),
    };
  }
  if (queryFieldOperator is BetweenQueryOperator) {
    return <String, dynamic>{
      filterExpression: <dynamic>[
        _getSerializedValue(queryFieldOperator.start),
        _getSerializedValue(queryFieldOperator.end),
      ],
    };
  }
  if (queryFieldOperator is AttributeExistsQueryOperator) {
    return <String, dynamic>{
      filterExpression: _getSerializedValue(queryFieldOperator.exists),
    };
  }

  throw ApiOperationException(
    "Unable to translate the QueryFieldOperator ${queryFieldOperator.type} to a GraphQL filter.",
  );
}

String _getGraphQLFilterExpression(QueryFieldOperatorType operatorType) {
  final dictionary = {
    QueryFieldOperatorType.equal: "eq",
    QueryFieldOperatorType.not_equal: "ne",
    QueryFieldOperatorType.less_or_equal: "le",
    QueryFieldOperatorType.less_than: "lt",
    QueryFieldOperatorType.greater_than: "gt",
    QueryFieldOperatorType.greater_or_equal: "ge",
    QueryFieldOperatorType.between: "between",
    QueryFieldOperatorType.contains: "contains",
    QueryFieldOperatorType.begins_with: "beginsWith",
    QueryFieldOperatorType.attribute_exists: "attributeExists",
  };
  final result = dictionary[operatorType];
  if (result == null) {
    throw ApiOperationException(
      "$operatorType does not have a defined GraphQL filter string.",
    );
  }
  return result;
}

/// Convert Temporal*, DateTime and enum values to string if needed.
/// Otherwise return unchanged.
dynamic _getSerializedValue(dynamic value) {
  if (value is TemporalDateTime ||
      value is TemporalDate ||
      value is TemporalTime ||
      value is TemporalTimestamp) {
    return value.toString();
  }
  if (value is DateTime) {
    return _getSerializedValue(TemporalDateTime(value));
  }
  if (value is Enum) {
    return value.name;
  }
  return value;
}

Now you can use this like this:

request = CustomModelQueries.listByIndex(
  Todo.classType,
  queryField: todoId,
  sortDirection: "DESC",
  limit: 10,
);

final response = await Amplify.API.query(request: request).response;

hangoocn commented 2 months ago

thnx @MarlonJD , i will have a try!

hangoocn commented 2 months ago

I also realized we can do 2 other walkarounds (I am now using option 1):

Option 1. Manully override the request before call the query api: first create a custom query like this (this should be auto generated in amplify js model gen when creating a secondary index)

String listMessageByDateQuery = '''
query ListByDate(
  \$createdAt: ModelStringKeyConditionInput
  \$filter: ModelMessageFilterInput
  \$limit: Int
  \$nextToken: String
  \$roomId: ID!
  \$sortDirection: ModelSortDirection
) {
  listByDate(
    createdAt:\$createdAt
    filter: \$filter
    limit: \$limit
    nextToken: \$nextToken
    roomId: \$roomId
    sortDirection: \$sortDirection
  ) {
    items {
      createdAt
      id
      owner
      roomId
      updatedAt
      __typename
    }
    nextToken
    __typename
  }
}
''';

Then call it like this:

    final request = ModelQueries.list<Message>(Message.classType);
    final requestOveride = request.copyWith(
      decodePath: 'listByDate',
      document: customMessageListByDateQuery,
      variables: {
        'limit': 100,
        'roomId': roomId,
        'sortDirection': 'DESC',
      },
    );

Option 2: Without override but just call with custom query like this: like option 1, add customMessageListByDateQuery first, then

    final request = GraphQLRequest<String>(
        document: customMessageListByDateQuery,
        variables: {
          'limit': 100,
          'roomId': roomId,
          'sortDirection': 'DESC',
        });

    final response = await Amplify.API.query(request: request).response;
    Map<String, dynamic> jsonMap = json.decode(response.data!);

    final items = jsonMap['listByDate']['items'];

Then we may need to cast items to List of Message so it is more work than option 1.

Reference: Official doc about creating custom query and mutation. We can even interact with the dynamodb directly by creating a custom lambda.

hangoocn commented 5 days ago

@Jordan-Nelson any updates on this? the doc is still showing tsx code samples for flutter.

So just to confirm the gen2 flutter api does not have built-in support for query by index, e.g. there is no way to get latest 10 posts without a full table scan?

MarlonJD commented 5 days ago

I don't wait to merge this pr, I created my custom queries now I'm using this, I recommend you to do this, I gave the files for this before

hangoocn commented 5 days ago

Thanks @MarlonJD I am also using an alternative approach by override query before sending to app sync, but is still expecting a offical way to do that. It seems it was supported in gen 1 data store, but unfortunately data store is not be supported in gen 2.

MarlonJD commented 5 days ago

@hangoocn I don't know plans about DataStore, I understand your hesitation about custom solutions for DataStore. It is not flexible enough like in AppSync GraphQL API.

NikaHsn commented 4 days ago

@hangoocn Amplify Flutter GraphQL API does not have built-in support to query with sorting by secondary index. this is an open feature request and we will provide update here as we have them. Thanks for your patience.

aws-amplify / amplify-flutter