GraphQL Transformer vNext RFC

[yuth]

Problem

The current GraphQL Transformer has tight coupling between transformer plugins and it is not modular.

In an effort to accelerate development, reduce operational load, enable more extensibility and also to meet the rising feature requests and enhancements, we're proposing a new architecture for the GraphQL Transformer moving forward. Refered henceforth as "GraphQL Transformer vNext"

Proposed Solution

The proposed solution will:

• Include well-defined plugin interfaces (i.e. increase feature development velocity for data modeling / directives)
• Modularize transformer plugins (i.e. @auth logic can be independent of datasource logic)
• Introduce phased transformation
• Provide runtime interfaces between resolver slots for inter-resolver communication ("The Resolver Runtime stash")
• Enable pipeline resolvers by default

Architecture

The GraphQL Transformer vNext is split into two main concepts:

• Transformer Plugins
• Resolver Runtime Stash

The Transformer plugins are categorized as follows:

• DataSource Providers (e.g. @model, @searchable)
• DataSource Enhancers (e.g. @key, @connection)
• Validation Transformers (e.g. @versioned, @unique)
• Data Enhancers (e.g. default value transformer, upper case transformer)
• Conditional Provider Transformers (e.g. @auth)

Lifecycle for transformer plugins

To support the inter-dependency between the following classes of transformers, the transformation will be done in a phased manner. These are the lifecycle methods that a transform plugin can implement to handle the different phases in the execution of the transformer.

Lifecycle Method	Description
before	Initialization of the transformer
GraphQL Vistor Pattern Functions	object for each type that has a directive defined by the transformer plugin interface for each interface that a directive defined by the transformer plugin field for each field that has a directive defined by the transformer plugin argument for each argument that has a directive defined by the transformer plugin union for each union that has a directive defined by the transformer plugin enum for each enum that has a directive defined by the transformer plugin enumValue for each enumValue that has a directive defined by the transformer plugin scalar for each scalar that has a directive defined by the transformer plugin input for each input that has a directive defined by the transformer plugin inputValue for each inputValue that has a directive defined by the transformer plugin
prepare	plugin to register themselves in the transformer context
validate	validate directive arguments
transformSchema	augment the output schema
generateResolver	generate resources such as resolvers, IAM policies, Tables, etc
after	clean up

The transformer plugin execution for each phase would run in the following order

1. DataSourceProvider Plugins
2. DataSourceEnhancer Plugins
3. Other Plugins
   1. Validation provider plugins
   2. DataEnhancer plugins
   3. Condition Provider plugin
   4. Auth plugin

This order ensures all operations and their resolvers are available for non-datasource specific plugins. These plugins will work with in the bounds of existing resolvers and operations.

Resolver Slots

In order for individual transformer plugin to contribute resolver snippets to a resolver, the transformer will generate resolvers that seperate the logic into slots. Based on the type of operation, the amount of slots will differ as follows:

Resolver Type	Slots
Query / Mutation	init, preAuth, auth, postAuth, preDataLoad, dataLoad, postDataLoad, finish
Subscription	init, preAuth, auth, postAuth, preSubscribe

Note: These can be rendered as individual AppSync Functions to be used within Pipeline Resolvers or they can be concatenated to generate a Unit Resolver.

Resolver Runtime Stash

The transformer will include a fixed object within $ctx.stash so the transformer plugins can store conditions, authConditions, default values, and transformed values. The datasource provider will use these values from these fields and merge default values, input values, and transformed values to generate the appropriate payload. (e.g. a query will have merged auth and input conditions)

The stash will have the following format

#set($ctx.stash = {
    defaultValues: {},
    transformedValues: {},
    authConditions: [],
    conditions: [],
    metadata: {}
})

Conditions and authConditions will take the shape of the input needed into the $util.transform functions.

$ctx.stash.metadata is a generic object that can be used to declare datasource specific information such as index name, keySchema information.

The resolver code generated by DataSourceProvider will merge defaultValues, arguments and transformedValues.

@model Implementation

export class DynamoDBModelTransformer extends Transformer implements TransformerModelProvider {
  modelTypes: Set<string> = new Set();
  modelDataSourceMap: Map<string, any> = new Map();
  constructor() {
    super(
      'DynamoDBModelTransformer',
      gql`
    directive @model(
      queries: ModelQueryMap
      mutations: ModelMutationMap
      subscriptions: ModelSubscriptionMap
      timestamps: TimestampConfiguration
    ) on OBJECT
    ....
  `,
    );
  }

  public object = (def: ObjectTypeDefinitionNode, directive: DirectiveNode, ctx: TransformerContext): void => {
    // Add a stack mapping so that all model resources are pulled
    // into their own stack at the end of the transformation.
    const typeName = def.name.value;
    this.modelTypes.add(typeName);
  };

  public prepare = (ctx: TransformerContext) => {
    for (let typeName of this.modelTypes.values()) {
      const objectDefinition = ctx.output.getObject(typeName);
      ctx.providerRegistry.registerDataSourceProvider(objectDefinition, this);
    }
  };

  public validate = (ctx: TransformerContext): void => {
    // do the validation of the directive arguments
  };

  transformSchema = (ctx: TransformerContext) => {
    for (let typeName of this.modelTypes.values()) {
      const objectDefinition = ctx.output.getObject(typeName);

      const newFields = [
        ...this.getQueryFieldNames(ctx, objectDefinition),
        ...this.getMutationFields(ctx, objectDefinition),
        ...this.getSubscriptionFields(ctx, objectDefinition),
      ];

      for (let op of newFields) {
        const opType = ctx.getObject(op.typeName);
        const inputs = this.getInputs(ctx, objectDefinition, op);
        // add the input if its missing in the output
        for (let input of inputs) {
          if (!ctx.output.getType(input)) {
            ctx.output.addInput(input);
          }
        }
        const field = makeField(op.fieldName, [inputs]);
        const extension = objectExtension(opType, field);
        this.output.addObjectExtension(extension);
      }
    }
  };

  generateResolvers = (ctx: TransformerContext) => {
    for (let typeName of this.modelTypes.values()) {
      const objectDefinition = ctx.output.getObject(typeName);

      const table = makeModelTable(typeName);
      ctx.resources.add(table);
      const role = makeIAMRole(typeName);
      ctx.resources.add(role);

      const datasource = makeDynamoDBDataSource(table, role, typeName, syncEnabled);
      ctx.datasources.add(objectDefinition, datasource);

      for (let query of this.getQueryFieldNames(ctx, objectDefinition)) {
        switch (query.type) {
          case MutationFieldType.CREATE:
            this.generateGetResolver(ctx, objectDefinition, query.typeName, query.fieldName);
            break;
          case QueryFieldType.LIST:
            this.generateListResolver(ctx, objectDefinition, query.typeName, query.fieldName);
            break;
          case QueryFieldType.SYNC:
            this.generateSyncResolver(ctx, objectDefinition, query.typeName, query.fieldName);
            break;
        }
      }

      for (let mutation of this.getMutationFieldName(ctx, objectDefinition)) {
        switch (mutation.type) {
          case MutationFieldType.CREATE:
            this.generateCreateResolver(ctx, objectDefinition, mutation.typeName, mutation.fieldName);
            break;
          case MutationFieldType.UPDATE:
            this.generateUpdateResolver(ctx, objectDefinition, mutation.typeName, mutation.fieldName);
            break;
          case MutationFieldType.DELETE:
            this.generateDeleteResolver(ctx, objectDefinition, mutation.typeName, mutation.fieldName);
            break;
        }
      }

      // Same step for subscriptions
    }
  };
}

Progress tracker

Foundational infrastructure of GraphQL Transform vNext

• [x] Well defined interfaces from graphql API, transformerPlugin
• [x] Feature Flags support for existing projects
• [x] Support Pipeline resolvers
• [x] Support inter dependency between transformer plugins
• [x] Decouple data source specific auth logic
• [ ] Easier to write custom transformers

Directives / Use cases

• [x] @model
• [x] @key
• [x] @auth
• [x] @connection
• [x] @function
• [x] @http
• [x] @predictions
• [x] @searchable
• [x] @default
• [ ] More directives / use cases - TBD

This list is not fully finalized. If you have additional use cases and scenarios you'd like to cover, please provide your comment below. As we get closer to implementation, the directives might change to be more intuitive towards the underlying use cases.

Trying it yourself

⚠️ THE GRAPHQL TRANSFORMER VNEXT IS NOT PRODUCTION-READY YET ⚠️

Currently, the GraphQL Transformer vNext will enable you to use pipeline resolvers and the @model directive. It showcases the architecture and provides a reference implementation of a custom directive. You can test this by:

1. Edit to amplify/cli.json
2. Set the value of useExperimentalPipelinedTransformer to true

⚠️ THE GRAPHQL TRANSFORMER VNEXT IS NOT PRODUCTION-READY YET ⚠️

[RossWilliams]

I’m confused about the unit resolvers vs pipeline resolvers. Which phase and which transformers will determine whether a pipeline or unit resolver is used ?

Will all the base transformers be implemented in VTL ?

Do you plan to maintain backwards compatibility with existing 1st party transformers’ schema outputs ?

Will the implementation be open to community members, or will major components be developed internally ? If implementation is public, how do you plan to coordinate the effort with community members ?

How will this new approach affect CreatedAt, createdBy, updatedAt, updatedBy properties on the @model transformer ? These should be handled by a data enhancer plugin according to the described RFC . Will this be the case ?

[GeorgeBellTMH]

Will this support union's and interfaces?

[r0zar]

I love hearing this. Reserving this spot for some deeper feedback, coming soon.

[renegoretzka]

I am looking forward to support interface / unions. It would be awesome if you can set the @model directive on the interface and merge all types which implement the interface into one database entry.

[renebrandel]

Happy new year. Thanks for the comments @GeorgeBellTMH and @RossWilliams

I’m confused about the unit resolvers vs pipeline resolvers. Which phase and which transformers will determine whether a pipeline or unit resolver is used ?

The new GraphQL Transformer will provide pipeline resolver functionality and unit resolver functionality. We're still brainstorming about the DX here (look out for an RFC soon, we'll use this RFC as a main tracking RFC). Some of the ideas could be based on special directives like @pipeline or even folder-structure enforced paradigms similar to how Next.js functions. Happy to hear your suggestions here.

Will all the base transformers be implemented in VTL ?

I assume for 'base transformers' you mean the Amplify-provided transformers? At this point, yes. VTL is the most cost-effective way but we're looking into the ability to extend it with pipeline resolvers that aren't written in VTL.

Do you plan to maintain backwards compatibility with existing 1st party transformers’ schema outputs ?

We're still working through the migration plan. The goal is to have feature-parity (not necessarily DX-parity) from GraphQL Transformer v1. We are exploring the efforts for "auto-migration" from v1 to v2. The reason for this is, we believe we have an opportunity here to improve on the DX of some of the directives / use cases but will want to maintain feature-parity.

Will the implementation be open to community members, or will major components be developed internally ? If implementation is public, how do you plan to coordinate the effort with community members ?

We'll address this on a case-by-case basis. We plan to publish individual RFCs for different directives and we can work through a collaboration / coordination plan in those specific RFCs.

How will this new approach affect CreatedAt, createdBy, updatedAt, updatedBy properties on the @model transformer ? These should be handled by a data enhancer plugin according to the described RFC . Will this be the case ?

Yes, these would be data enhancer plugins' responsibility.

Will this support union's and interfaces? @GeorgeBellTMH - can you give me an example of your use case where you're using unions & interfaces? What kind of a DDB table setup would you expect given those examples? Let's say if you have an interface, would you expect implementors to be just using a single table or individual tables for the implementors?

[phs1919]

Does this support custom auth(not AWS cognito based)? I hope it does since migrating existing auth system to AWS cognito is currently one of the blockers from using Amplify

[GeorgeBellTMH]

@renebrandel from my perspective I don't too much care what the ddb ends up looking like - as they say - "it's an implementation detail". Other than it shouldn't break other functionality and it should be performant (and ideally cheap).

My main reason for wanting this functionality is that it simplifies the graphql and allows for improved typing in our front end.

[estyh]

I need to use custom auth logic for all of my queries/mutations. I would like to be able to write a custom auth function (vtl is fine) and have it run by default for all the generated resolvers, without having to specify it each time (that would become very tedious and error-prone). Specifically, my use-case is as follows: I have a ddb table that specifies permissions across roles, eg Admin role has a collection of permissions that include "writePosition" and "readPosition", and HR role includes "readPosition" permission. Each of my users belongs to one or many user groups that define their roles at a particular company, eg "Company1#Admin" or "Company2#HR". Every model would have separate read and write permissions, eg a Position model can have "readPosition" and "writePosition". In addition, the model has a company field, since the object belongs to a specific company. When a user tries to perform an action, eg update a Position, I need to first check that the user belongs to a group where: 1) the company matches the position company he wants to update, 2) and where the role includes the "writePosition" permission. Ideally, I would be able to write an auth middleware type of function and specify that it is a default pre-resolver auth transformer type of thing. It would need to be able to have access to the cognito user groups, possibly his custom attributes, and to read some custom directives on the model (eg the read and write permissions).

I'm not sure this RFC is the right place for this, but it seems somewhat related. In any case, I'm really excited about the direction of this RFC, many thanks to the amplify team!

[yaquawa]

Looks like the framework is going to gain an evolution from the new architecture 🎉 I'm glad to see the newly added @default! I'd like to see more "validation rules directive" for server side validation. Also please consider this feature request aws-amplify/amplify-category-api#210

[undefobj]

Tracking cross table authorization with vNext which will support Pipeline resolvers: aws-amplify/amplify-category-api#341

[yaquawa]

Hi, I'd like to know that are there any milestones for this? Has had this project dead?

[ronaldocpontes]

I also can't wait for RFC to get implemented.

We are having many issues using Amplify to build multi-tenant authentication and simple business logic that need to happen before the model kicks in.

Is this thread still active?

[yaquawa]

Please consider implement "Transformer order" too. This is common and significant for a plugin architecture. https://github.com/aws-amplify/amplify-cli/issues/7297

[renebrandel]

Hi folks, we've started making more progress on this. We've published an RFC now for "@searchable" improvements https://github.com/aws-amplify/amplify-cli/issues/7546

[ronaldocpontes]

That's great! Is there any traction happening outside @searchable? I am particularly interested in lifecycle plugins and executing custom logic before and after @model entities.

[mansdahlstrom1]

Any status updates on unions & interfaces?

[MontoyaAndres]

Is there any possibility to support Lambda Authorizers? It's very important for Saas applications...

[renebrandel]

🚀 Preview announcement!

Hi everyone! Today we’re announcing the PREVIEW of our next generation GraphQL Transformer. This will be the biggest update we’ve shipped to our GraphQL functionality since its original inception in Amplify CLI. We’ve rearchitected the GraphQL capabilities from the ground up to ship new features and updates faster.

Install the new GraphQL Transformer preview:

DO NOT USE THIS IN PRODUCTION!

1. npm i -g @aws-amplify/cli@6.1.0-graphql-vnext-dev-preview.2 (make sure to uninstall existing Amplify CLI versions!)
2. Create a new Amplify project: amplify init
3. Add a new GraphQL API: amplify add api

What's New in this Preview

Summary:

• “deny-by-default” Authorization pattern. If auth rule isn’t specified on a model, it’s denied by default. Note: When you create a new schema, there's a special directive on the top of the schema template file that enables full CRUD access via API key to help you get started faster. To "lock down" your schema with "deny-by-default" authorization, remove the @allow_public_data_access_with_api_key directive.
• More descriptive directives for relationship modeling: @hasOne, @hasMany, and @belongsTo. @connection will be deprecated.
• More descriptive table index directives @primaryKey and @index. @key will be deprecated.
• Simpler, yet more feature-rich amplify add api and amplify update api workflows
• @searchable now uses OpenSearch version 7.10 and provides capabilities to do multi-field sort, per-user aggregation values (count, sum, avg, min, max), pagination, and faster data backfilling.
• GraphQL Transformer is now fully pipeline resolver-based!
• DataStore support for @hasOne, @hasmany, @manytomany, @belongsto, @primaryKey, and @index
• New “@default” directive to assign default values to optional fields upon creation
• New “@manyToMany” directive that helps customers model many-to-many relationships
• Pipeline resolver overwrites and slotting experience
• Ability to add custom pipeline resolvers

What’s not ready yet in this preview (but will be there at GA launch):

• Lambda authorization modes for the GraphQL API
• “Auto-migration” workflow from transformer v1 schemas to this new transformer schema.
• Admin UI support

In detail:

Authorization capability enhancements:

• All Amplify GraphQL APIs now use a “deny-by-default” authorization pattern
• Details on the specific authorization rule changes
  • Prior to this migration, Amplify CLI will automatically allow all data access with the default authorization type (API Key, UserPools, OIDC only) unless otherwise specified. With the new GraphQL Transformer, all data access is deny-by-default and customers need to explicitly specify authorization rules to enable clients to access their data.
  • Prior to this migration, when customers used owner-based authorization @auth(rules: [{allow: owner, operations: [create, read, update, delete]}]), the “operations” fields will be used to deny others access to the listed operations. In this example: “others can’t read, update, or delete”. With the new GraphQL Transformer, given the new deny-by-default paradigm, the owner-based authorization’s operation now specify what owners are allowed to do. The same example above now means: “Owners can create, read, update, and delete”.
  • Prior to this migration, when customers specified an @auth rule on a field, the authorization is also applied on the model. With the new GraphQL Transformer, @auth rules specified on a field will overwrite all authorization rules defined on the model level. If the some model level authorization rules need to be applied on a field level, you need to explicitly specify it.

Developer experience enhancements:

• A slicker set up experience. amplify add api has been significantly shortened to help customer set up GraphQL APIs faster. (From previously 10 steps down to 4 steps)
• On amplify push, we’ll also scan your schema files and ensure that the AppSync API’s authorization mode is correctly configured. (This helps if you’ve used “owner” @auth rules but forgot to run amplify update api to configure Cognito as an authorization mechanism)

Relational data modeling enhancements (formerly @key and @connection)

• @connection is being replaced with new directives.

  • @hasOne creates a “has one” relationship with the target model.
    • directive @hasOne(fields: [String!]) on FIELD_DEFINITION
  • @hasMany creates a "has many" relationship with the target model.
    • directive @hasMany(indexName: String, fields: [String!], limit: Int = 100) on FIELD_DEFINITION
  • @belongsTo facilitates a bi-directionality for a relationship. For example, use the @belongsTo directive on a “has many”-relationship’s target model to create a relationship back to the source model.
    • directive @belongsTo(fields: [String!]) on FIELD_DEFINITION

  • @manyToMany creates a new relationship cardinality for Amplify’s GraphQL Transformer. Traditionally, customers had to create a “join table” between two models and create hasMany relationships from both models into that join table as a work around for this feature. With the new transformer, customers can just specify a @manyToMany relationship between the models, Amplify CLI will create the join tables under the hood on behalf of the customer.

    type Post @model {
    id: ID!
    title: String!
    editors: [User] @manyToMany(relationName: "PostEditors")
    }
    
    type User @model {
    id: ID!
    username: String!
    posts: [Post] @manyToMany(relationName: "PostEditors")
    }

• The @key directive is being replaced by two new directives.
  • Customers can now specify @primaryKey on a field to define it as the primary key of a table. Customers can also specify sort key fields via a directive argument.
  • Customers can now specify @index on a field to use it as the hash key for a DynamoDB Global Secondary Index. Customers can optionally also specify a queryField or sort key fields by passing additional parameters.
  • 

Searchable enhancements:

• Customers can get aggregation values such as “total counts”, “average”, “sum”, "min", and "max" information. Authorization rules are also automatically applied on the OpenSearch query to restrict the aggregation values to only account for authorized data.
• Customers can now paginate over their OpenSearch search results via the nextToken
• OpenSearch version 7.10 is now the new default version.
• Backfilling batch size has been increased to 100 which will bring down the cost of backfill and allows faster backfilling of existing data.
• Customers can now sort their OpenSearch search results via one or more field types to better refine their search queries.

[kldeb]

This part isn't clear to me:

Prior to this migration, when customers specified an @auth rule on a field, the authorization is also applied on the model. With the new GraphQL Transformer, @auth rules specified on a field will overwrite all authorization rules defined on the model level. If the some model level authorization rules need to be applied on a field level, you need to explicitly specify it.

By overwrite auth rules specified on the model that means only for that field with the auth rules?

[JamieSlome]

@adam-nygate @benharvie @psmoros @zidingz - here we come! 🎉

[renebrandel]

This part isn't clear to me:

Prior to this migration, when customers specified an @auth rule on a field, the authorization is also applied on the model. With the new GraphQL Transformer, @auth rules specified on a field will overwrite all authorization rules defined on the model level. If the some model level authorization rules need to be applied on a field level, you need to explicitly specify it.

By overwrite auth rules specified on the model that means only for that field with the auth rules?

The intent here is to create a developer experience where the authorization rules are "explicit" and "what you see is what you get". Maybe the example below can help:

type Test @model @auth(rules: [{ allow: owner }]) {
  id: ID!
  name: String @auth(rules: [{allow: private }])
  description: String @auth(rules: [{allow: private, operations: [ read ] }, {allow: owner}])
}

In this case:

Owners are allowed to:

• create, read, update, delete id
• create, read, update, delete description but denied to:
• create, read, update, delete name

Any signed-in user (private) is allowed to:

• create, read, update, delete name
• read description but denied to:
• create, read, update, delete id
• create, update, delete description

[renebrandel]

Hi folks - a quick update. We just published a new build!

Install by running npm install -g @aws-amplify/cli@6.1.0-graphql-vnext-dev-preview.1

Change log:

• A bug fix for invalid "OpenSearch" instance size prompt
• @default directive is now available!! Use @default on optional fields to assign a default value upon record creation
• DataStore support for @hasOne, @hasMany, @manyToMany, @belongsTo, @primaryKey, and @index

[warlord987]

Can we Goto a specific page number now?

[PeteDuncanson]

This is all lovely to see and read about. Well done all.

Only bit that looked a bit odd is the @index directive, that seems overloaded in my head with indexing into a search engine and what not. Could we not find a more descriptive name to follow on from the hasMany, primaryKey style we have. Index just feels a bit too generic and a catch up, I'd have to keep looking it up to remind myself what it means I think.

Just me? Do I make sense?

[renebrandel]

This is all lovely to see and read about. Well done all.

Only bit that looked a bit odd is the @Index directive, that seems overloaded in my head with indexing into a search engine and what not. Could we not find a more descriptive name to follow on from the hasMany, primaryKey style we have. Index just feels a bit too generic and a catch up, I'd have to keep looking it up to remind myself what it means I think.

Just me? Do I make sense?

We discussed this quite a bit internally and never felt like we've found a better "word" for it. Do you have another recommendation?

[PeteDuncanson]

I really wanted to have one, I really did but the trouble is I can't see what is actually doing now so I don't know the context to be able to come up with another word...which sort of proves my point :)

[PeteDuncanson]

Can you sum up what its doing in a digestible way?

[kjones]

When using owner auth, will we be able to create an @index on the generated owner field? I currently need to add the owner field to the type to be able to do this which also exposes owner to the clients.

[renebrandel]

@PeteDuncanson - @index will create a new access pattern for the particular model. It's "as if" you created a global secondary index on the DDB table. Previously this was done via the @key directive if you've specified both the fields and a key name. This essentially helps you optimize your data access patterns. Potentially something like @query (also another overloaded term, could be better?)

[renebrandel]

@kjones - what's your particular use case where you'd need a GSI on the owner field? Like what access patterns / customer scenario are you trying to achieve? That'll help me better understand how we want to incorporate something built-in by default.

[renebrandel]

Hi folks! Another update:

Install by running npm install -g @aws-amplify/cli@6.1.0-graphql-vnext-dev-preview.2.

If you try this in an existing project, you'll need to edit the cli.json file and set "transformerVersion": 2 under features.graphqltransformer

Change log:

• @manyToMany directive to help you model that easier
• Pipeline resolver overwrites and slotting experience
• Ability to add custom pipeline resolvers

[PeteDuncanson]

@renebrandel

It's "as if" you created a global secondary index on the DDB table. Previously this was done via the @key directive if you've specified both the fields and a key name. This essentially helps you optimize your data access patterns. Potentially something like @query (also another overloaded term, could be better?)

What about @lookup as that to me sounds like what its doing "Lookup this data by this id". When you go use an old school libary (as in books) style card index that is sort of what you would do, "go look it up in the card index"...thoughts?

BTW exciting to see @default released, keep them coming :)

[renebrandel]

type Employee
  @model 
  @lookup(by: "name", sortBy: "description")
  @searchable {
  id: ID!
  firstName: String!
  lastName: String!
  age: String
}

Something like this?

[PeteDuncanson]

Oh my....that reads beautifully. Yes. Just like that.

[PeteDuncanson]

Wondered what it would look like with multiple lookups.

type Employee
  @model 
  @lookup(by: "lastName", sortBy: "description")
  @lookup(by: "age", sortBy: "description")
  @searchable {
  id: ID!
  firstName: String!
  lastName: String!
  age: String
  description: String
}

Still works.

The by attribute is the field we are using as the key right? Or is it the name of this index like it used to be? I've changed it to lastName in my example to match the field name...am I understanding that correctly?

[renegoretzka]

I think @lookup is a bit confusing. It's a declaration on how to index data into the GSI tables, right? But to me, it looks more like an equivalent to the @connection directive.

[PeteDuncanson]

but @connection (which I don't like currently) is getting binned for the nicer sytaxical sugar of @hasMany @hasOne etc. so it wouldn't be in the mix right?

I like that we are moving away from the language of the internals of DyanmoDB to instead make it easier to pick up (I came to AWS because of Amplify so I was a complete newbie to AWS land, this easier language is helping greatly with mapping concepts in my mind).

[ptejada]

I think index is a common term in relational databases. However, I always see the phrase access patterns around the current @key and the next @index directives. Whenever I see this phrase query comes to mind. You are basically creating a way to query your data so I will also be fine if the directive is called @query as I saw in some previous comments. I am not really sure what an overloaded word or term is.

Here is an example of how it can look like

type Employee @model  @query(by: "age", sortBy: ["firstName", "lastName"], name: "listByAge") {
  id: ID!
  firstName: String!
  lastName: String!
  age: String
}

The name of the query could be generated or offered if omitted. I personally don't know of a reason to make an index/query/GSI if is not to query data.

[kldeb]

While I agree that the language used in amplify needs to be approachable for new devs it also needs to point to the underlying technology.

Index is close to gsi so makes most sense to me.

There are cases where you may want to use an index to make a connection between types but not expose a query to the ui.

[michalpierzchlewicz]

@renebrandel great staff!

Regarding:

@manyToMany creates a new relationship cardinality for Amplify’s GraphQL Transformer. Traditionally, customers had to create a “join table” between two models and create hasMany relationships from both models into that join table as a work around for this feature. With the new transformer, customers can just specify a @manyToMany relationship between the models, Amplify CLI will create the join tables under the hood on behalf of the customer.

How will this work, if we want to store extra meta data inside the join table? E.g. date created or status (pending, rejected, approved). WIth this new approach, will there be an option for this?

[renebrandel]

You'll be able to use our built-in auto-generated fields like createdAt, updatedAt. What's your use case, where you'd like the join table to have more meta data, wouldn't you usually want the meta data on each of the origin tables? Also do you have a proposal for what you'd like that API to look like?

[michalpierzchlewicz]

createdAt, updatedAt are great. Regarding our use case, we run a social media application and we are shifting to AWS Neptune slowly, but at this moment we are modeling relations between users and posts in DynamoDB. Example: you can follow a user, but they need to first accept the follow. We store the status (pending, rejected, approved) of this follow in the "join table".

Regarding the proposal for the API, I have not reviewed the details of the new @manytomany directive. From what I can see from this discussion, is that you can pass the relationName parameter to it, but I am not sure what this parameter does. Is there any place I could reference to check the details? Also, I am not sure how with the new directive will effect the queries and mutations. E.g. how will you create or delete the connection. In short it would be great to be able to somehow pass the "join table" model to the @manytomany.

[majirosstefan]

What about transactWriteItems and batch writes?

Transactions for Dynamo DB were introduced in 2018 (3! years ago). In 2019 https://github.com/aws/aws-appsync-community/issues/14, they were supported by AWS-Appsync, and still, I am not seeing any mentions here.

Same with BatchWrites: aws-amplify/amplify-category-api#443 BatchWrites, just for start, in the same table, should be relatively easy to implement, not? One new @directive, and two VTL resolvers (for request and response) - that we now have to create manually.

@model Item **@withBatch** {
 id: ID!
}

Mutation.createItemsBatch.req.vtl and Mutation.createItemsBatch.res.vtl

Resolvers content is already here: https://docs.aws.amazon.com/appsync/latest/devguide/tutorial-dynamodb-batch.html

Then, if you would need to write into different tables with batch, you can do something like :

@model Item @withBatch(queryBatchName: "sameQueryName") { id: ID! }

@model Item2 @withBatch(queryBatchName: "sameQueryName") { id: ID! }

so by using the same queryBatchName you will know, that you need to create just one query, and just one req/res VTL resolver pair.

[sherl0cks]

@renebrandel Some feedback from testing. For context I'm working on a greenfield product which is centered around DataStore. It's a great solution, and I'm excited for the new transformer release to smooth out sharp edges. If you want to discuss that use case in more detail, I'm happy to connect:

1. Unions are still not supported. This is extremely disappointing. What's going on? This is a real pain point for me, as my team is already stringifying data and then in the application parsing it to types defined outside of amplify. It would be greatly appreciated to remove this complexity. Same applies to custom scalars to support GeoJSON in appsync, though I suspect this outside your domain.
2. The provided example above where @primaryKey and @index are on the same field does not work for me. I needed to add @primaryKey.sortKeyFields which makes sense if you think through the resulting dynamo infra. In this case scenario it would actually make sense to create an LSI and not a GSI but that's low priority.
3. type AMPLIFY_GLOBAL @allow_public_data_access_with_api_key(in: "dev") # FOR TESTING ONLY! is a nice touch - it should be added to the docs here.
4. New dev experience for amplify add api is nice.
5. The @index language is perfectly clear to me. I see no reason to change it.

[renebrandel]

Hi @sherl0cks - thanks for all this feedback!

1. Unions are on our roadmap but will come after the release of the new transformer. It's unfortunately a deceptively difficult problem to support all/most existing features. We'll look into it more after this transformer v2 release.
2. Good catch. I think there's some work here on the docs in the mean time, while we figure out the GSI vs. LSI story.
3. Thanks!
4. Thanks some more!
5. Awesome!

[renebrandel]

What about transactWriteItems and batch writes?

Transactions for Dynamo DB were introduced in 2018 (3! years ago). In 2019 aws/aws-appsync-community#14, they were supported by AWS-Appsync, and still, I am not seeing any mentions here.

Same with BatchWrites: aws-amplify/amplify-category-api#443 BatchWrites, just for start, in the same table, should be relatively easy to implement, not? One new @directive, and two VTL resolvers (for request and response) - that we now have to create manually.

@model Item **@withBatch** {
 id: ID!
}

Mutation.createItemsBatch.req.vtl and Mutation.createItemsBatch.res.vtl

Resolvers content is already here: https://docs.aws.amazon.com/appsync/latest/devguide/tutorial-dynamodb-batch.html

Then, if you would need to write into different tables with batch, you can do something like :

@model Item @withBatch(queryBatchName: "sameQueryName") { id: ID! }

@model Item2 @withBatch(queryBatchName: "sameQueryName") { id: ID! }

so by using the same queryBatchName you will know, that you need to create just one query, and just one req/res VTL resolver pair.

Thanks for raising the attention on this. It's a theme we're also looking to tackle after the initial release of Transformer v2. I'll see if in the mean time we can at least come up with a guide or something similar to educate customers on how to do it using a custom resolver within Amplify.

[sherl0cks]

Thanks for the quick reply @renebrandel.

1. Unions are on our roadmap but will come after the release of the new transformer. It's unfortunately a deceptively difficult problem to support all/most existing features. We'll look into it more after this transformer v2 release.

Understood. Could you please update release notes on the feature requests you intend to support on the v2 roadmap but not in the initial release?

Since we're talking roadmap, a few other feature requests I'd like to share:

• Support _deleted as a filter option in list operations. This is particularly useful during testing where you may be regularly deleting items through appsync and you don't want to page through a bunch of deleted items to get new.
• Single table support. I've seen this discussed but cannot find the issue. It would be good to get a direction here. Is it being considered? Definitely not going to happen?
• Native support for the dynamodb version control pattern. This is a higher order feature, and it can be built on top of the generated gql / datastore today ( (once this bug is fixed)[https://github.com/sherl0cks/datastore-reproducer] - support case is open). This is in addition to managing item level conflict detection through @versioned. This is a pretty complex feature and it may even be sufficient to provide example code / blog post.
• CLI option to stage cloudformation changes like you can do with cdk

[24jr]

So is this stuff in the current normal amplify cli or do we still do something like mentioned above npm install -g @aws-amplify/cli@6.1.0-graphql-vnext-dev-preview.2 also if its the case that we can use today are there any resources that I can use to help me even though official docs are still for current setup.

[24jr]

Actually I believe this is the latest version now https://www.npmjs.com/package/@aws-amplify/cli/v/6.4.0-graphql-vnext-dev-preview.4 I see some talk on different ways of doing stuff above like @index @lookup @query not sure if there are sketchpad notes on how to do some of stuff available now or if anyone knows some basics. I'm super excited for this I just can't wait but idk what to try will just say like Unknown directive "query" or Unknown directive "index"

[413n]

Is a release planned for this new GraphQL Transformer?

I see a version 7.3.0-graphql-vnext-dev-preview.12 so how much are we close to the release day?