Closed yuth closed 1 year ago
I’m confused about the unit resolvers vs pipeline resolvers. Which phase and which transformers will determine whether a pipeline or unit resolver is used ?
Will all the base transformers be implemented in VTL ?
Do you plan to maintain backwards compatibility with existing 1st party transformers’ schema outputs ?
Will the implementation be open to community members, or will major components be developed internally ? If implementation is public, how do you plan to coordinate the effort with community members ?
How will this new approach affect CreatedAt, createdBy, updatedAt, updatedBy properties on the @model transformer ? These should be handled by a data enhancer plugin according to the described RFC . Will this be the case ?
Will this support union's and interfaces?
I love hearing this. Reserving this spot for some deeper feedback, coming soon.
I am looking forward to support interface / unions. It would be awesome if you can set the @model directive on the interface and merge all types which implement the interface into one database entry.
Happy new year. Thanks for the comments @GeorgeBellTMH and @RossWilliams
I’m confused about the unit resolvers vs pipeline resolvers. Which phase and which transformers will determine whether a pipeline or unit resolver is used ?
The new GraphQL Transformer will provide pipeline resolver functionality and unit resolver functionality. We're still brainstorming about the DX here (look out for an RFC soon, we'll use this RFC as a main tracking RFC). Some of the ideas could be based on special directives like @pipeline
or even folder-structure enforced paradigms similar to how Next.js functions. Happy to hear your suggestions here.
Will all the base transformers be implemented in VTL ?
I assume for 'base transformers' you mean the Amplify-provided transformers? At this point, yes. VTL is the most cost-effective way but we're looking into the ability to extend it with pipeline resolvers that aren't written in VTL.
Do you plan to maintain backwards compatibility with existing 1st party transformers’ schema outputs ?
We're still working through the migration plan. The goal is to have feature-parity (not necessarily DX-parity) from GraphQL Transformer v1. We are exploring the efforts for "auto-migration" from v1 to v2. The reason for this is, we believe we have an opportunity here to improve on the DX of some of the directives / use cases but will want to maintain feature-parity.
Will the implementation be open to community members, or will major components be developed internally ? If implementation is public, how do you plan to coordinate the effort with community members ?
We'll address this on a case-by-case basis. We plan to publish individual RFCs for different directives and we can work through a collaboration / coordination plan in those specific RFCs.
How will this new approach affect CreatedAt, createdBy, updatedAt, updatedBy properties on the @model transformer ? These should be handled by a data enhancer plugin according to the described RFC . Will this be the case ?
Yes, these would be data enhancer plugins' responsibility.
Will this support union's and interfaces? @GeorgeBellTMH - can you give me an example of your use case where you're using unions & interfaces? What kind of a DDB table setup would you expect given those examples? Let's say if you have an interface, would you expect implementors to be just using a single table or individual tables for the implementors?
Does this support custom auth(not AWS cognito based)? I hope it does since migrating existing auth system to AWS cognito is currently one of the blockers from using Amplify
@renebrandel from my perspective I don't too much care what the ddb ends up looking like - as they say - "it's an implementation detail". Other than it shouldn't break other functionality and it should be performant (and ideally cheap).
My main reason for wanting this functionality is that it simplifies the graphql and allows for improved typing in our front end.
I need to use custom auth logic for all of my queries/mutations. I would like to be able to write a custom auth function (vtl is fine) and have it run by default for all the generated resolvers, without having to specify it each time (that would become very tedious and error-prone). Specifically, my use-case is as follows: I have a ddb table that specifies permissions across roles, eg Admin role has a collection of permissions that include "writePosition" and "readPosition", and HR role includes "readPosition" permission. Each of my users belongs to one or many user groups that define their roles at a particular company, eg "Company1#Admin" or "Company2#HR". Every model would have separate read and write permissions, eg a Position model can have "readPosition" and "writePosition". In addition, the model has a company field, since the object belongs to a specific company. When a user tries to perform an action, eg update a Position, I need to first check that the user belongs to a group where: 1) the company matches the position company he wants to update, 2) and where the role includes the "writePosition" permission. Ideally, I would be able to write an auth middleware type of function and specify that it is a default pre-resolver auth transformer type of thing. It would need to be able to have access to the cognito user groups, possibly his custom attributes, and to read some custom directives on the model (eg the read and write permissions).
I'm not sure this RFC is the right place for this, but it seems somewhat related. In any case, I'm really excited about the direction of this RFC, many thanks to the amplify team!
Looks like the framework is going to gain an evolution from the new architecture 🎉
I'm glad to see the newly added @default
! I'd like to see more "validation rules directive" for server side validation.
Also please consider this feature request aws-amplify/amplify-category-api#210
Tracking cross table authorization with vNext which will support Pipeline resolvers: aws-amplify/amplify-category-api#341
Hi, I'd like to know that are there any milestones for this? Has had this project dead?
I also can't wait for RFC to get implemented.
We are having many issues using Amplify to build multi-tenant authentication and simple business logic that need to happen before the model kicks in.
Is this thread still active?
Please consider implement "Transformer order" too. This is common and significant for a plugin architecture. https://github.com/aws-amplify/amplify-cli/issues/7297
Hi folks, we've started making more progress on this. We've published an RFC now for "@searchable" improvements https://github.com/aws-amplify/amplify-cli/issues/7546
That's great! Is there any traction happening outside @searchable? I am particularly interested in lifecycle plugins and executing custom logic before and after @model entities.
Any status updates on unions & interfaces?
Is there any possibility to support Lambda Authorizers? It's very important for Saas applications...
Hi everyone! Today we’re announcing the PREVIEW of our next generation GraphQL Transformer. This will be the biggest update we’ve shipped to our GraphQL functionality since its original inception in Amplify CLI. We’ve rearchitected the GraphQL capabilities from the ground up to ship new features and updates faster.
DO NOT USE THIS IN PRODUCTION!
npm i -g @aws-amplify/cli@6.1.0-graphql-vnext-dev-preview.2
(make sure to uninstall existing Amplify CLI versions!)amplify init
amplify add api
@allow_public_data_access_with_api_key
directive.@hasOne
, @hasMany
, and @belongsTo
. @connection
will be deprecated.@primaryKey
and @index
. @key
will be deprecated.@hasOne
, @hasmany
, @manytomany
, @belongsto
, @primaryKey
, and @index
@connection
is being replaced with new directives.
@hasOne
creates a “has one” relationship with the target model.
directive @hasOne(fields: [String!]) on FIELD_DEFINITION
@hasMany
creates a "has many" relationship with the target model.
directive @hasMany(indexName: String, fields: [String!], limit: Int = 100) on FIELD_DEFINITION
@belongsTo
facilitates a bi-directionality for a relationship. For example, use the @belongsTo
directive on a “has many”-relationship’s target model to create a relationship back to the source model.
directive @belongsTo(fields: [String!]) on FIELD_DEFINITION
@manyToMany
creates a new relationship cardinality for Amplify’s GraphQL Transformer. Traditionally, customers had to create a “join table” between two models and create hasMany relationships from both models into that join table as a work around for this feature. With the new transformer, customers can just specify a @manyToMany
relationship between the models, Amplify CLI will create the join tables under the hood on behalf of the customer.
type Post @model {
id: ID!
title: String!
editors: [User] @manyToMany(relationName: "PostEditors")
}
type User @model {
id: ID!
username: String!
posts: [Post] @manyToMany(relationName: "PostEditors")
}
@key
directive is being replaced by two new directives.
@primaryKey
on a field to define it as the primary key of a table. Customers can also specify sort key fields via a directive argument.This part isn't clear to me:
Prior to this migration, when customers specified an @auth rule on a field, the authorization is also applied on the model. With the new GraphQL Transformer, @auth rules specified on a field will overwrite all authorization rules defined on the model level. If the some model level authorization rules need to be applied on a field level, you need to explicitly specify it.
By overwrite auth rules specified on the model that means only for that field with the auth rules?
@adam-nygate @benharvie @psmoros @zidingz - here we come! 🎉
This part isn't clear to me:
Prior to this migration, when customers specified an @auth rule on a field, the authorization is also applied on the model. With the new GraphQL Transformer, @auth rules specified on a field will overwrite all authorization rules defined on the model level. If the some model level authorization rules need to be applied on a field level, you need to explicitly specify it.
By overwrite auth rules specified on the model that means only for that field with the auth rules?
The intent here is to create a developer experience where the authorization rules are "explicit" and "what you see is what you get". Maybe the example below can help:
type Test @model @auth(rules: [{ allow: owner }]) {
id: ID!
name: String @auth(rules: [{allow: private }])
description: String @auth(rules: [{allow: private, operations: [ read ] }, {allow: owner}])
}
In this case:
Owners are allowed to:
id
description
but denied to:name
Any signed-in user (private) is allowed to:
name
description
but denied to:id
description
Hi folks - a quick update. We just published a new build!
Install by running npm install -g @aws-amplify/cli@6.1.0-graphql-vnext-dev-preview.1
Change log:
@hasOne
, @hasMany
, @manyToMany
, @belongsTo
, @primaryKey
, and @index
Can we Goto a specific page number now?
This is all lovely to see and read about. Well done all.
Only bit that looked a bit odd is the @index directive, that seems overloaded in my head with indexing into a search engine and what not. Could we not find a more descriptive name to follow on from the hasMany, primaryKey style we have. Index just feels a bit too generic and a catch up, I'd have to keep looking it up to remind myself what it means I think.
Just me? Do I make sense?
This is all lovely to see and read about. Well done all.
Only bit that looked a bit odd is the @Index directive, that seems overloaded in my head with indexing into a search engine and what not. Could we not find a more descriptive name to follow on from the hasMany, primaryKey style we have. Index just feels a bit too generic and a catch up, I'd have to keep looking it up to remind myself what it means I think.
Just me? Do I make sense?
We discussed this quite a bit internally and never felt like we've found a better "word" for it. Do you have another recommendation?
I really wanted to have one, I really did but the trouble is I can't see what is actually doing now so I don't know the context to be able to come up with another word...which sort of proves my point :)
Can you sum up what its doing in a digestible way?
When using owner auth, will we be able to create an @index
on the generated owner
field? I currently need to add the owner
field to the type to be able to do this which also exposes owner
to the clients.
@PeteDuncanson - @index
will create a new access pattern for the particular model. It's "as if" you created a global secondary index on the DDB table. Previously this was done via the @key
directive if you've specified both the fields and a key name. This essentially helps you optimize your data access patterns. Potentially something like @query
(also another overloaded term, could be better?)
@kjones - what's your particular use case where you'd need a GSI on the owner field? Like what access patterns / customer scenario are you trying to achieve? That'll help me better understand how we want to incorporate something built-in by default.
Hi folks! Another update:
Install by running npm install -g @aws-amplify/cli@6.1.0-graphql-vnext-dev-preview.2
.
If you try this in an existing project, you'll need to edit the cli.json
file and set "transformerVersion": 2
under features.graphqltransformer
Change log:
@manyToMany
directive to help you model that easier@renebrandel
It's "as if" you created a global secondary index on the DDB table. Previously this was done via the
@key
directive if you've specified both the fields and a key name. This essentially helps you optimize your data access patterns. Potentially something like@query
(also another overloaded term, could be better?)
What about @lookup
as that to me sounds like what its doing "Lookup this data by this id". When you go use an old school libary (as in books) style card index that is sort of what you would do, "go look it up in the card index"...thoughts?
BTW exciting to see @default
released, keep them coming :)
type Employee
@model
@lookup(by: "name", sortBy: "description")
@searchable {
id: ID!
firstName: String!
lastName: String!
age: String
}
Something like this?
Oh my....that reads beautifully. Yes. Just like that.
Wondered what it would look like with multiple lookups.
type Employee
@model
@lookup(by: "lastName", sortBy: "description")
@lookup(by: "age", sortBy: "description")
@searchable {
id: ID!
firstName: String!
lastName: String!
age: String
description: String
}
Still works.
The by
attribute is the field we are using as the key right? Or is it the name of this index like it used to be? I've changed it to lastName
in my example to match the field name...am I understanding that correctly?
I think @lookup
is a bit confusing. It's a declaration on how to index data into the GSI tables, right? But to me, it looks more like an equivalent to the @connection
directive.
but @connection
(which I don't like currently) is getting binned for the nicer sytaxical sugar of @hasMany
@hasOne
etc. so it wouldn't be in the mix right?
I like that we are moving away from the language of the internals of DyanmoDB to instead make it easier to pick up (I came to AWS because of Amplify so I was a complete newbie to AWS land, this easier language is helping greatly with mapping concepts in my mind).
I think index
is a common term in relational databases. However, I always see the phrase access patterns around the current @key
and the next @index
directives. Whenever I see this phrase query
comes to mind. You are basically creating a way to query your data so I will also be fine if the directive is called @query
as I saw in some previous comments. I am not really sure what an overloaded word or term is.
Here is an example of how it can look like
type Employee @model @query(by: "age", sortBy: ["firstName", "lastName"], name: "listByAge") {
id: ID!
firstName: String!
lastName: String!
age: String
}
The name of the query could be generated or offered if omitted. I personally don't know of a reason to make an index/query/GSI if is not to query data.
While I agree that the language used in amplify needs to be approachable for new devs it also needs to point to the underlying technology.
Index is close to gsi so makes most sense to me.
There are cases where you may want to use an index to make a connection between types but not expose a query to the ui.
@renebrandel great staff!
Regarding:
@manyToMany creates a new relationship cardinality for Amplify’s GraphQL Transformer. Traditionally, customers had to create a “join table” between two models and create hasMany relationships from both models into that join table as a work around for this feature. With the new transformer, customers can just specify a @manyToMany relationship between the models, Amplify CLI will create the join tables under the hood on behalf of the customer.
How will this work, if we want to store extra meta data inside the join table? E.g. date created or status (pending, rejected, approved). WIth this new approach, will there be an option for this?
You'll be able to use our built-in auto-generated fields like createdAt, updatedAt. What's your use case, where you'd like the join table to have more meta data, wouldn't you usually want the meta data on each of the origin tables? Also do you have a proposal for what you'd like that API to look like?
createdAt, updatedAt are great. Regarding our use case, we run a social media application and we are shifting to AWS Neptune slowly, but at this moment we are modeling relations between users and posts in DynamoDB. Example: you can follow a user, but they need to first accept the follow. We store the status (pending, rejected, approved) of this follow in the "join table".
Regarding the proposal for the API, I have not reviewed the details of the new @manytomany
directive. From what I can see from this discussion, is that you can pass the relationName
parameter to it, but I am not sure what this parameter does. Is there any place I could reference to check the details? Also, I am not sure how with the new directive will effect the queries and mutations. E.g. how will you create or delete the connection.
In short it would be great to be able to somehow pass the "join table" model to the @manytomany
.
What about transactWriteItems and batch writes?
Transactions for Dynamo DB were introduced in 2018 (3! years ago). In 2019 https://github.com/aws/aws-appsync-community/issues/14, they were supported by AWS-Appsync, and still, I am not seeing any mentions here.
Same with BatchWrites: aws-amplify/amplify-category-api#443 BatchWrites, just for start, in the same table, should be relatively easy to implement, not? One new @directive, and two VTL resolvers (for request and response) - that we now have to create manually.
@model Item **@withBatch** {
id: ID!
}
Mutation.createItemsBatch.req.vtl and Mutation.createItemsBatch.res.vtl
Resolvers content is already here: https://docs.aws.amazon.com/appsync/latest/devguide/tutorial-dynamodb-batch.html
Then, if you would need to write into different tables with batch, you can do something like :
@model Item @withBatch(queryBatchName: "sameQueryName") { id: ID! }
@model Item2 @withBatch(queryBatchName: "sameQueryName") { id: ID! }
so by using the same queryBatchName you will know, that you need to create just one query, and just one req/res VTL resolver pair.
@renebrandel Some feedback from testing. For context I'm working on a greenfield product which is centered around DataStore. It's a great solution, and I'm excited for the new transformer release to smooth out sharp edges. If you want to discuss that use case in more detail, I'm happy to connect:
@primaryKey
and @index
are on the same field does not work for me. I needed to add @primaryKey.sortKeyFields
which makes sense if you think through the resulting dynamo infra. In this case scenario it would actually make sense to create an LSI and not a GSI but that's low priority.type AMPLIFY_GLOBAL @allow_public_data_access_with_api_key(in: "dev") # FOR TESTING ONLY!
is a nice touch - it should be added to the docs here.amplify add api
is nice.@index
language is perfectly clear to me. I see no reason to change it. Hi @sherl0cks - thanks for all this feedback!
What about transactWriteItems and batch writes?
Transactions for Dynamo DB were introduced in 2018 (3! years ago). In 2019 aws/aws-appsync-community#14, they were supported by AWS-Appsync, and still, I am not seeing any mentions here.
Same with BatchWrites: aws-amplify/amplify-category-api#443 BatchWrites, just for start, in the same table, should be relatively easy to implement, not? One new @directive, and two VTL resolvers (for request and response) - that we now have to create manually.
@model Item **@withBatch** { id: ID! }
Mutation.createItemsBatch.req.vtl and Mutation.createItemsBatch.res.vtl
Resolvers content is already here: https://docs.aws.amazon.com/appsync/latest/devguide/tutorial-dynamodb-batch.html
Then, if you would need to write into different tables with batch, you can do something like :
@model Item @withBatch(queryBatchName: "sameQueryName") { id: ID! }
@model Item2 @withBatch(queryBatchName: "sameQueryName") { id: ID! }
so by using the same queryBatchName you will know, that you need to create just one query, and just one req/res VTL resolver pair.
Thanks for raising the attention on this. It's a theme we're also looking to tackle after the initial release of Transformer v2. I'll see if in the mean time we can at least come up with a guide or something similar to educate customers on how to do it using a custom resolver within Amplify.
Thanks for the quick reply @renebrandel.
- Unions are on our roadmap but will come after the release of the new transformer. It's unfortunately a deceptively difficult problem to support all/most existing features. We'll look into it more after this transformer v2 release.
Understood. Could you please update release notes on the feature requests you intend to support on the v2 roadmap but not in the initial release?
Since we're talking roadmap, a few other feature requests I'd like to share:
_deleted
as a filter option in list
operations. This is particularly useful during testing where you may be regularly deleting items through appsync and you don't want to page through a bunch of deleted items to get new.@versioned
. This is a pretty complex feature and it may even be sufficient to provide example code / blog post.So is this stuff in the current normal amplify cli or do we still do something like mentioned above npm install -g @aws-amplify/cli@6.1.0-graphql-vnext-dev-preview.2
also if its the case that we can use today are there any resources that I can use to help me even though official docs are still for current setup.
Actually I believe this is the latest version now https://www.npmjs.com/package/@aws-amplify/cli/v/6.4.0-graphql-vnext-dev-preview.4
I see some talk on different ways of doing stuff above like @index @lookup @query not sure if there are sketchpad notes on how to do some of stuff available now or if anyone knows some basics. I'm super excited for this I just can't wait but idk what to try will just say like Unknown directive "query" or Unknown directive "index"
Is a release planned for this new GraphQL Transformer?
I see a version 7.3.0-graphql-vnext-dev-preview.12
so how much are we close to the release day?
Problem
The current GraphQL Transformer has tight coupling between transformer plugins and it is not modular.
In an effort to accelerate development, reduce operational load, enable more extensibility and also to meet the rising feature requests and enhancements, we're proposing a new architecture for the GraphQL Transformer moving forward. Refered henceforth as "GraphQL Transformer vNext"
Proposed Solution
The proposed solution will:
@auth
logic can be independent of datasource logic)Architecture
The GraphQL Transformer vNext is split into two main concepts:
The Transformer plugins are categorized as follows:
@model
,@searchable
)@key
,@connection
)@versioned
,@unique
)@auth
)Lifecycle for transformer plugins
To support the inter-dependency between the following classes of transformers, the transformation will be done in a phased manner. These are the lifecycle methods that a transform plugin can implement to handle the different phases in the execution of the transformer.
The transformer plugin execution for each phase would run in the following order
This order ensures all operations and their resolvers are available for non-datasource specific plugins. These plugins will work with in the bounds of existing resolvers and operations.
Resolver Slots
In order for individual transformer plugin to contribute resolver snippets to a resolver, the transformer will generate resolvers that seperate the logic into slots. Based on the type of operation, the amount of slots will differ as follows:
Note: These can be rendered as individual AppSync Functions to be used within Pipeline Resolvers or they can be concatenated to generate a Unit Resolver.
Resolver Runtime Stash
The transformer will include a fixed object within
$ctx.stash
so the transformer plugins can store conditions, authConditions, default values, and transformed values. The datasource provider will use these values from these fields and merge default values, input values, and transformed values to generate the appropriate payload. (e.g. a query will have merged auth and input conditions)The stash will have the following format
Conditions and authConditions will take the shape of the input needed into the
$util.transform
functions.$ctx.stash.metadata
is a generic object that can be used to declare datasource specific information such as index name, keySchema information.The resolver code generated by DataSourceProvider will merge defaultValues, arguments and transformedValues.
@model
ImplementationProgress tracker
Foundational infrastructure of GraphQL Transform vNext
Directives / Use cases
@model
@key
@auth
@connection
@function
@http
@predictions
@searchable
@default
This list is not fully finalized. If you have additional use cases and scenarios you'd like to cover, please provide your comment below. As we get closer to implementation, the directives might change to be more intuitive towards the underlying use cases.
Trying it yourself
⚠️ THE GRAPHQL TRANSFORMER VNEXT IS NOT PRODUCTION-READY YET ⚠️
Currently, the GraphQL Transformer vNext will enable you to use pipeline resolvers and the
@model
directive. It showcases the architecture and provides a reference implementation of a custom directive. You can test this by:amplify/cli.json
useExperimentalPipelinedTransformer
totrue
⚠️ THE GRAPHQL TRANSFORMER VNEXT IS NOT PRODUCTION-READY YET ⚠️