[Discussion] Extrapolate GraphQL from data Schema

[JJK801]

Environment information

System:
  OS: macOS 13.3.1
  CPU: (10) arm64 Apple M2 Pro
  Memory: 92.92 MB / 16.00 GB
  Shell: /bin/zsh
Binaries:
  Node: 20.0.0 - ~/.nvm/versions/node/v20.0.0/bin/node
  Yarn: 1.22.19 - ~/.nvm/versions/node/v20.0.0/bin/yarn
  npm: 9.6.4 - ~/.nvm/versions/node/v20.0.0/bin/npm
  pnpm: undefined - undefined
NPM Packages:
  @aws-amplify/backend: Not Found
  @aws-amplify/backend-cli: Not Found
  aws-amplify: Not Found
  aws-cdk: Not Found
  aws-cdk-lib: Not Found
  typescript: Not Found
AWS environment variables:
  AWS_STS_REGIONAL_ENDPOINTS = regional
  AWS_NODEJS_CONNECTION_REUSE_ENABLED = 1
  AWS_SDK_LOAD_CONFIG = 1
No CDK environment variables

Description

Hi guys 🖖

Today i come with some improvement ideas (i will be back with issues soon 😄 ) about the use of GraphQL in this new amazing Code-First backend.

Root Cause of my tought

I need to use some lambda functions to access data (ideally by using AppSync to take advantage of subscriptions), so after some hours (days) wiring everything (thanks to @edwardfoyle for his help here), i got stuck on handling my GraphQL queries, because the codegen can only run after i deploy my app (so i cant use this feature to make sure my queries are up to date with data).

My first approach was "okay, lets rewrite the queries in my handlers", and, 2 deploys later: "F***, i missed updating my query in the handler", so i figured that it's really not a professional way as we can't cover it with tests or static checks.

Solution

What i did to manage this issue, is taking advantage of the Schema type (only the type, because using the implementation leads to embedding the whole world in my handler) to extrapolate the GraphQL.

It works very well, and each time i miss something it allows to fail the deploy because of typescript complains.

Code

Don't be too attentive to naming and structuration as it's a 1 hour made solution, it probably require some rework and some abstraction layers to be integrated to the amplify ecosystem.

PS: it requires a path to export ResolvedModel which is not exported by default (and it should because it's really useful)

Types

import type { Schema } from '../../data/resource';
import { ResolvedModel } from '@aws-amplify/data-schema-types';

export enum GQLRequestType {
    QUERY = "query",
    MUTATION = "mutation"
}

export enum GQLQueryActionType {
    GET = "Get",
    LIST = "List",
}

export enum GQLMutationActionType {
    CREATE = "Create",
    UPDATE = "Update",
    DELETE = "Delete"
}

type GQLActionFor<RequestType extends GQLRequestType> = {
    [GQLRequestType.QUERY]: GQLQueryActionType;
    [GQLRequestType.MUTATION]: GQLMutationActionType;
}[RequestType]

interface GraphQLResponse<
    Query,
    Data extends Query extends GraphQLQuery<any, infer ResolverName, infer ResultType> ? Record<ResolverName, ResultType> : unknown = Query extends GraphQLQuery<any, infer ResolverName, infer ResultType> ? Record<ResolverName, ResultType> : unknown
> {
    data: Data
}

interface AppSyncHttpBody<
    InputType extends Record<string, any> = Record<string, any>,
    OutputType = any,
    Query extends GraphQLQuery<any, any, OutputType, InputType> = GraphQLQuery<any, any, OutputType, InputType>
> {
    query: Query;
    variables: InputType
}

interface AppSyncLambdaClientConfig {
    awsRegion: string;
    endpointUrl: string;
    assumedRole?: string;
}

//-----------------------------------------
type GraphQLQuery<
    RequestType extends GQLRequestType = GQLRequestType,
    Name extends string = string,
    RawResultType = any,
    ResultType = any,
    Variables extends Record<string, any> = {}
> = {
    __rawResultType: RawResultType;
    __resultType: ResultType;
    __requestType: RequestType;
    __variables: Variables;
    __resolverName: Name;
}

type AppSyncGraphQLResult<
    TypeName extends string,
    DataType
> = {
    readonly __typename: TypeName;
} & {
    readonly [K in keyof DataType]: DataType[K]
}

type AppSyncListGraphQLResult<
    TypeName extends string,
    DataType
> = AppSyncGraphQLResult<
    `Model${TypeName}Connection`,
    {
        items: DataType[],
        nextToken: string
    }
>

type LogicalFilters<
    Type extends Record<string, any>
> = {

    and?: Array<AppSyncConditionVariable<Type> | null> | null,
    not?: AppSyncConditionVariable<Type> | null,
    or?: Array<AppSyncConditionVariable<Type>  | null> | null,
};

type FilterBetween<T> = {
    between?: [T, T];
}

type CommonFilter<T> = {
    attributeExists?: boolean;
    eq?: T;
    ne?: T;
}

type ComparatorFilter<T> = {
    ge?: T;
    gt?: T;
    le?: T;
    lt?: T;
}

type SizeFilter = FilterBetween<number> & ComparatorFilter<number> & {

};

type StringFilter = CommonFilter<string> & FilterBetween<number> & ComparatorFilter<string> & {
    beginsWith?: string;
    contains?: string;
    notContains?: string;
    size?: SizeFilter;
};

type StringKeyFilter = FilterBetween<string> & ComparatorFilter<string> & {
    beginsWith?: string | null,
    eq?: string | null
}

type NumericFilter = CommonFilter<Number> & FilterBetween<number> & ComparatorFilter<number> & {

};

type BooleanFilters = CommonFilter<boolean> & {

};

type Filter<T> = 
    T extends string ? StringFilter :
    T extends number ? NumericFilter :
    T extends boolean ? BooleanFilters :
    unknown

enum SortDirection {
    ASC = "ASC",
    DESC = "DESC",
}

type AppSyncConditionVariable<
    Type extends Record<string, any>
> = LogicalFilters<Type> & {
    [K in keyof Type]?: Filter<Type[K]> | null
}

type AppSyncInputVariable<
    Type extends Record<string, any>
> = {
    [K in keyof Type]?: Type[K] | null
}

type AppSyncQueryVariables<
    Type extends Record<string, any>,
    IDFields extends keyof Type
> = {
    [K in IDFields]: Type[K]
}

type AppSyncListGraphQLQueryVariables<
    Type extends Record<string, any>,
    IDFields extends keyof Type
> = {
    filter?: AppSyncConditionVariable<Type> | null,
    limit?: number | null,
    nextToken?: string | null,
    sortDirection?: SortDirection | null,
} 
// & {
//     [K in IDFields]?: StringKeyFilter | null
// }

type AppSyncCreateGraphQLMutationVariables<
    Type extends Record<string, any>
> = {
    condition?: AppSyncConditionVariable<Type> | null,
    input: AppSyncInputVariable<Type>,
}

type AppSyncGraphQLRequest<
    RequestType extends GQLRequestType,
    Action extends GQLActionFor<RequestType>,
    Name extends string,
    RawResultType,
    ResultType,
    Variables extends Record<string, any> = {}
> = GraphQLQuery<
    RequestType,
    Name,
    RawResultType,
    ResultType,
    Variables
> & {
    __action: Action
}

type AppSyncListGraphQLQuery<
    Name extends string,
    TypeName extends string,
    Type extends Record<string, any>,
    IDFields extends keyof Type
> = AppSyncGraphQLRequest<
    GQLRequestType.QUERY,
    GQLQueryActionType.LIST,
    Name,
    Type,
    AppSyncListGraphQLResult<TypeName, Type>,
    AppSyncListGraphQLQueryVariables<Type, IDFields>
>

type AppSyncCreateGraphQLMutation<
    Name extends string,
    TypeName extends string,
    Type extends Record<string, any>
> = AppSyncGraphQLRequest<
    GQLRequestType.MUTATION,
    GQLMutationActionType.CREATE,
    Name,
    Type,
    AppSyncGraphQLResult<TypeName, Type>,
    AppSyncCreateGraphQLMutationVariables<Type>
>

type AppSyncUpdateGraphQLMutation<
    Name extends string,
    TypeName extends string,
    Type extends Record<string, any>
> = AppSyncGraphQLRequest<
    GQLRequestType.MUTATION,
    GQLMutationActionType.UPDATE,
    Name,
    Type,
    AppSyncGraphQLResult<TypeName, Type>,
    AppSyncCreateGraphQLMutationVariables<Type>
>

type AmplifyListQuery<
    Name extends string,
    TypeName extends Extract<keyof Schema, string>,
    Schema extends Record<string, {}>
> = AppSyncListGraphQLQuery<
    Name,
    TypeName,
    ResolvedModel<Schema[TypeName]>,
    TypeID<TypeName, Schema>
>

type TypeID<
    TypeName extends Extract<keyof Schema, string>,
    Schema extends Record<string, {}>,
    MetaKey extends Exclude<keyof Schema, number|string> = Exclude<keyof Schema, number|string>
> = TypeName extends keyof Schema[MetaKey] ? 
    Schema[MetaKey][TypeName] extends keyof Schema[TypeName] ? 
        Schema[MetaKey][TypeName] : 
        keyof Schema[TypeName]: 
    keyof Schema[TypeName]

type AmplifyCreateMutation<
    Name extends string,
    TypeName extends Extract<keyof Schema, string>,
    Schema extends Record<string, {}>
> = AppSyncCreateGraphQLMutation<
    Name,
    TypeName,
    ResolvedModel<Schema[TypeName]>
>

type AmplifyUpdateMutation<
    Name extends string,
    TypeName extends Extract<keyof Schema, string>,
    Schema extends Record<string, {}>
> = AppSyncUpdateGraphQLMutation<
    Name,
    TypeName,
    ResolvedModel<Schema[TypeName]>
>

Queries defintitions

We probably can compute all the model's queries automatically with a type, but for now i handwrite only those i need.

export type ListCompaniesQuery = AmplifyListQuery<
    "listCompanies",
    'Company',
    Schema
>

export type ListSubscribedPlansQuery = AmplifyListQuery<
    "listSubscribedPlans",
    'SubscribedPlan',
    Schema
>

export type ListPeopleQuery = AmplifyListQuery<
    "listPeople",
    'Person',
    Schema
>

export type ListProfilesQuery = AmplifyListQuery<
    "listProfiles",
    'Profile',
    Schema
>

export type CreateSubscribedPlanQuery = AmplifyCreateMutation<
    "createSubscribedPlan",
    'SubscribedPlan',
    Schema
>

export type UpdateCompanyMembershipQuery = AmplifyUpdateMutation<
    "updateCompanyMembership",
    'CompanyMembership',
    Schema
>

export type UpdateSubscribedPlanQuery = AmplifyUpdateMutation<
    "updateSubscribedPlan",
    'SubscribedPlan',
    Schema
>

export type UpdateProfileQuery = AmplifyUpdateMutation<
    "updateProfile",
    'Profile',
    Schema
>

type Queries =  ListCompaniesQuery | ListSubscribedPlansQuery | ListPeopleQuery | ListProfilesQuery |
                CreateSubscribedPlanQuery | 
                UpdateCompanyMembershipQuery | UpdateSubscribedPlanQuery | UpdateProfileQuery

The client / Query builder

import type { Schema } from '../../data/resource';
import { ModelPath, ResolvedModel, SelectionSet } from '@aws-amplify/data-schema-types';

type WithoutWildcard<T extends string> = T extends `${string}.*` ? never : T

const set = (obj: any, path: any, val: any) => {
    const keys = path.split(".");
    const lastKey = keys.pop();
    const lastObj = keys.reduce((obj: any, key: any) => obj[key] = obj[key] || {}, obj);
    lastObj[lastKey] = val;

    return obj;
};

interface FieldSet {
    [K: string]: FieldSet|undefined
}

const flattenGQLFields = (fields: FieldSet) => Object.entries(fields).map(([k, v]): string => {
    if (v === undefined) {
        return k;
    }

    return `${k} { ${flattenGQLFields(v)} }`
}).join("\n")

const guessGQLType = (modelName: string, action: string, varName: string) => {
    switch(varName) {
        case 'input':
            return `${action}${modelName}Input!`;
        case 'condition':
            return `Model${modelName}Input`;
        case 'filter':
            return `Model${modelName}FilterInput`;
        case 'limit':
            return 'Int';
        case 'nextToken':
            return 'String';
        case 'sortDirection':
            return 'ModelSortDirection';
    }
}

export class AppSyncLambdaClient {
    protected readonly endpoint: URL;
    protected readonly stsClient: STSClient;

    protected _signer?: SignatureV4;

    constructor (
        protected readonly config: AppSyncLambdaClientConfig
    ) {
        this.endpoint = new URL(this.config.endpointUrl);
        this.stsClient = new STSClient({ region: this.config.awsRegion })
    }

    async getCredentials () {
        if (this.config.assumedRole) {
            const { Credentials } = await this.stsClient.send(new AssumeRoleCommand({
                RoleArn: this.config.assumedRole,
                RoleSessionName: 'amplify_appsync_superuser'
            }))

            return {
                accessKeyId: Credentials!.AccessKeyId!,
                secretAccessKey: Credentials!.SecretAccessKey!,
                sessionToken: Credentials!.SessionToken!
            }
        }

        return defaultProvider()
    }

    async getSigner () {
        if (!this._signer) {
            this._signer = new SignatureV4({
                credentials: await this.getCredentials(),
                region: this.config.awsRegion,
                service: 'appsync',
                sha256: crypto.Sha256
            });
        }

        return this._signer;
    }

    async getSignedHttpRequest(body: AppSyncHttpBody): ReturnType<SignatureV4['sign']> {
        const signer = await this.getSigner();

        const requestToBeSigned = new HttpRequest({
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
                host: this.endpoint.host
            },
            hostname: this.endpoint.host,
            body: JSON.stringify(body),
            path: this.endpoint.pathname
        });

        return signer.sign(requestToBeSigned)
    }

    async request<Query extends GraphQLQuery, OutputType>(body: AppSyncHttpBody<any, OutputType, Query>) {
        const request = new Request(
            this.endpoint,
            await this.getSignedHttpRequest(body)
        );

        return fetch(request).then((response) => response.json()) as Promise<GraphQLResponse<Query>>
    }

    static buildRequest<
        ModelName extends Extract<keyof Schema, string>,
        Fields extends WithoutWildcard<ModelPath<ResolvedModel<ResultType>>>[],
        FilteredResultType extends SelectionSet<ResultType, Fields>,
        Query extends Extract<Queries, { __requestType: RequestType, __rawResultType: ResolvedModel<Schema[ModelName]>, __action: Action }>,
        RequestType extends Extract<Queries, { __rawResultType: ResolvedModel<Schema[ModelName]> }>['__requestType'],
        Action extends Extract<Queries, { __requestType: RequestType, __rawResultType: ResolvedModel<Schema[ModelName]> }>['__action'],
        ResultType extends Query['__resultType'],
        ResolverName extends Query['__resolverName']
    >(modelName: ModelName, requestType: RequestType, action: Action, {
        resolverName,
        variables,
        fields
    }: {
        resolverName: ResolverName,
        variables: Extract<Queries, { __requestType: RequestType, __resultType: ResultType, __resolverName: ResolverName }>['__variables'],
        fields: Fields
    }) {
        const name = `${action}${modelName}`;
        const tokens = Object.keys(variables).map((k) => {
            return ["$", k, ": ", guessGQLType(modelName, action, k)].join('')
        }).join("\n");
        const filters = Object.keys(variables).map((k) => [k, ": ", "$", k].join('')).join(", \n");
        const expandedFields = fields.reduce((acc, path) => {
            return set(acc, path, undefined)
        }, {})

        const query = /* GraphQL */`
            ${requestType} ${name}(
                ${tokens}
            ) {
                ${resolverName}(
                    ${filters}
                ) {
                    ${flattenGQLFields(expandedFields)}
                }
            }
        `.trim() as unknown as GraphQLQuery<RequestType, ResolverName, FilteredResultType, typeof variables>

        return {
            query,
            variables
        }
    }
}

Usage

In my handlers, i use like bellow

const {
    AMPLIFY_GRAPHQL_ENDPOINT_URL,
    AWS_REGION,
    APPSYNC_ROLE_ARN
} = process.env

const appSyncClient = new AppSyncLambdaClient({
    awsRegion: AWS_REGION!,
    endpointUrl: AMPLIFY_GRAPHQL_ENDPOINT_URL,
    assumedRole: APPSYNC_ROLE_ARN
})

export const handler = async (
    {
        detail: {
            data: {
                object: {
                    id,
                    status,
                    current_period_end,
                    metadata: {
                        userId,
                        companyCountryCode,
                        companyLegalId
                    }
                }
            }
        }
    }: EventBridgeEvent<'StripeEvent', Stripe.CustomerSubscriptionUpdatedEvent>
) => {
    const existingSubscribedPlanResult = await appSyncClient.request(
        AppSyncLambdaClient.buildRequest('SubscribedPlan', GQLRequestType.QUERY, GQLQueryActionType.LIST, {
            resolverName: 'listSubscribedPlans',
            variables: {
                filter: {
                    providerId: {
                        eq: id
                    },
                    provider: {
                        eq: "stripe"
                    }
                }
            },
            fields: [
                'items.id',
                'items.expiresAt',
            ]
        })
    );

   // [...] the handler buisness logic
}

[renebrandel]

Hi @JJK801 - first off, WOW! Thanks for putting all this together! Really LOVE the details and the overall workflows. I've got a few follow-up questions:

1. One thing we're exploring is to offer the generateClient operation within a Node.js environment. That way you'd get the same typing end-to-end no matter what JS environment you run in (SSR function, Node.js, or Browser). Would that also adders your use case?
2. Are you heavily opinionated on the GraphQL-style syntax, is there something you woludn't achieve if we just offered what is outlined in (1.)?

[JJK801]

Hi @renebrandel,

Thanks for your reply.

Yes, the solution mentioned in 1 is exactly what i aim to solve, but it must be enough configurable to issue the exact query we expect (so that it match all use cases).

My opinion is that an ideal solution must be "multi staged", i mean there should be many abstraction layers from the more verbose (GraphQL oriented) to the smallest code lines (using syntaxic sugars), so that we can plug in the stage we need to achieve what we want to do (like CDK does) without being limited by the framework.

In my case, i have to deal with many limitations due to the define functions (which are great to quickly bootstrap an app, but are a pain when we want to integrate custom resources), and i had to reproduce some features (like secret management, which is very usefull), because i can't plug on the logic behind those high level functions and the public API is limited.

In the example above, the builder is very simple is sense of code, because the only thing it handles is formatting the GQL query, anything else is done via the user input, which is driven by the Typescript shape. This is lightweight in the end (once compiled in the lambda function) and it secures developers with the TSC validating that everything is OK, and that's barely what we expect (more than writing less code)

PS: And with the TS-First solution, writing the query is not a pain because Typescript autocompletes almost everything.

[renebrandel]

Hi @JJK801 - would you be down to hop on a call sometime? I'd love to discuss this more and brainstorm on the DX. If you're open to it, can you DM me on Twitter? https://x.com/renebrandel

[renebrandel]

Talking to @JJK801. We're brainstorming on an ideal DX for running Amplify in Lambda:

1. CDK-created function shouldn't need to pass in the explicit Role ARN into the data/resource's defineData function. This could eliminate a circular dependency issue. Generally the IAM permissions should be governed by IAM, not another allowAdminRoleNames property on defineData. Similarly linked: https://github.com/aws-amplify/amplify-backend/issues/1043
2. We need an ability to easily pass in the Amplify/AppSync endpoint config and the GraphQL config to the Lambda function. The challenge today, is that the experience is that the graphql client code helpers get generated after deployment, so it needs technically "two deployments". One way to get around this is to have a Lambda layer that has the shared configuration across or pull the config from the codegen asset bucket that we generate today. Potentially to get fully typed GraphQL queries (without the client.models.) notation, we could look into leveraging the good work of https://gql-tada.0no.co/
3. GenerateNodeClient in the server should ideally use the Lambda function's IAM execution role by default. Customers shouldn't need to manually write a signer or pass in the AWS credentials from the customer code. Customers can always override this behavior and use all Amplify-supported authorization modes.

import { generateNodeClient } from 'aws-amplify/adapter-nodejs'
import type { Schema } from '../../data/resource'

const client = generateNodeClient<Schema>(process.env.AMPLIFY_CONFIGURATION)
// Leverage the existing `models.` notation for fully typed experience.
client.models.Todo.create()

// Execute raw GraphQL queries that are fully typed
client.graphql({
  query: /*GraphQL*/ `query ...` // ideally this is a fully typed string
  variables: {
     filters: {} // the variables and filters should be type matched to the query
  }
}

[edwardfoyle]

There's an initial DX for this outlined here: https://github.com/aws-amplify/docs/pull/7029 But it still needs more work to be a production-ready solution

[ideen1]

@edwardfoyle The initial DX mentions needing to import @env/. However this module is not an NPM. Could you elaborate on where we would generate this module? There’s also mentions to SECRET_KEYS to access the API, do these need to be generated manually right now? Maybe I can submit a PR to the docs to elaborate on this a bit because it is an little unclear right now. Thanks!

[ykethan]

transferring this to our API repository.

[AnilMaktala]

Hi @JJK801, We are marking this as a feature request for the team to evaluate further.