Contributors experience - adding new resources

[ainvoner]

In our current repositories state, adding a new resource will require the contributor to update the following repositories:

1. wingsdk
2. wingsdk-client
3. wing-local (server/client)
4. wing-console application

Our goal is to improve our contributors experience and to reduce the complexity (f.e: the amount of repositories that need to be changed) for adding new resources.

How is the local Bucket implementation looks like today?

1. wingsdk-client: local implementation of the resource methods endpoints using wing-local client (not creating the resource object itself - which is an important note)
2. wingsdk: defines the Bucket schema values and instantiate the bucket client
3. wing-local (server/client): implements the Bucket local behavior. Creates a bucket object, define the routes for each bucket method (put/get) and implement those methods locally (writing and deleting files form disk)
4. wing-console: renders a bucket UI with its' supported actions so the user will be able to put/get object to/from the bucket. Show list of items currently in the bucket etc...

In the following discussions we should try and find a different architecture for our sdk -> local experience that will make contribution much easier.

[ainvoner]

A very high level idea for reducing the amount of involved repositories for each new resource addition:

1. merge wingsdk and wingsdk-client repos.
2. move resource construction implementation from wing-local to the sdk

   a. the wx compressed file will contain not only the inflight code of functions, but also the construction code for each resource. b. the schema for each resource will contain the path to a js file and the construction function name c. wing-local will require the relevant file and execute the construction function (we will need a predefined convention for the resource route /:resourceType/:resourceId for example)

3. REALY NOT SURE ABOUT THIS ONE: If we would like to reduce complexity in adding new resources to wing-console as well, we can agree on a unified UI for all resources where only the supported actions changes according the provided schema (which also provide the endpoint etc...)

bottom line - contributors will need to know the wingsdk repo, the schema structure shared between the wing-local server and the sdk and the resources routes endpoints conventions.

I know it is not a fully baked solution, but I believe it will get us moving/thinking at the right direction of simplifying our contributors lives

@ShaiBer @eladb @skyrpex @Chriscbr please add your comments

[ShaiBer]

Thanks @ainvoner! Yes, I think that we can do a merge the code of the wingsdk, wingsdk-clients and the local server for each resource into one repo and then have the simulator code only deal with provisioning of these resources and running their code without knowing them in advance. In such a solution, contributing a new resource would require the creation of only one PR in one repo. Now the question is - what repos do we use?

I think that the process for adding external resources (that are not part of the SDK core resources) should be the same as for the SDK core resources in order to make sure we dog food the process we want external contributors to go through.

There are a few options to achieve this:

1. We have one repo per resource (doesn't matter if they are core SDK or not). When we package the SDK we select the core resource and include them in a single package
2. We use a monorepo internally and then each resource has its own workspace in it. External resources have the exact same file structure, only they have their own repos each and they are packages separately from the core and from each other.
3. We have one monorepo for the code SDK resources and all the rest. Some of the resources are packaged together as the core SDK and the rest are packages separately as extras.

I lean towards monadahq/wingsdk#2 as it gives us the best of both worlds - we have very similar processes and file structures internally and externally, but we avoid the pains of having too big monorepos. On the other hand, we avoid having separate repos for each resource of the core SDK when we package them together - so we keep the rule of having one repo per packed library.

[eladb]

I love the idea of working backwards from the contributor experience.

As @ekeren suggested, we should actually define the "anatomy" of a general case of a wing resource library, and use this anatomy for our sdk.

This is a complex topic and requires some research and prototyping. There are constraints in jsii and in how monorepos work and I think this can't be decided with just a GitHub issue discussion.

Let's kick off an RFC for this, start with defining the ideal experience of a 3rd party library author and work backwards from there to define the mechanics.

[Chriscbr]

Yeah, this feels like an important part of our design to get right, especially once we want to get contributors on board. Thanks for raising this issue @ainvoner.

Combining different packages or using monorepos are definitely all options I could see being on the table, but I agree it's hard to understand what exactly are the constraints that we should focus on without being aligned on the end goal. I'd add on to @eladb and suggest that we should work backwards from both the contributor experience and the library author experience -- and I think the latter will require us to consider how packages are going to be consumed.


For example -- consider that the Wing SDK, or at least the resource library portion, might someday be re-written in Wing (or at least in theory it could - right?). What would the format of a Wing library be? Is a Wing library an npm package? Or is it something else? I'm really curious what other people think about this / if anyone has already given thought to this :)

[eladb]

@Chriscbr wrote:

Consider that the Wing SDK, or at least the resource library portion, might someday be re-written in Wing (or at least in theory it could - right?). What would the format of a Wing library be? Is a Wing library an npm package?

As far as I see it, Wing libraries are npm packages. Preflight is JSII and inflight is TS. But we don't have a requirement for MVP to support Wing libraries, so let's focus on the TS experience first. I think if we nail the TS experience it will inform the Wing package format basically.

[eladb]

Like always, it will be useful to start by collecting the requirements. Here is an initial list on top of my head:

Contents (per resource):

1. Resource polycon (abstract API) (e.g. cloud.Bucket) (P0).
2. Polycon implementations for 0 or more cloud targets (simulator is just another target) (P0)
3. Resource clients for each target in TypeScript (P0)
4. Resource clients for each target in other languages (P2).
5. Simulator resource implementation (server-side) (we don't need an additional client here I believe - this is basically the same client as 3/4 above).
6. Tests that can run on all targets
7. Wing Console UI component (P2)

Non-functional:

8. Multiple resources (e.g. the Wing SDK is a project with multiple resources) (P0)
9. Publishing workflows to package managers (conventional commits version bumps, release notes, etc) (P0).
10. Preflight (constructs) should be compiled using JSII and published to all package managers
11. Inflight (clients) is hand-written for each language and published to relevant package manager (P0 is for JS/NPM, P2 is for other languages)
12. Generate API documentation for all languages (P0)
13. Breaking change protection (P0 is for preflight API, P1 is for inflight API)
14. Ability to upgrade build & publishing logic as our project evolves (implies projen or projen-like tooling) (P0).
15. Automatic dependency upgrades (P0).
16. Workflows that run tests on all cloud providers.

[ekeren]

I would love if we can address this issue like any other part of our product surface area, and as mentioned before by @eladb I believe that the problem we need to optimize for is - It is easy to develop a resource for wing SDK.

Off course, the above statement is not enough... There is a discussion that needs to happen in order for us to place a bet on what we want to optimize for.

I suggest we assign a developer that will own the RFC, and I'll work with him on defining the problem what should our solution optimize for, after that we will setup a discussion and once there is alignment suggest a solution

[skyrpex]

This is helpful to me, so maybe it'll helpful to you too. Here's the current code needed through all of our packages in order to implement a resource.

wingsdk

```ts // cloud/bucket.ts export interface IBucket extends IResource {} export const BUCKET_ID = "wingsdk.cloud.Bucket"; export interface BucketProps { /** * Whether objects in the bucket are publicly accessible. * @default false */ readonly public?: boolean; } export abstract class BucketBase extends Resource implements IBucket { public readonly stateful = true; constructor(scope: Construct, id: string, props: BucketProps) { super(scope, id); if (!scope) { return; } props; } public abstract capture(consumer: any, capture: Capture): Code; } export class Bucket extends BucketBase { constructor(scope: Construct, id: string, props: BucketProps = {}) { super(null as any, id, props); return Polycons.newInstance(BUCKET_ID, scope, id, props) as Bucket; } public capture(_consumer: any, _capture: Capture): Code { throw new Error("Method not implemented."); } } // tf-aws/bucket.ts import { s3 } from "@cdktf/provider-aws"; export class Bucket extends cloud.BucketBase implements cloud.IBucket { private readonly bucket: s3.S3Bucket; private readonly public: boolean; constructor(scope: Construct, id: string, props: cloud.BucketProps) { super(scope, id, props); this.public = props.public ?? false; this.bucket = new s3.S3Bucket(this, "Default"); // best practice: (at-rest) data encryption with Amazon S3-managed keys new s3.S3BucketServerSideEncryptionConfigurationA(this, "Encryption", { bucket: this.bucket.bucket, rule: [ { applyServerSideEncryptionByDefault: { sseAlgorithm: "AES256", }, }, ], }); if (this.public) { const policy = { Version: "2012-10-17", Statement: [ { Effect: "Allow", Principal: "*", Action: ["s3:GetObject"], Resource: [`${this.bucket.arn}/*`], }, ], }; new s3.S3BucketPolicy(this, "PublicPolicy", { bucket: this.bucket.bucket, policy: JSON.stringify(policy), }); } else { new s3.S3BucketPublicAccessBlock(this, "PublicAccessBlock", { bucket: this.bucket.bucket, blockPublicAcls: true, blockPublicPolicy: true, ignorePublicAcls: true, restrictPublicBuckets: true, }); } } public capture(consumer: any, capture: Capture): Code { if (!(consumer instanceof Function)) { throw new Error("buckets can only be captured by tfaws.Function for now"); } const name = `BUCKET_NAME__${this.node.id}`; const methods = new Set(capture.methods ?? []); if (methods.has("put")) { consumer.addPolicyStatements({ effect: "Allow", action: ["s3:PutObject*", "s3:Abort*"], resource: [`${this.bucket.arn}`, `${this.bucket.arn}/*`], }); } if (methods.has("get")) { consumer.addPolicyStatements({ effect: "Allow", action: ["s3:GetObject*", "s3:GetBucket*", "s3:List*"], resource: [`${this.bucket.arn}`, `${this.bucket.arn}/*`], }); } // The bucket name needs to be passed through an environment variable since // it may not be resolved until deployment time. consumer.addEnvironment(name, this.bucket.bucket); return NodeJsCode.fromInline( `new (require("${CLIENTS_PACKAGE_PATH}")).aws.BucketClient(process.env["${name}"]);` ); } } // local/bucket.ts export class Bucket extends cloud.BucketBase implements cloud.IBucket, IResource { private readonly public: boolean; constructor(scope: Construct, id: string, props: cloud.BucketProps) { super(scope, id, props); this.public = props.public ?? false; } /** @internal */ public _toResourceSchema(): BucketSchema { return { id: this.node.id, type: "cloud.Bucket", path: this.node.path, props: { public: this.public, }, callers: [], callees: [], }; } public capture(consumer: any, _capture: Capture): Code { if (!(consumer instanceof Function)) { throw new Error("buckets can only be captured by a function for now"); } return NodeJsCode.fromInline( `new (require("${CLIENTS_PACKAGE_PATH}")).local.BucketClient("${this.node.id}");` ); } } ```

wingsdk-clients

This package is basically translating specific SDK clients to a common interface that Wing SDK understands. ```ts // bucket-interface export interface IBucketClient { putObject(key: string, value: string): Promise; } // aws-bucket import { GetObjectCommand, PutObjectCommand, S3Client, } from "@aws-sdk/client-s3"; export class AwsBucketClient implements IBucketClient { constructor( private readonly bucketName: string, private readonly client: S3Client, ) {} public async putObject(key: string, value: string): Promise { const command = new PutObjectCommand({ Bucket: this.bucketName, Key: key, Body: body, }); await this.client.send(command); } } // local-bucket import { LocalClient } from "@monadahq/wing-local-client"; export class LocalBucketClient implements IBucketClient { constructor( private readonly resourceId: string, private readonly client: LocalClient, ) {} public async putObject(key: string, value: string): Promise { await this.client.mutation("bucket.PutObject", { resourceId: this.resourceId, key, value, }); } } ```

wing-local-server

```ts // server routes (note that the client is automatically generated from the server routes) export function createBucketRouter(repository: Repository) { return trpc .router() .mutation("bucket.PutObject", { input: z.object({ resourceId: z.string(), key: z.string(), value: z.string(), }), async resolve({ input }) { const bucket = repository.findBucket(input.resourceId); await bucket.putObject(input.key, input.value); }, }); // implementation export interface BucketResource { putObject(key: string, value: string): Promise; } export interface CreateBucketResourceOptions { bucketDir: string; } export function createBucketResource({ bucketDir, }: CreateBucketResourceOptions): BucketResource { return { async putObject(key, value) { const path = filenamify(key); await fs.writeFile(`${bucketDir}/${path}`, value); }, }; } ```

TLDR: In order to create a Bucket polycon, one must:

• Create a unique ID for the resource type
• Define the bucket props BucketProps
• Define the bucket construct interface IBucket
• Define the bucket inflight interface IBucketClient
• cdktf: Implement the construct and inflight interfaces AwsBucket, AwsBucketClient
  • Must understand the relationship with other cdktf resources (eg, use instanceof to cast consumers like AwsFunction)
• local: Define the bucket JSON schema, the server API and implement the local behavior

[ekeren]

Very helpful, thanks @skyrpex

BTW 1

learned I can do this in readme 🤯

BTW 2

learned about Constructor Assignment in tsc 🤯

[eladb]

@skyrpex what about the Wing SDK constructs (polycons and their targets)?

[skyrpex]

@eladb yeah, I'm missing a bunch... Will try to complete it now.

[skyrpex]

I think that https://github.com/monadahq/winglang/issues/174 is complete now. Also, added a TLDR in there.

I'm not sure about this (I'm still thinking), but maybe the overall process of creating a resource would feel somewhat simpler if wing compiled a tree.json (along with the bundled functions) and every specific implementation (eg, cdktf, local, etc) would use that schema. That would mean that every construct would only have to offer an interface to build their schema.

So, for a bucket polycon it would look like this:

// This schema will end in the tree.json file.
interface BucketSchema {
    // Must be unique!
    type: "cloud.Bucket";
    // Whether the bucket is public.
    public: boolean;
    // The function handlers that listen to bucket events.
    handlers: string[];
}

interface BucketProps {
    public?: boolean;
}

interface IBucket extends IResource {
    setPublic(public: boolean): void;
    addEventHandler(handler: IFunction): void;
}

// Every resource must have a `toJSON()` method that returns the schema JSON.
class Bucket extends Resource<BucketSchema> implements IBucket {
    private public: boolean;

    private handlers: string[] = [];

    constructor(scope, id, props?: BucketProps) {
        this.public = props?.public ?? false;
    }

    setPublic(public: boolean) {
        this.public = public;
    }

    addEventHandler(handler: IFunction) {
        this.handlers.push(handler.id);
    }

    toJSON(): BucketSchema {
        return {
            public: this.public,
            handlers: this.handlers,
        };
    }
}

interface IBucketClient {
    putObject(key: string, value: string): Promise<void>;
}

Then, the CDKTF and local implementations would be elsewhere.

[Chriscbr]

@skyrpex Interesting, I really like the proposal! 💪 So this would create a single format/schema which could be run on any cloud?

What would the experience look like for the end user? I imagine we could provide at least a few options:

1.

1. wingc hello.w -o hello.wx - the user "wing compiles" the application into a .wx (Wing assembly?) file.
2. wingl hello.wx -o cdktf.out --target aws - the user "wing launches" the wing assembly into a target cloud format.
   2. wing hello.w -o cdktf.out --target aws - the user runs a single command to compile their application into a target cloud format, performing both steps together.

We could start with just letting users do (2), but then allow users to run the steps individually as described in (1) to enable additional portability and use cases perhaps.

The main thing I want to call out is that the second step (compiling from an intermediate format into a target cloud format) is always necessary, even if you want to run the application locally with the cloud simulator. This is because depending on the target cloud, the Wing SDK needs to bundle your code with different capture clients and different dependencies (e.g. AWS SDK or Wing Local SDK...).

Maybe it would be a good idea to invest in this re-architecture after our MVP?

[eladb]

Blasphemous thoughts related to repositories: I know that I've been the main proponent of avoiding mono-repos, but I am slowly turning around because I've realized that there are important business/marketing considerations around consolidating our efforts and our community around a single GitHub repository due to how GitHub works.

We will invest a lot of energy and money in directing traffic to wing, and if we have multiple repositories, it will be harder to gain critical mass. For example, stars are collected at the repository level, statistics around engagement and contributions, and generally a sense of activity and liveness.

I was also thinking that maintaining multiple release lines (winglang, wingsdk, simulator, more?) from a versioning and breaking changes compatibility perspective, will be expensive initially since the boundaries between the projects will undergo a lot of changes.

All in all, my conclusion (yes, I know I am late to the game) is that I think we should consolidate winglang, wingsdk and the simulator into a single repository (winglang) and set it up with a modern monorepo tool so that it works great for everyone.

Curious what people think...

[staycoolcall911]

I agree. Heard good things about Nx monorepo build system in the past, but I have no experience with it or other similar tools.

[ShaiBer]

I also agree that the benefits outweigh the drawbacks. The industry seems to be moving in the Monorepo direction, with strong support from GitHub, npm, and others. So I'm sure we'll find the right tools to fit our needs (which at the moment are not too complex). But we should do a more thorough investigation of this before we decide and commit on anything because it might be very difficult to change it once we have a community set up.

On Mon, Sep 5, 2022, 16:20 Uri Bar @.***> wrote:

I agree. Heard good things about Nx monorepo build system https://github.com/nrwl/nx in the past, but I have no experience with it or other similar tools.

— Reply to this email directly, view it on GitHub https://github.com/monadahq/winglang/issues/174, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANGGYAGZXBWZYPCHMZI5LTV4XXTXANCNFSM573EGNYA . You are receiving this because you were mentioned.Message ID: @.***>

[Chriscbr]

We have moved all packages to a monorepo. Now, contributors wishing to add a new resource to the Wing SDK will only need to make a single pull request, and all of the code will go in the wingsdk package - not to multiple packages.

The Wing console will stay in a separate repo, and any information needed to display the resources in the console should eventually be produced by the Wing SDK. Or, as @ainvoner called out, we can design a unified UI later, and for now keep all resource specific details in the console separately.

I've added comments in bold to the list of features/requirements with details about which features have been implemented or which I've added issues to track:

1. Resource polycon (abstract API) (e.g. cloud.Bucket) (P0). (implemented)
2. Polycon implementations for 0 or more cloud targets (simulator is just another target) (P0) (implemented)
3. Resource clients for each target in TypeScript (P0) (implemented)
4. Resource clients for each target in other languages (P2). (tracked in #204)
5. Simulator resource implementation (server-side) (we don't need an additional client here I believe - this is basically the same client as 3/4 above). (implementation started in #192)
6. Tests that can run on all targets (see #205)
7. Wing Console UI component (P2) (in progress)
8. Multiple resources (e.g. the Wing SDK is a project with multiple resources) (P0) (implemented)
9. Publishing workflows to package managers (conventional commits version bumps, release notes, etc) (P0). (in progress)
10. Preflight (constructs) should be compiled using JSII and published to all package managers (tracked in #200, #201, #202, #203)
11. Inflight (clients) is hand-written for each language and published to relevant package manager (P0 is for JS/NPM, P2 is for other languages) (tracked in #204)
12. Generate API documentation for all languages (P0) (tracked in #204)
13. Breaking change protection (P0 is for preflight API, P1 is for inflight API) (tracked in #205)
14. Ability to upgrade build & publishing logic as our project evolves (implies projen or projen-like tooling) (P0). (in progress - we will be adding a monorepo tool like NX etc. soon)
15. Automatic dependency upgrades (P0). (tracked in #206)
16. Workflows that run tests on all cloud providers. (tracked in #205)

[eladb]

@Chriscbr this is great. @mbonig I think we should include these instructions in the contribution guide, no?

[Chriscbr]

Seems like this discussion has diverged into several directions. I'll close this for housekeeping, but let's open new issues if there are any unhandled threads here.