Discuss the specs for bundled cloud functions

[skyrpex]

The exports for functions bundled by wingsdk

We need to define the format of the functions bundled by wingsdk and put it in common with wing-local. More specifically, we need to define what's the entry name that will be exported, and what kind of object will it expect when being called.

We currently assume that a wingsdk-bundled function will look like this:

// Exports a "handler", expects a single "event" object
exports.handler = async function (event) {
    // returns JSON object (or a string, which happens to be valid json too)
    return "hello world"; // return { hello: "world" };
}

Does it make sense?

Different event types for functions

Depending on where does the function gets called from, the event object will be different. For example:

• Direct call: the event object will be a plain JSON object, or undefined
• From endpoint: the event object will contain the HTTP method, the headers, etc
• From queue: the event object will contain the message body, etc

That's what happens in AWS lambdas, at least. Here's some inspiration:

// See https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/aws-lambda/trigger/api-gateway-proxy.d.ts#L116.
export interface EndpointEvent {
    body: string|undefined;
    headers: Record<string, string|undefined>;
    httpMethod: string;
    path: string;
    queryStringParameters: Record<string, string|undefined>;
}

// See https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/aws-lambda/trigger/sqs.d.ts#L8.
export interface QueueEvent {
    messages: Array<{
        messageId: string;
        body: string;
        // ...
    }>
}

The wing-local implementation must know this format to comply with it. We need to be in sync with this, too!

[skyrpex]

This is what we discussed earlier, @Chriscbr @ainvoner. Feel free to mention whoever can contribute to this!

[ShaiBer]

Good points, so maybe we can add another parameter before the event itself that will give us the type. Or the event can be an object with a type and then the internal event?

[skyrpex]

Good points, so maybe we can add another parameter before the event itself that will give us the type. Or the event can be an object with a type and then the internal event?

The event type would be known when the developer writes the wing code, for example:

// user-defined-events-example.w
use cloud

let bucket = new cloud.Bucket

// User defined event
proc putObject(event: { key: string, value: string }) {
  bucket.putObject(event.key, event.value)
}

let fn = new cloud.Function(putObject)

// Must pass the user defined event above
fn.invoke({ key: "hello", value: "world" })

// endpoint-example.w
use cloud

// This event will have HTTP method, headers, etc.
proc putObject(event: cloud.EndpointEvent) {
  bucket.putObject(event.key, event.value)
}

let fn = new cloud.Function(putObject)

let endpoint = new cloud.Endpoint(fn)

From wing-local, we just need to know what shape do those events have.

[Chriscbr]

Looks good to me. I agree, for wing local the function exports has to be part of the contract of the produced .wx file. We could say for example "If the sourceCodeLanguage of the function .wx is "javascript", then the referenced asset file must have an exported function named handler, and ...."

The wing-local implementation must know this format to comply with it.

Yeah, this is a great callout. If that's the case, maybe the definitions should go in the wing-local-api / wing-local-schema repo?

From a user perspective, we also need these types to be exported in the Wing SDK so that they can referenced by the user in Wing, as shown in your code example.

If wing-local-api is a JSII repo, that should be straightforward, since one JSII library can re-export types from another JSII library (I think). If it's not, then we could try doing something fancy with code generation to generate JSII types in the wingsdk based on the wing-local-api source code.

(I don't think these types should be defined in the wingsdk repo, since that would mean wing-local takes a dependency on wingsdk... which I believe we want to avoid?)

---

BTW - from the compiler side, I think Wing's type checker should require that the cloud.Function passed to add_consumer has a correct signature. Maybe this could be modeled something like...

let fn = cloud.Function(putObject); // type: cloud.Function<fn(cloud.EndpointEvent) -> nil>

Maybe we can skip this for MVP though. cc @yoav-steinberg

[Chriscbr]

Good points, so maybe we can add another parameter before the event itself that will give us the type. Or the event can be an object with a type and then the internal event?

I think the issue is that wing-local needs to know how to generate events of these types. For example, when a queue receives a message, in order to invoke a cloud function with a QueueEvent, it must first construct an object complying with the QueueEvent interface. So I think this type/schema needs be known to the simulator implementation at compile time

[skyrpex]

The wing-local implementation must know this format to comply with it.

Yeah, this is a great callout. If that's the case, maybe the definitions should go in the wing-local-api / wing-local-schema repo?

We ended up having it in the wing-local repository (the JSON schema interface, the local client, and the local server). The wingsdk-clients repository will have wing-local as a dependency (for the JSON schema and the local client). The JSON schema interface is owned by wing-local.

We could have a separate repository (wingsdk-shared? 🤔) for the name of the handler that we choose and the interface of the resource-specific events. The wingsdk, wingsdk-clients and wing-local packages would depend on it. The definitions in this package would be very important since they affect all of the possible targets for wing (AWS, google, local, etc).

So, it'd look like this...:

• wing-local
  • depends on wingsdk-shared
• wingsdk
  • depends on wingsdk-shared
  • depends on wingsdk-clients
• wingsdk-clients
  • depends on wingsdk-shared
  • depends on wing-local

This is what wingsdk-shared could export:

/**
 * The name of the JavaScript handler exported in a WingSDK Cloud Function.
 */
export const HANDLER_NAME = "handler"

/**
 * The shape of an event that a Cloud Function will receive when called via a Cloud Endpoint.
 */
export interface EndpointEvent {
    method: string;
    body: string | undefined;
    // TODO
}

// TODO: Other resource-specific events... Shared ownership, too!

[Chriscbr]

Got it - I'm cool with either ways of packaging things.

I was trying to think about this and I started trying to categorize all of the types of information that need to be defined, thought I'd share in case anyone else finds it useful:

Types of data:

1. preflight resource interfaces (cloud.Bucket, add_uploader_handler...)
   • currently defined in wingsdk - https://github.com/monadahq/wingsdk/tree/main/src/cloud
2. inflight client interfaces (cloud.BucketClient, upload, download...)
   • currently defined in wingsdk-clients - https://github.com/monadahq/wingsdk-clients/tree/main/src/shared
3. inflight event interfaces (cloud.QueueEvent, ...)
4. .wx format (wing.local.json)
5. wing local server endpoint names and types ("function/invoke", FunctionInvokeInput, FunctionInvokeOutput, ...)

Constraints:

1. preflight resource interfaces must be in a jsii library, so that our constructs can be used in other languages
2. inflight client interfaces must not be in a jsii library, to avoid jsii type limitations with promise types used in inflight clients
3. wing compiler must be able to integrate the preflight resource interfaces, inflight client interfaces, and inflight event interfaces into its type system
4. wing local simulator needs to know how to parse the .wx format, and the Wing SDK needs to know how to generate the .wx format
5. (what else?)

BTW - this is a random idea, but what would happen if we had the local simulator as part of the wingsdk package?

// From the wing-console (main process)
import { WingLocal } from "wingsdk";

const app = WingLocal.createServer("demo.wx", { port: 5000 });

class MyWatchConfig implements IWatchConfig {
  public onChange() {
    app.reload();
  }
}

app.watch("demo.wx", new MyWatchConfig());

[skyrpex]

Nice job, Chris :) I have nothing to add at the moment.

I just wanted to point out a small perf change: we can provide wing-local-server and wing-local-client separately, so the wingsdk-clients don't need to include the server code in node_modules unnecessarily. The local json schema would still be on wing-local-server, which can be installed as a dev dependency to import the types. Sounds good?

[skyrpex]

BTW - this is a random idea, but what would happen if we had the local simulator as part of the wingsdk package?

// From the wing-console (main process)
import { WingLocal } from "wingsdk";

const app = WingLocal.createServer("demo.wx", { port: 5000 });

class MyWatchConfig implements IWatchConfig {
  public onChange() {
    app.reload();
  }
}

app.watch("demo.wx", new MyWatchConfig());

Just for the sake of convenience, or is there any other reason to do so?

[Chriscbr]

I just wanted to point out a small perf change: we can provide wing-local-server and wing-local-client separately, so the wingsdk-clients don't need to include the server code in node_modules unnecessarily. The local json schema would still be on wing-local-server, which can be installed as a dev dependency to import the types. Sounds good?

Sure, sounds good.

Just for the sake of convenience, or is there any other reason to do so?

Yeah, a little bit - I thought maybe this could eliminate some of the complexity of sharing types/interfaces between these packages. For example, the ".wx" format types (like ResourceSchema) are only needed by wingsdk and wing local, so if they're in the same package, all of the type information can be internal, and it would be easier for us to develop (and there would be fewer versioning issues). But I'm not sure if it would work, since wingsdk-clients still needs wing-local-client...

[ShaiBer]

Nice discussion @skyrpex and @Chriscbr ! So now we'll have?:

• wing-local-client
  • depends on wingsdk-shared
• wing-local-server
  • depends on wingsdk-shared
• wingsdk
  • depends on wingsdk-shared
  • depends on wingsdk-clients
• wingsdk-clients
  • depends on wingsdk-shared
  • depends on wing-local-server

From a Monada developer perspective it makes sense, but if I look at it from a contributor's perspective it makes it very difficult to contribute new resources because you need to contribute to 5 different repos, which means that you can't contribute a new resource with one PR, and also a lot of learning to do before you are able to do so..

[ainvoner]

@ShaiBer which type of contributors are you referring to?

while the sdk project is designed to be an open source project, the wing-local project (server and console) isn't.

If we are talking about "in-house" contributors developing "in-house" resources (pre defined resources that supported by Monada) - you are right, in order to add a full supported resources to all relevant projects, there are many repositories to be familiar with (wing-dev-console is another one :)). - although, a contributor could just add the sdk impl. without being familiar with the local server (only the schema).

Regarding external contributors adding "external" resources - they will only have access to the sdk projects. Post MVP we will need to add wing-local (server and console) support for "external" resources dynamically. for example:

• add to the schema properties like:

  {
  "app.cloud.Kuku": {
          "id": "app.cloud.Kuku",
          "path": "app-cloud/app.cloud.Kuku",
          "type": "cloud.Kuku",
          "props": {
            // will be required and used by the wing server and wing-console
            "externalLib": "com.external.lib.kuku",
            "constructionFunction": "create" // can be defined as default
            "renderFunction": "render" // can be defined as default
          },
          "callers": [],
          "callees": ["ImageStore.cloud.Bucket"]
        }
   }  
  }

• local-server: create resource object and route
• dev-console: render the resource data UI

[ainvoner]

@Chriscbr

I was trying to think about this and I started trying to categorize all of the types of information that need to be defined, thought I'd share in case anyone else finds it useful:

nice :) also have nothing to add at the moment...

BTW - this is a random idea, but what would happen if we had the local simulator as part of the wingsdk package?

Since the sdk is an open source project and wing-local isn't (and should be monetised) I don't think it will be possible

[ShaiBer]

I was referring to external contributors. For them, even if we add support for dynamic external resources, which is great!, they will still have 3 different open source repos to work with in order to add a single resource. BTW, I understand why the wing local console can't be open source, but are we sure the server is also commercial?

On Sun, Aug 28, 2022, 08:09 Ainvoner @.***> wrote:

@ShaiBer https://github.com/ShaiBer which type of contributors are you referring to?

while the sdk project is designed to be an open source project, the wing-local project (server and console) isn't.

If we are talking about "in-house" contributors developing "in-house" resources (pre defined resources that supported by Monada) - you are right, in order to add a full supported resources to all relevant projects, there are many repositories to be familiar with (wing-dev-console is another one :)). - although, a contributor could just add the sdk impl. without being familiar with the local server (only the schema).

Regarding external contributors adding "external" resources - they will only have access to the sdk projects. Post MVP we will need to add wing-local (server and console) support for "external" resources dynamically. for example:

• add to the schema properties like:

{ "app.cloud.Kuku": { "id": "app.cloud.Kuku", "path": "app-cloud/app.cloud.Kuku", "type": "cloud.Kuku", "props": { // will be required and used by the wing server and wing-console "externalLib": "com.external.lib.kuku", "constructionFunction": "create" // can be defined as default "renderFunction": "render" // can be defined as default }, "callers": [], "callees": ["ImageStore.cloud.Bucket"] } } }

• local-server: create resource object and route
• dev-console: render the resource data UI

— Reply to this email directly, view it on GitHub https://github.com/monadahq/winglang/issues/176, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANGGYBVXSVWLFAM7HVTCV3V3LYBZANCNFSM57TKUMYQ . You are receiving this because you were mentioned.Message ID: @.***>

[eladb]

Shall we continue the discussion around contributor experience on a different issue?

[eladb]

Seems like this issue diverged into two separate topics. Maybe we can create a new issue to reflect what's still open and close this one?

[staycoolcall911]

Seems like this issue diverged into two separate topics. Maybe we can create a new issue to reflect what's still open and close this one?

@skyrpex, @Chriscbr, @ShaiBer - I noticed @eladb's remark - can one of you pls split this issue into what's done and what still needs to be done?

[Chriscbr]

@staycoolcall911 I created a few issues based on some remaining threads in the discussion above:

• https://github.com/monadahq/winglang/issues/328
• https://github.com/monadahq/winglang/issues/329

I think the other issues about the simulator has been addressed (we've moved it into the SDK package) or are captured elsewhere like here: https://github.com/monadahq/winglang/issues/214 - so I'll close this.