Ts-codegen doesn't recognize types by reference

[jawoznia]

I have the execute message defined as such

#[derive(sylvia::serde::Serialize, Clone, Debug, PartialEq, sylvia::schemars::JsonSchema)]
#[serde(rename_all = "snake_case", untagged)]
pub enum ContractExecMsg {
    Cw1(cw1::Cw1ExecMsg),
    Whitelist(whitelist::WhitelistExecMsg),
    Cw1WhitelistContract(ExecMsg),
}

This is a wrapper of three messages that my contract should support.

Schema.json generated for it (using write_api!) is:

  "execute": {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "ExecuteMsg",
    "anyOf": [
      {
        "$ref": "#/definitions/Cw1ExecMsg"
      },
      {
        "$ref": "#/definitions/WhitelistExecMsg"
      },
      {
        "$ref": "#/definitions/ExecMsg"
      }
    ],
    "definitions": 
    ...
      "Cw1ExecMsg": {
        "oneOf": [
          {
            "type": "object",
            "required": [
              "execute"
            ],
            "properties": {
              "execute": {
                "type": "object",
                "required": [
                  "msgs"
                ],
                "properties": {
                  "msgs": {
                    "type": "array",
                    "items": {
                      "$ref": "#/definitions/CosmosMsg_for_Empty"
                    }
                  }
                }
              }
            },
            "additionalProperties": false
          }
        ]
      },
    ...

So we have execute defined as any of theses three messages which are defined in definitions section. But when I ran

cosmwasm-ts-codegen generate \
          --plugin client \
          --schema ./schema \
          --out ./ts \
          --name whitelist \
          --no-bundle

I get

node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

MissingPointerError: Token "Cw1ExecMsg" does not exist.
    at Pointer.resolve (/usr/lib/node_modules/@cosmwasm/ts-codegen/node_modules/@pyramation/json-schema-ref-parser
}

It looks like ts-codegen doesn't understand ref to definition.

[adairrr]

This looks related to #101 which has an open PR: https://github.com/CosmWasm/ts-codegen/pull/102. Would you mind testing your generating against that branch?

[jawoznia]

102 didn't fix this issue

[jawoznia]

To reproduce this issue:

git clone https://github.com/CosmWasm/sylvia
cd sylvia/contracts/cw1-whitelist
cargo schema
cosmwasm-ts-codegen generate

[jawoznia]

@adairrr here is shortened schema json

{
  "contract_name": "cw1-whitelist",
  "contract_version": "0.3.0",
  "idl_version": "1.0.0",
  "instantiate": {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "InstantiateMsg",
    "type": "object",
    "required": [
      "admins",
      "mutable"
    ],
    "properties": {
      "admins": {
        "type": "array",
        "items": {
          "type": "string"
        }
      },
      "mutable": {
        "type": "boolean"
      }
    }
  },
  "execute": {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "ExecuteMsg",
    "anyOf": [
      {
        "$ref": "#/definitions/ExecMsg"
      }
    ],
    "definitions": {
      "ExecMsg": {
        "type": "string",
        "enum": []
      }
    }
  },
  "query": {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "QueryMsg",
    "anyOf": [
      {
        "$ref": "#/definitions/QueryMsg"
      }
    ],
    "definitions": {
      "QueryMsg": {
        "type": "string",
        "enum": []
      }
    }
  },
  "migrate": null,
  "sudo": null,
  "responses": {}
}

[adairrr]

Yes, this issue is because sylvia has the definitions via reference which requires resolution prior to entering the logic. @pyramation is someone working on a solution for this already?

[pyramation]

Yes, this issue is because sylvia has the definitions via reference which requires resolution prior to entering the logic. @pyramation is someone working on a solution for this already?

Nobody is working on this feature yet. Not sure I understand the reference stuff, if there are some examples outputs we can put it on a todo list. If it's simple, would love to accept a PR :)

[bgoober]

I'm receiving a similar error but for Uint128. Unfortunately I can't substitute Uint128 for another structure because I use a Coin. I am not using sylvia. Can anyone point me in the right direction to fix? A friend is not experiencing this issue with his contract and it uses Uint128. edit He does experience it with my contract, however.

node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^

MissingPointerError: Token "Uint128" does not exist.

[hashedone]

It is a long time since the issue was created, and there is an important question not been answered which I think is probably holding it:

if there are some examples outputs we can put it on a todo list

I am as far from TS as I can be sop not using codegen directly, but I think I can help with that a bit.

The idea is that the Rust schemars use refs to handle untagged enums. So the idea is that whenever we have two different Foo and Bar types like this:

#[derive(Serialize, Deserialize, JsonSchema)]
enum Foo {
  Variant1 {
    data: u32,
  },
  Variant2 {
    name: String
  },
  Varient3 {}
}

#[derive(Serialize, Deserialize)]
enum Part1 {
  Variant1 {
    data: u32,
  },
}

#[derive(Serialize, Deserialize)]
enum Part2 {
  Variant2 {
    name: String
  },
  Varient3 {}
}

#[derive(Serialize, Deserialize, JsonSchema)]
#[serde(untagged)]
enum Bar {
  P1(Part1),
  P2(Part2),
}

We want those Bar and Foo enums to give exactly the same output.

In terms of refs themself, it is fairly simple - whenever there is "$ref": "..." anywhere in the schema, it should be substituted directly by the proper subschema using the provided patch.

I believe what might be more tricky there is the anyOf part. For "tagged" enums, generated schemas use oneOf item, which basically, I assume you get the only "required" field from every variant and use it as message discriminant, and then you will it with fields from "properties". The thing is that oneOf allows you to pretty easily list all used variants.

Things get trickier when anyOf comes into play, as the meaning is "any of those variants should match", but by definition, there is no mutual exclusiveness of them. But at least for Sylvia, we can give a guarantee that those variants are mutably exclusive.

What is needed to be done is that when you see the occurrence of anyOf, you expect that it is an array, and all of its items are oneOf items which you would traditionally expect (arrays with are distinct on their required field). What is to be done is to flatten all of those oneOf arrays into a single oneOf array, treating it as a normal oneOf array. Obviously, technically it might be a case that there is a conflict in the required fields, but it means it is invalid input for us - at least for Sylvia, it is good enough to raise an error if there are any conflicts there (in fact it itself fails to compile in such a case).

The little thing to keep in mind is that the elements of anyOf would most probably be references, so all elements it references should be flattened at the end.

Thu/Fri I will prepare the very simple post-processing tool which takes Sylvia schemas, and translates them to the schema accepted by codegen right now (it would perform this flattening), so I will link it here so you can use it as a reference (or figure out some more sophisticated solutions for a case of the mentioned conflict).

[adairrr]

I'm receiving a similar error but for Uint128. Unfortunately I can't substitute Uint128 for another structure because I use a Coin. I am not using sylvia. Can anyone point me in the right direction to fix? A friend is not experiencing this issue with his contract and it uses Uint128. edit He does experience it with my contract, however.

node:internal/process/promises:279

            triggerUncaughtException(err, true /* fromPromise */);

            ^

MissingPointerError: Token "Uint128" does not exist.

Would you mind removing the 'raw' folder and any other schema files beside the contract schema and try again? There are sometimes weird conflicts.

[adairrr]

@hashedone yes, you are correct.

This issue is not difficult to fix, though I currently do not have the bandwidth to implement the changes.

@pyramation this issue would be great to prioritize over others as the number of developers using Sylvia increases.

[pyramation]

hey can somebody explain this to me from a TypeScript lens? I'm not a Rust/CosmWasm dev myself. Maybe @adairrr can you explain what changes may be needed to make this all work?

[pyramation]

the shortened schema isn't creating errors, can somebody make a PR to add to this branch the full schema causing trouble?

here is the file to update: https://github.com/CosmWasm/ts-codegen/blob/issue-103/__fixtures__/issues/103/schema.json

[ethanfrey]

git clone https://github.com/CosmWasm/sylvia
cd sylvia/contracts/cw1-whitelist
cargo schema
cosmwasm-ts-codegen generate

In the desire to get a reproduceable test case, I tried doing this (with a bit more documentation on version).

$ node --version
v19.4.0
$ rustc --version
rustc 1.69.0 (84c898d65 2023-04-16)

$ npm install -g @cosmwasm/ts-codegen

$ git clone https://github.com/CosmWasm/sylvia
$ cd sylvia/contracts/cw1-whitelist
$ cargo schema

$ sha256sum schema/cw1-whitelist.json
4f330e56136aa6a654f951a0fba9241ccaf839e28158d563ba9d58de690b333d  schema/cw1-whitelist.json

# this is the recommended cli usage in the README
$ cosmwasm-ts-codegen generate --plugin client --schema ./schema --out ./ts --name Cw1Whitelist --no-bundle

Gives the following error:

node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

MissingPointerError: Token "Cw1ExecMsg" does not exist.
    at Pointer.resolve (.../node_modules/@cosmwasm/ts-codegen/node_modules/@pyramation/json-schema-ref-parser/lib/pointer.js:89:13)
    at $Ref.resolve (.../node_modules/@cosmwasm/ts-codegen/node_modules/@pyramation/json-schema-ref-parser/lib/ref.js:115:20)
    at $Refs._resolve (.../node_modules/@cosmwasm/ts-codegen/node_modules/@pyramation/json-schema-ref-parser/lib/refs.js:156:15)
    at dereference$Ref (.../@cosmwasm/ts-codegen/node_modules/@pyramation/json-schema-ref-parser/lib/dereference.js:125:23)
    at crawl (.../node_modules/@cosmwasm/ts-codegen/node_modules/@pyramation/json-schema-ref-parser/lib/dereference.js:72:26)
    at crawl (.../node_modules/@cosmwasm/ts-codegen/node_modules/@pyramation/json-schema-ref-parser/lib/dereference.js:84:28)
    at crawl (.../node_modules/@cosmwasm/ts-codegen/node_modules/@pyramation/json-schema-ref-parser/lib/dereference.js:84:28)
    at dereference (.../node_modules/@cosmwasm/ts-codegen/node_modules/@pyramation/json-schema-ref-parser/lib/dereference.js:20:22)
    at $RefParser.dereference (.../node_modules/@cosmwasm/ts-codegen/node_modules/@pyramation/json-schema-ref-parser/lib/index.js:303:5)

[ethanfrey]

Update: there was raw files there as well.

$ ls -l schema/*

-rw-r--r-- 1 ethanfrey staff 38556 May 16 20:15 schema/cw1-whitelist.json

schema/raw:
total 52
-rw-r--r-- 1 ethanfrey staff 17839 May 16 20:15 execute.json
-rw-r--r-- 1 ethanfrey staff   323 May 16 20:15 instantiate.json
-rw-r--r-- 1 ethanfrey staff 17326 May 16 20:15 query.json
-rw-r--r-- 1 ethanfrey staff   326 May 16 20:15 response_to_admin_list.json
-rw-r--r-- 1 ethanfrey staff   221 May 16 20:15 response_to_can_execute.json

After rm -rf schema/raw, running the codegen works. Maybe codegen could be more robust, but it seems like the issue is raw somehow

(I looked at ./ts/Cw1Whitelist.types.ts and it looks good)

[pyramation]

Successfully published:
 - @cosmwasm/ts-codegen@0.28.0

this version is published for the raw/ fix... we're just ignoring anything in a raw/ folder

[ethanfrey]

@jawoznia Can you try this agaim with 0.28.0 version?

It would be great if this works with Sylvia now.

[jawoznia]

@pyramation @adairrr It's working now. Thanks for the fix.

[ethanfrey]

Thank you for the help!