Generate docs from the functions

[swfsql]

For each method, expose or generate more information from near_bindgen into a module of the same name of the method:

• Make the Input struct public.
  • Copy the function's docs into that struct.
    • Allow for doc-commenting on the function's parameters (which rust doesn't allow), and move them over as docs to that struct's fields.
    • Additionally derive a JsonSchema for the struct.
      • This can be used to be a part of an automatic OpenAPI doc of that function.
    • The simulation tests can use that to create the arguments that the function requires.
• Create a trait that has a method with the same name as the original method, and also with the same parameters (except self).
  • This allows an automatic generic implementation that can help with the tests. In this way, some wrapper type (over the contract) can, after importing all of the contract's generated traits, to directly call the methods, and have the IDE to show all off the arguments, return with a defined type, and to be able to go to the function definition. In this scenario, the tests also don't need to use the view, call and deploy macros, nor to make usage of the json macro.
• Implement a common trait for the Input that allows access to useful information, such as whether the function has no return, and so on. This information can later be used during the OpenAPI generation.
• Also generate or implement JsonSchemas for all of the types inside of the sdk, since they can be depended upon by the contracts from the users.

I'm not sure of how a proper design would be, since this touches a lot of inner workings of the near_bindgen code generation; nor how it would interact with other parts of the system that are not json-based.

[willemneal]

Check out https://raen.dev, This already does mostly as you describe. Even allow for the validation of inputs.

Furthermore, this NEP, proposes using wit as the underlying contract interface. Then schemas can be generated from it: https://gov.near.org/t/proposal-use-webassembly-interface-types-to-describe-all-standards-interfaces-and-application-contract-interface-aci/23256

[swfsql]

Whoa, thanks for sharing this! I didn't know of it, nor of the WIT format - it will probably be more general than the jsonschema/openapi ones, if I had to guess. I myself will try to learn more about those, even openapi itself is news to me!

And well, I actually made this issue after trying something out, after making a best-effort (messy) trial with this, but specifically with openapi. This is an example of what it could look like, if this can help with ideas and whatnot.

[willemneal]

Looks great! I had been wanting to do something similar generating from WIT. @zcstarr has similar ideas for using OpenRPC.

[zcstarr]

@willemneal thanks for the shoutout. @swfsql awesome. Documented interfaces are great. I have a some things going already with OpenRPC but on the static description side.

We've done some work with using decorators in golang to automatically provide documentation for golang rpc interfaces. Filecoin lotus uses it currently for their interface https://github.com/filecoin-project/lotus/search?q=openrpc. Eth2 execution apis https://github.com/ethereum/execution-apis uses OpenRPC as well , it's basically what you mention openapi for json-rpc.

I have a repo where I've done this kind of annotation for near smart contracts https://github.com/shipsgold/open-rpc-near-client-generator https://github.com/swappland/open-rpc-near-token-standards https://github.com/swappland/swappland-market-contract-spec

https://open-rpc.org - just general info about open-rpc

We'd love to bring our OpenRPC Rust ecosystem back into parity with the rest of OpenRPC tooling. Aka do the same thing we've done with decorators in golang with macros in rust.

The way it works in OpenRPC is we use the rpc.discover on endpoints to return back the spec itself, meaning you can request this json-rpc endpoint ("rpc.discover"), and then go to town.

It makes it possible to then generate clients and servers straight from the docs, potentially straight from the rpc endpoint.

In the Near case that could be straight from the contract itself. Additionally you can then using tooling like https://playground.open-rpc.org and https://inspector.open-rpc.org to interact with and get typesafe autofilled usage examples.

The main thing, I think with the returning of a spec from the macro route is the potential gas limitation on returning back a potentially large spec, which might need an extension to allow for some sort of framing/pagination. Those are I think workable details to sort through.

Would love to chat about it we have discord for OpenRPC here, I'm the same handle on discord. Happy to carry this discussion here as well so people can chime in!

[willemneal]

The main thing, I think with the returning of a spec from the macro route is the potential gas limitation on returning back a potentially large spec, which might need an extension to allow for some sort of framing/pagination. Those are I think workable details to sort through.

I've been injecting the types directly in the contract. One example is the tenk contract which compressed as JSON is 5Kb and as wit 4.1 Kb (about to switch since the syntax is not stable). Then you just download the contract costing no gas, though you pay some for the extra storage within the wasm.

[zcstarr]

The main thing, I think with the returning of a spec from the macro route is the potential gas limitation on returning back a potentially large spec, which might need an extension to allow for some sort of framing/pagination. Those are I think workable details to sort through.

I've been injecting the types directly in the contract. One example is the tenk contract which compressed as JSON is 5Kb and as wit 4.1 Kb (about to switch since the syntax is not stable). Then you just download the contract costing no gas, though you pay some for the extra storage within the wasm.

Very true that's the benefit of static generation, there's no reason why the tooling couldn't just download the wit and work things out from there. It would be probably be an easier operational step than going from macro to contract endpoint.

[swfsql]

That's amazing @zcstarr! Let alone the documentation, but also for the client building side of things! I also didn't know of OpenRPC, and I myself will be studying on it! Thx

[swfsql]

One note is that I believe it would not be a problem for the tooling to use some macros to get the information, as it could try avoid storing that info on the contract wasm binary (at least to not store immediately). For example, it could create a non-wasm binary that could generate the artifacts needed (wit/openapi/openrpc) and then create the files alongside the contract wasms - I edited the example I linked to a repo that does something along this line (ps. the code and forks behind this particular experiment is extremely drafty).

Mostly in the case for wit, I believe, maybe a different tool could then merge/insert the wit file into the contract if desired; or else the contract could give a url reference to the files as well - and this could work for any of the wit/openapi/openrpc, if not even for all of those.

[austinabell]

• Make the Input struct public.

This is a maintenance issue, we want to be able to update the functionality of deserialization without breaking APIs.

The simulation tests can use that to create the arguments that the function requires.

So the issue with this generally is that you need to add rlib to your crate which disables lto and makes the wasm binary bloated, but also it requires that the tests you write reference the code directly.

It seems we are solving all of what you are looking to do and already have a PoC working for all tools connected through a schema file we are referring to as ABI (similar to Ethereum). We are moving toward making this production-ready and generally usable (current PRs https://github.com/near/near-sdk-rs/pull/831 and https://github.com/near/cargo-near/pull/2).

This is similar in concept to what Willem is describing, we are just using JSON instead of wit because it's more commonly used/read as well as the details of NEAR contracts not being expressible in wit. wit can be misleading for developers since it expresses the interface as if the parameters are passed directly into the exposed function, which is not how it works with the runtime. Because of this, tooling for wit won't translate so it's just like generating a DSL for expressing the semantics of the contract interface.

@zcstarr that's really interesting about the openrpc client generation, I think it could be cool for us to support adding the openrpc json target to our output. (Also able to add a wit target if there were use cases for that)

cc @itegulov wdyt about the open-rpc tooling? Do you think it's viable to allow that target through cargo-near, maybe under some feature flag? I wasn't aware of these tools previously, were you?

[willemneal]

This is similar in concept to what Willem is describing, we are just using JSON instead of wit because it's more commonly used/read as well as the details of NEAR contracts not being expressible in wit. wit can be misleading for developers since it expresses the interface as if the parameters are passed directly into the exposed function, which is not how it works with the runtime. Because of this, tooling for wit won't translate so it's just like generating a DSL for expressing the semantics of the contract interface.

@austinabell wit can transform into the current ABI and vice versa. Currently it assumes json serialization but attributes in the documentation allow for defining the serialization.

I take it that in your view they are incompatible and thus is the reason this post hasn't been responded to: https://gov.near.org/t/proposal-use-webassembly-interface-types-to-describe-all-standards-interfaces-and-application-contract-interface-aci/23256

wit provides a human readable definition for the interface, while also having the added benefit of compressing smaller than the JSON schema will.

wit can be misleading for developers since it expresses the interface as if the parameters are passed directly into the exposed function, which is not how it works with the runtime.

This is especially strange way to put it since this is exactly the abstraction we want developers to have. They shouldn't need to know that the runtime is not implemented as if it was a function call. This is why WIT is as an Application Contract Interface rather than an ABI. The standards should be defined as ACI's and WIT can provide this.

If you think that ABI and WIT are at odds please express that in the forum post or here. I understand y'all are in a rush for nearcon, but I honestly beg you to consider working with the community.

[zcstarr]

@austinabell that'd be cool to support OpenRPC as well. If folks want a walkthrough of OpenRPC or have questions that need clarification feel free to ping me( I'm one of the maintainers of the project) or pop into the discord we're pretty active about answering questions https://discord.gg/ea4HKYm.

[swfsql]

@austinabell That's amazing, I'll be following on the progress!

This is a maintenance issue, we want to be able to update the functionality of deserialization without breaking APIs.

I see, would this be in the sense that contracts that depend on the generation could stop compiling (or miss-compile)? Perhaps considering the increase of the major version for those cases could be a possibility.

edit: removed a section I wrote on the rlib thing. Indeed, rlib is required when trying to test it.

[austinabell]

I take it that in your view they are incompatible and thus is the reason this post hasn't been responded to

Nope, @willemneal, not an intentional choice, just simply a lot going on and busy to read in-depth and add a thoughtful response to this.

Also, @itegulov has been leading this work, so he would have the most context on the tradeoffs.

If you think that ABI and WIT are at odds please express that in the forum post or here. I understand y'all are in a rush for nearcon, but I honestly beg you to consider working with the community.

Nope, not at odds at all, this is why I suggested the possibility of incorporating with the path we are going. Nothing here is a standard or protocol change so no reason why these would be conflicting :)

I see, would this be in the sense that contracts that depend on the generation could stop compiling (or miss-compile)? Perhaps considering the increase of the major version for those cases could be a possibility.

@swfsql yes so it's increasing the surface area of an API that isn't necessarily polished or stable. It restricts improvements that can be made in the future for a very specific use case. Any changes would break usage which always would have to come with a major version change and effort in migration from developers who used it.

[itegulov]

wdyt about the open-rpc tooling? Do you think it's viable to allow that target through cargo-near, maybe under some feature flag? I wasn't aware of these tools previously, were you?

@austinabell I wasn't aware of OpenRPC's existence either, seems like a decent candidate to explore. Given that the OpenRPC specification uses JSON Schema natively, should not even take too much effort to add support for it.

I take it that in your view they are incompatible and thus is the reason this post hasn't been responded to

@willemneal just to supplement @austinabell's statement, I don't think they are incompatible either and I think it is great that the community is working on solving the same fundamental problem with their own vision. I have wanted to write up my personal criticism of the wit approach under the governance post for the last week but just had no time to do so.

Maybe we can find a common ground on the approach (it is not set in stone yet), maybe we can support both formats in our tooling. There are a lot of options available to us and in no way do I think our work is at odds.

[willemneal]

@itegulov look forward to your feedback. The way I view wis is ACI for higher level views like generated markdown, and a more easily embedded format. Then fetch contracts and generate the JSON ABI to be consumed by the tooling. I'm almost done with Wit --> JSON and JSON --> Wit will be even easier. So once we have them isomorphic all tooling will be available to devs.

[austinabell]

Our ABI tooling includes docs from functions/types. https://github.com/near/abi is the landing page for all of these tools.