Feature Idea: Utilize workspaces as a pipeline tool

[TrevorJTClarke]

for example: 1. workspaces can be used for testing + deployment to testnet 2. local & testnet can be mocked and deployed to 3. Workspace tests all pass, then re-use some methods for deployment + configure + migration phases 4. Output the report into a snapshot that can be tagged as a release for github (so CI/CD can do release)

For more context, truffle users will be familiar with: [Truffle Migrate](https://trufflesuite.com/docs/truffle/getting-started/running-migrations) – while im not in any way proposing this model, just suggesting as a possible thought pattern

[willemneal]

Does simulation mean near-sdk-sim? From my understanding it's being deprecated.

[TrevorJTClarke]

Does simulation mean near-sdk-sim? From my understanding it's being deprecated.

@willemneal sorry no, i know thats being deprecated. I was purely meaning the "simulation" of sequence of events/txns in testnet/localnet.

[ChaoticTempest]

As discussed internally with @TrevorJTClarke, I'm all for this. But also could easily end up bloating the simple case where people want to just do tests if we're not careful enough with the boundaries when it comes to having CI/CD be apart of workspaces. This part could just end up being something separate from workspaces entirely.

@matklad I know you had some thoughts about this, and how it could complicate the API if we try to support beyond just testing. Tagging just in case you want to chime in here

[TrevorJTClarke]

Added note about truffle migrate pattern: https://trufflesuite.com/docs/truffle/getting-started/running-migrations

[matklad]

Yeah, I don't have any object-level thoughts here (to unfamiliar with the domain!), but I want to share several meta-level observations:

1) Path dependencies matters: it might be wrong to look at the snapshot of the truffle ecosystem today, and try to re-create that for near in one go. Truffle ecosystem probably grew over time, and we should try to recreate the growth process, rather than the end result. 2) In particular, we should be mindful of the potential midlayer mistake: providing fancy opinionated convenience API without providing lower-level verbose-but-direct APIs as well. Such situation make it harder for community to try to correct our mistakes by building better APIs. It also locks us into providing strict compatability guarantees for the high-level API. 3) We should be mindful of the design tension between "testing DSL" and "production DSL". As an example, for testing I imagine it would be convenient if we attached max gas by default. For prod, gas amount obviously needs to be specified explicitly.

With this in mind, I would suggest the following overall evolution story:

• ship near-rpc 1.0 as a very low-level API with strict semver guarantees. It shouldn't be user friendly, it should be a stable foundation other people can build user-friendly APIs on top. In particular, this thing should not have specific knowledge of a sandbox. It should be able to call sendbox' methods treating sandbox as a URL, but it shouldn't be able to, eg, download sandbox node itself.
• based on that, release a high-level library with a razor-focus on testing the contracts, and just that
• after this, now that we've gained experience with both low-level direct access to RPC and a specific high-level wrapper, try to build a unified API which is conveninet both for testing and deploy.

I am looking at the development from pretty far, but it seems to me that we are trying to do some combination of 2 and 3, rather than going through 1, 2, 3 in order. Again, I don't have enough hands-on knowledge to judge whether this is the right approach in the current context, but the risks I perceive are:

• By not doing 1 (solid low-level library) we
  • make it harder for the ecosystem to run ahead of us. If someone thinks "wait, I know much better way to do what workspaces does", they have to do a lot of grunt work before they can actually experiment with the idea
  • make it harder for us to evolve API. If we have high-level and low-level API, we can say "If you want stability, use low-level API as that's going to be at 1.0 for the next ten years or so. If you want convenince, use high-level API, but we are still refining it, so expect a semver-incompatible version every 6 months or so".
• By trying to do both 2&3 at the same time we:
  • delay the moment we ship a solid testing story. As far as I understand, we don't have any testing approach which a) works today b) will work next year. We are "typical google" here, where one solution is not yet stable, and the other is already deprecated. I think it's important that we ship just the testing as soon as possible, and think about everything else later.
  • run a higher risk of shipping suboptimal API (and not doing 1 makes bad API choices more costly).

[ChaoticTempest]

All very good points. I wouldn't imagine us trying to tackle this feature (or bullet point 3) anytime soon as we are refining the API for bullet points 1 & 2. Some convenience stuff like being able to pull down stuff from prod such as contracts will be apart of 2, but could easily extend to 3. But yeah, I agree we should keep it simple for now until it's stable, otherwise the increase in scope will be pretty daunting.

I'll keep this issue open though since this was more to track the possibility of it being worked on in the future.

[frol]

@TrevorJTClarke What prevents someone from building pipelining tool as a separate project? Maybe I am missing some context, but I would rather prefer to have some simple tool addressing testing problem space and a separate tool providing deployment utilities.

ship near-rpc 1.0 as a very low-level API with strict semver guarantees. It shouldn't be user friendly, it should be a stable foundation other people can build user-friendly APIs on top. In particular, this thing should not have specific knowledge of a sandbox. It should be able to call sendbox' methods treating sandbox as a URL, but it shouldn't be able to, eg, download sandbox node itself.

@matklad It sounds like near-jsonrpc-client-rs fulfills this requirement, doesn't it?

based on that, release a high-level library with a razor-focus on testing the contracts, and just that after this, now that we've gained experience with both low-level direct access to RPC and a specific high-level wrapper, try to build a unified API which is conveninet both for testing and deploy.

Totally agree here 👍

[matklad]

It sounds like near-jsonrpc-client-rs fulfills this requirement, doesn't it?

@frol I'd phare this as "aspires to fulfil". Looking at the deps of workspaces themselves,

https://github.com/near/workspaces-rs/blob/7e2e1064436191bea3abcd406d74f52139e0855d/workspaces/Cargo.toml#L29-L33

it seems that we still leak impl details. For "definition of done", i'd expect this to read like

near-jsonrpc-client = { version = "1.0.0", features = ["sandbox"] } 

or

near-account-id = "1.0"
near-jsonrpc-types = "1.0"
near-jsonrpc-client = { version = "1.0.0", features = ["sandbox"] } 

This is not so much about the number of entries in Cargo.toml or cute version numbers, but rather exposing crates from nearcore which lack strict semver guarantees.

It seems that near-jsonrpc-client is stable enough for folks affiliated with nearcore to build their projects (indexer, workspaces), but isn't stable enough for third parties (as it is exposed to changes in nearcore).