Cadence Test Framework Improvements

[m-Peter]

Cadence Test Framework Improvements

Grant category

• [x] Developer tools / services
• [x] Open source maintenance

Description

The Flow Emulator is the developer tool for the Flow blockchain which allows the development of Cadence smart contracts, scripts and transactions. It is bundled with the flow-cli tool. Among the supported commands of the flow-cli, there is also a test command, which runs tests written with the Cadence testing framework, while utilizing the Flow Emulator under the hood.

I would like to make the Cadence testing framework more feature-rich, by implementing the features described in https://github.com/onflow/cadence/issues/1889 (Matchers / Unit Testing / Integration Testing / Deployed Contracts / Logs & Events)

Problem statement

Testing is an essential part of the development workflow. The Cadence testing framework, as of now, has a basic set of features that makes it functional, however it needs more features to make it a fully-fledged testing framework.

Proposed solution

The TestFramework interface has to include the proposed API from https://github.com/onflow/cadence/issues/1889. This API has to be implemented in the Cadence testing stdlib (runtime/stdlib/test.go) and the emulator-backed implementation (test/emulator_backend.go). The relevant code files can be found below:

• https://github.com/onflow/cadence/blob/master/runtime/stdlib/test-framework.go
• https://github.com/onflow/cadence/blob/master/runtime/stdlib/test.go
• https://github.com/onflow/cadence-tools/blob/master/test/emulator_backend.go

Impact

Provide test / development parity, by allowing developers to write unit/integration tests in Cadence (https://12factor.net/dev-prod-parity). Developers will also be able to measure how well their tests cover their contracts, and gain sufficient confidence before deploying to testnet, and more importantly on mainnet. Code coverage will be natively offered by the framework.

Milestones and funding

Milestone	Deliverables	Timeline	Risks	USD proposal
1 - MVP	Implementation with detailed documentation in git repo including usage examples	~ 12 weeks	-	33,750
2 - Adoption	Use the new testing features on some of our existing repositories and get 5+ projects on Flow to using the Cadence testing framework	~ 8 weeks	-	22,500
3 - Maintenance	Resolve and triage issues, feature requests. Write some official documentation.	3 months	-	15,000

Total Cost: 71,250 USD. Of course, I am open to discussion, regarding the cost and timeline :slightly_smiling_face: .

Team

Name	Role	Bio	Contact
Ardit Marku	Back-end Engineer	A seasoned back-end engineer with a focus on problem solving and designing robust software systems. Over 8 years of experience building high-traffic search engines, marketplaces, exchanges and in-house analytics solutions. Led cross-functional teams to deliver both web and mobile apps, participating in all steps of the software development lifecycle. Open source lover and advocate.	markoupetr@gmail.com m_peter

[chrisackermann]

Hi @m-Peter! Thanks for submitting your proposal - just wanted to let you know that we've accepted your proposal and are following up directly with next steps.

[m-Peter]

Hey @chrisackermann :wave:, I have actively started development on this proposal as well. The 1st PR tackling the addition of new built-in matchers, can be found here https://github.com/onflow/cadence/pull/2420 :pray:

[chrisackermann]

Excellent news @m-Peter !

[satyamakgec]

Love the proposal, I Like to include the minor stuff as well like -

• https://github.com/onflow/cadence/issues/2076 (proper error management to test the failure of the transaction)
• Some un-reported issues as well like how to take care (handle it from the return statement or whatnot) of the custom types in the test file, This isn't possible right now because I can't import the smart contracts in the test file because of that test file can't access the custom types like struct or resources of that contract. Although I am expecting this to cover in Integration testing functionality but just want to emphasise the problem statement.
• There are some improvements that can be done in handling the return values of the scripts like the cadence test framework doesn't able to handle the tuple values in return.
• Able to write modular test cases means a developer can import the utility functions from a cadence file to the test file. to avoid repetitive code.
• In the logging and event section. I want a simpler methodology of console.log where I can console in the contracts and in the test file as well .. it will make the debugging a hell lot easier.

[m-Peter]

@satyamakgec Much thanks for your feedback :bow: I will certainly take these features in account, and provide an implementation for them. In fact, regarding your last bullet point about logging, I have already added this feature in https://github.com/onflow/cadence-tools/pull/109. The log built-in function can be used in contracts/scripts/transactions and test files as well. It will be included probably in the next release :pray:

[m-Peter]

@franklywatson The repository containing usage examples about the testing framework has moved here: https://github.com/m-Peter/flow-code-coverage. This is where I will put samples for any new features.

[franklywatson]

Nice work! Closing since this is now completed