Python or Rust for the tests ?

[sydhds]

Intro

Even if the cloud simulator is still WIP (for the docker/K8S part), we need to decide which programming language we want to use to write the tests (aka functional tests).

For now, we have 2 candidates: Rust or Python

What we want to write

Here is a first test already written (in Python) as a demonstration

• get status: Call JSONRPC API get_status on nodes and check for connection/ban count/ip

And for the others tests, we would like to write:

• "Basic" tests
  • TX between nodes and check the result (transfer coins / simple SC)
  • Buy rolls and check the node is joining the stakers
• "Perf" tests
  • Sending transactions until the network is down
  • Sending SC transactions until the network is down
• "Fuzzing" tests
  • Generate fake blocks (or any others objects e.g. endorsements) and check that the "sender" is banned
  • Generate random SC and check the nodes are ok
  • Generating random API calls

While most of the tests look quite easy to write, the difficulty remains in the data/logs/infos we retrieve. Writing a test, and asserting the network is down is not that useful, without nice information. A framework to help retrieving those information would be nice and avoid that each tests reinvent the wheel; if flexible enough, it would be a great tool to easily add additional info in case, we are (blindly) debugging a particular issue.

Rust pros & cons

• Pros:
  • Easy for the core Team as we are almost all Rust dev
• Cons:
  • What about the others dev in core team like fullstack/security ?

Python pros & cons

• Pros:
  • Easy to learn for core devs that did not know about Python (could do Python with Typing to help), add some diversity (Rust + Python)
  • In the future, Massa plans to hire Testers and in the industry, it's very common to use Python.
  • So it would release time for core dev to focus on other things
  • Good tests frameworks like Robot framework would be a nice thing to build a test team
• Cons:
  • A binding from Rust code should be written. Using PyO3 (so writing Rust code), the generated bindings are then very pythonic
  • A PyO3 example is available here

Conclusion

While writing those tests is not a very fun tasks, it could help use to catch bugs before going into labnet or testnets.

[sydhds]

@AurelienFT @Eitu33 @aoudiamoncef @Reda-R @modship @Ben-PH @Leo-Besancon @Ben-Rey @damip @bilboquet Need you to vote on Rust versus Python thing?

[sydhds]

@massalabs/core-team ping

[AurelienFT]

I vote for Python.

[aoudiamoncef]

IMHO, Python 🐍 checks more boxes than Rust.

For load tests, Python have Locust

[Eitu33]

Python is probably the best choice in the long run, it gets my vote as well

[Ben-PH]

I'm erring on the side of python.

The Pros for rust, and cons for python, only seem to carry weight when viewed from the perspective of the core team only, and that would be to the detriment of the project as a whole. The inverse also remains: The cons for rust, and pros for python, seem to be weak only in the context of the core team, but with respect to the project as a whole, seem to carry significant weight.

..I also think the diversity argument is particularly valuable.

I would like to see something that ensures the use of explicit typing, as that would address some of my concerns about using python, though that might just be a personal thing. I'd be mild voice of push-back if we are going for untyped, but I won't insist on it too strongly.

[modship]

I’m okay with both. It depends on the use cases and the machines on which it is used but if we do not look for the performance I think that python, not being compiled is more easily modifiable/maintainable

[Leo-Besancon]

I'd probably be happy with both choices. I never wrote python tests and I usually like Rust tests, but it may be easier to quickly iterate with Python.

[Ben-Rey]

Python: We will find many more developers able to code in Python than in Rust.

[bilboquet]

I'd vote for Rust :)

About Rust main cons, I'd say that it's an opportunity to learn Rust for other guys, as a C and C++ programmer, learning Rust had a positive impact on my work in those languages. I fill it would be same for others.

The major problems I had in the past with functional tests were related to duration (it took days). Last time I've seen comparisons Python was 80x slower than C and Rust 1.1x, if it's still relevant it may not be a blocker for test writings.

If we go the Python way, I really hope we'll use Typing.

[Eitu33]

On second thoughts, is the argument of saying "Python is easier to maintain than Rust" true? In the short run I believe this isn't true since the current team is more proficient in Rust. For the long run on the other hand it first felt obvious that Python would be better as I said in https://github.com/massalabs/massa-cloud-simulator/issues/25#issuecomment-1427599149, but after thinking a bit more about it using Rust for tests isn't the same as using Rust for a complex project, the understanding of underlying behaviours is not as necessary in tests, and I believe a good developer (no matter the background) would be able to maintain that kind of code. This is a personal opinion that might not be true. I also understand that using Rust for testing would probably be frustrating for someone that is used to Python.

Edit: Also as @bilboquet pointed out in https://github.com/massalabs/massa-cloud-simulator/issues/25#issuecomment-1429479335, the performance is definitely something we'll need given the nature of our system if we want scalable tests

[sydhds]

There is no doubt that Python performance is not an par with Rust performance but I think this is not very important in our case because:

• Python will use the python api written in Rust (using PyO3) so serialization / comparison / object creation perf should be good
• Tests will mostly use the jsonrpc / gprc API so waiting for the (network) response or will wait a lot (let's say for some finalization)

[bilboquet]

"In the future, Massa plans to hire Testers and in the industry, it's very common to use Python. So it would release time for core dev to focus on other things" and "We will find many more developers able to code in Python than in Rust."

How true is it? I mean, there are obviously more Python devs than Rust devs, but are they easier to hire to write tests? An experienced Python dev will probably have more fun projects to deal with. Whereas someone with just basic Rust understanding (enough to write scenarios) will probably be more inclined to do functional testing in Rust to gain in experience.

[AurelienFT]

@massalabs/core-team Last week to give your thoughts on this.

[bilboquet]

@massalabs/core-team

Just to elaborate a bit about my choice.

My feeling (a bit driven by previous experiences, I may give details if asked) is that we will have to manage functional tests ourselves.

That's why I vote Rust.

Maybe it's no the best choice, I, for sure, don't have the big picture (better test framework in Python? anything else?), but I think that this point (that we'll have to do them) was underestimated.

[damip]

For all intents and purposes I would vote for rust as well. Python has one advantage tho: data analysis and visualization. If we want to gather de data from many nodes and run statistics, data analysis and plot graphs quickly, Python is the way to go.

[AurelienFT]

I don't think the first usecase is data visualisation it could be in a second time.

[sydhds]

I'm still in favor of Python as it shows many advantage (libs for visualization / statistics / graph / testing framework), it allows us to easily onboard other dev from the run team (and maybe others: inno / secu ...), and it's more flexible.

Not sure to see any interest in using Rust besides "we are all Rust dev"...

[damip]

Looks like we need a vote :) can you organize one on discord @sydhds by summarizing all arguments for python and for rust ?

[bilboquet]

Truly naive question: do we really have to choose?

I mean do we have to choose now? Isn't it possible to have the tests done in either Rust or Python (as long as we have the traceability) depending on the use case or the ability of the person in charge?

[AurelienFT]

We will all make tests for massa. So we will all be "in charge".

[damip]

Note: the security team will also be using this

[bilboquet]

We will all make tests for massa. So we will all be "in charge".

I totally agree.

[sydhds]

Looks like we need a vote :) can you organize one on discord @sydhds by summarizing all arguments for python and for rust ?

Make a thread in our Discord :)

[damip]

Chosen: python. Note: when GRPC is available, the pyo3 bindings won't be necessary anymore