Blockchain code has to be robust, and has to be maintenable too, so that it stays robust.
Tests help because they check the program behavior at runtime (and while we have good test coverage for the decred library, we don't yet for the tinywallet program).
Another way is types. While Python does have strong typing (you can't do 2 + 'a' as in Javascript), it also has dynamic typing. While that makes for simpler programs and easier introspection and metaprogramming, in static-typing languages (like Go, C and Rust) the compiler checks the data types and does not even generate an executable if something does not match.
What
But Python doesn't have a compiler, so what can we do? Man, I wish Guido Van Rossum spent the last few years before retirement writing some static typing checker... oh wait, he actually did! :wink:
"Mypy is an optional static type checker for Python that aims to combine the benefits of dynamic (or "duck") typing and static typing. Mypy combines the expressive power and convenience of Python with a powerful type system and compile-time type checking. Mypy type checks standard Python programs; run them using any Python VM with basically no runtime overhead."
It is a mature tool used on large codebases at Dropbox, Zulip (awesome chat program, by the way) and many others. See the FAQ for details.
How
Interestingly, we already explicitly declare types: we document them in function/method docstrings (and, more recently, in the field.pxd file that Cython uses to generate a fast C extension).
We need to move those types from the docstrings to type hints in function/method signatures, and add type hints to the main variables and class attributes. Not all of them: mypy is smart enough to deduce most data types from the code, and warns when it can't.
But man, all that work by hand? I wish there was a tool that would watch execution, come up with most types and put those type hints in place... Glad you asked.
"MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type annotations directly to your Python code based on the types collected at runtime."
It writes type information to a SQLite database and uses that information to add type hints to the code.
But how are we supposed to run it? I wish there was a simple way to hook it up to our tests... ok, you got it by now. :slightly_smiling_face:
Introducing pytest-monkeytype
pytest-monkeytype is a pytest plugin that (surprise, surprise) integrates pytest and monkeytype.
Who
I run pytest-monkeytype on the decred library. The SQLite file is about 1.4GB and the added type hints are quite good, they only need checking and a few tweaks (PR #175 is a by-product of that, by the way).
When
As with the secp256k1 field optimization work, it looks like it's going to take much less work than expected, and the benefits seem rather large. Furthermore the work can be gradual as needed, every little bit helps, and there's definitely not going to be any touch-everything-at-once, huge pull request.
Introducing gradual typing
Why
Blockchain code has to be robust, and has to be maintenable too, so that it stays robust.
Tests help because they check the program behavior at runtime (and while we have good test coverage for the
decredlibrary, we don't yet for thetinywalletprogram).Another way is types. While Python does have strong typing (you can't do
2 + 'a'as in Javascript), it also has dynamic typing. While that makes for simpler programs and easier introspection and metaprogramming, in static-typing languages (like Go, C and Rust) the compiler checks the data types and does not even generate an executable if something does not match.What
But Python doesn't have a compiler, so what can we do? Man, I wish Guido Van Rossum spent the last few years before retirement writing some static typing checker... oh wait, he actually did! :wink:
Introducing mypy
From the homepage:
"Mypy is an optional static type checker for Python that aims to combine the benefits of dynamic (or "duck") typing and static typing. Mypy combines the expressive power and convenience of Python with a powerful type system and compile-time type checking. Mypy type checks standard Python programs; run them using any Python VM with basically no runtime overhead."
It is a mature tool used on large codebases at Dropbox, Zulip (awesome chat program, by the way) and many others. See the FAQ for details.
How
Interestingly, we already explicitly declare types: we document them in function/method docstrings (and, more recently, in the
field.pxdfile that Cython uses to generate a fast C extension).We need to move those types from the docstrings to type hints in function/method signatures, and add type hints to the main variables and class attributes. Not all of them: mypy is smart enough to deduce most data types from the code, and warns when it can't.
But man, all that work by hand? I wish there was a tool that would watch execution, come up with most types and put those type hints in place... Glad you asked.
Introducing MonkeyType
From the Github page:
"MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type annotations directly to your Python code based on the types collected at runtime."
It writes type information to a SQLite database and uses that information to add type hints to the code.
But how are we supposed to run it? I wish there was a simple way to hook it up to our tests... ok, you got it by now. :slightly_smiling_face:
Introducing pytest-monkeytype
pytest-monkeytype is a pytest plugin that (surprise, surprise) integrates pytest and monkeytype.
Who
I run pytest-monkeytype on the
decredlibrary. The SQLite file is about 1.4GB and the added type hints are quite good, they only need checking and a few tweaks (PR #175 is a by-product of that, by the way).When
As with the secp256k1 field optimization work, it looks like it's going to take much less work than expected, and the benefits seem rather large. Furthermore the work can be gradual as needed, every little bit helps, and there's definitely not going to be any touch-everything-at-once, huge pull request.
So, why wait? :slightly_smiling_face: