Discussion on features of a full steem client

[youkaicountry]

A decision should be reached on the functionality of a fully featured steem client. Currently simple_steem_client only implements RPC calls.

[theoreticalbts]

The way I see it, the Steem client functions are as follows:

• RPC interface. Give JSON data to steemd (or a compatible endpoint, e.g. Jussi) over HTTP JSON-RPC
• Type wrappers. Implement native types or classes for "simple" types like asset, public keys, datetimes. And for "compound" types, datastructures such as operations, transactions, blocks, etc.
• Serialization. Allow type-wrapped types to be converted to/from JSON and/or bytestrings.
• Signing. Allow bytestrings to be signed with private keys in WIF format. Convert between internal privkey/pubkey types and strings (WIF-format private keys, string formatted "STMxxxx" / "TSTxxxx" addresses). Signing tx's requires us to fill in the block ID of a recent block and a time in the near future [1] for the TaPoS and expiration fields; serialize the tx; and prepend chain ID.
• Wallet. Implement a persistent storage strategy for private keys. [2]
• Multisig support. Answer questions such as "I have this transaction, what are the pubkeys whose signatures might help?" and "I have this transaction and the privkeys corresponding to these public keys, what should I sign with?". This might require duplicating some of the multisig logic in client code, or writing some additional API calls to route this functionality out of steemd.

Ideally these should be separate modules, each of which is its own Python package, and then there's a "kitchen sink" package that wraps them all and provides a simple interface for e.g. people who want to simply write a bot that can read the blockchain, generate and broadcast transactions.

[1] The near future according to blockchain timestamps. If we're willing to assume the blockchain will always be reasonably up-to-date with real time, then a reasonably accurate UTC time will do -- but this is not always a reasonable assumption! For example, while tinman is setting up a testnet, transactions are generated on a blockchain whose time may be days / months / years in the past, according to the system clock.

[2] If it lives on disk, how/where does that disk file live? Is it encrypted? What are the low-level details (serialization, encryption algorithm, how is the key input for encrypting / decrypting)? Or maybe it's a key generated from a passphrase? How about if we want families of autogenerated keys (for example tinman generates keys for multiple accounts for multiple purposes)? How are the pubkeys indexed e.g. for multisig queries? Do we use OS level API's to lock memory pages containing privkeys from being swapped and make sure the memory's scrubbed? Do we offload indexing of which account(s) a given key can sign for to the account_by_key plugin's API, or do we somehow track / cache that information ourselves?

[theoreticalbts]

Project organization

• (a) RPC
• (b) Serializer
• (c) Signing
• (d) Wallet
• (e) Multisig support
• (f) "Swiss army knife" wrapper which depends on the above five projects
• (g) Utility scripts using the above libraries

Packaging

Each of the above projects should have its own Git(hub) repo and Python packaging. The standard for separation of packages is this: Each package should be a tool that a single developer could code a basic replacement for in a day (with general competence and prior working knowledge of steemd). I chose this standard for a reason: We want to occasionally do exactly this! For example the serialization library will be completely replaced with fully auto-generated code.

With independent projects, we force disciplined separation of concerns and communicating over well-defined interfaces. Refactoring or even re-writing has a relatively low cost, so we can iterate on each piece as needed.

Install documentation

Projects should always have installation and common usage scenarios thoroughly documented in the README.md. Especially if using pipenv instead of pip.

Dependencies

To avoid PyPI dependency hell [1], we should strive to have projects (a)-(e) function only with the built-in library and vendorized [2] MIT-licensed pure-Python dependencies. Any installation procedure must be THOROUGHLY DOCUMENTED AND TESTED. Especially if it is different than the usual way of installing Python packages [3].

Python version

All new Python projects should be compatible with the system version of Python 3 on Ubuntu 16.04, or after Ubuntu 16.04 is no longer supported, the system version of Python 3 on the most recent still-supported Ubuntu LTS release [4].

Python 2.7 compatibility is nice to have, but a low priority. I personally won't spend any time on it. In my code I freely use yield from and other constructions that don't have 2.7 equivalents [5].

[1] My packaging paranoia is a consequence of me wasting a day on ultimately unsuccessful attempts to use the previous steem-python library: https://github.com/steemit/steem-python/issues/96

[2] Copy the dependency's source files into the project's source tree.

[3] "The usual way of installing Python packages" for me is:

virtualenv -p $(which python3) /path/to/ve
. /path/to/ve/bin/activate
pip install /path/to/project/source

[4] steem-python requires a custom Python build to run on Ubuntu. This is a bit of an annoyance.

[5] Python 2.x support is a non-goal for new projects, unless the new project must interoperate in-process with specific existing Python 2.x code. People have been warned that Python 3 is the future for 10 years now. It's not a surprise.

[youkaicountry]

I think making each of those a separate github repository would cause a bit of a mess. It seems like unnecessary organization, when simple python modules within the same repository would work just as well.

If we are vigilant in reviewing additions, we can maintain the disciplined seperation of modules.