Modularizing Counterblockd functionality & plugin interface for third-party modules

[ghost]

Cleaner version of #79. Also started on some tests,

[ouziel-slama]

@porcupine, look good, tell me please when you feel the code is ready to test.

[ghost]

@JahPowerBit It's just a timer for counterblockd-reparse (but it's also an easy example for anyone looking at how to plugin a method) - I temporarily reverted collection hash logging since it wasn't working well with the default mongodb objectID's. I pushed it back in as hashlib.md5(str(list( on an aggregate minus objectid's but it's less than ideal.

The code should be good to test anytime. (To run the reparse_timer and database_hash logging run counterblockd.py enmod lib/modules/tests before running server.)

[robby-d]

guys, I noticed that a lot of the logic from blockfeeds was chunked together in processors/messages.py ... what do you think of instead splitting things up by functional areas more (e.g. assets, markets/orders, etc)...so for instance, parse_trade_book is a market-related thing, so it'd go under `components/markets.py`` ...then, you'd also take the market related API calls and throw those into that file as well, as well as any related tasks/initialization functionality, and whatever else.

the core blockchain parsing stuff could still go into processors/messages.py, or maybe a components/core.py file.

This way, the logic for a particular functional area is more self contained. However, depending on how we chunk the areas of functionality, one may be able to state that the distinctions are arbitary in some areas (e.g. is compiling balances an asset-related thing that should go in a components/assets.py file, or should we make a separate components/asset_balances.py file)... I would say in cases like this, if the component file is getting large enough, split it up as necessary.

EDIT: another area that would matter is for the stuff that is core and where the priority IS definitely important (e.g. a lot of the initial stuff that was in blockfeed.py), I'd keep that in components/core.py

@jahpowerbit, @porqupine, any opinions here?

[robby-d]

@porcupine , can I get push access to the repo? I'm using this repo with a plugin I'm working on, and would like to be able to fix any bugs I come across (one is that configobj is lacking from pip-requirements.txt)

[porqupine]

Hey! This is kinda funny. While I do sometimes go by that handle, and this email is finding me just fine, I can't say I have any idea what you're talking about. At first I thought it was a very well done phishing attempt, but now I will spend some time tomorrow investigating what's going on. Unfortunately, to the best of my knowledge I can't grant you access. If you have another way to reach the person you think is behind this handle, you should ping them and see what's up.

On Fri, Dec 12, 2014 at 8:27 PM, Robby Dermody notifications@github.com wrote:

@porcupine https://github.com/porcupine , can I get push access to the repo? I'm using this repo with a plugin I'm working on, and would like to be able to fix any bugs I come across (one is that configobj is lacking from pip-requirements.txt)

— Reply to this email directly or view it on GitHub https://github.com/CounterpartyXCP/counterblockd/pull/80#issuecomment-66864739 .

[porqupine]

Ah, one letter off. Disregard the above then.

On Fri, Dec 12, 2014 at 11:42 PM, Stoytcho Stoytchev sstoytchev@gmail.com wrote:

Hey! This is kinda funny. While I do sometimes go by that handle, and this email is finding me just fine, I can't say I have any idea what you're talking about. At first I thought it was a very well done phishing attempt, but now I will spend some time tomorrow investigating what's going on. Unfortunately, to the best of my knowledge I can't grant you access. If you have another way to reach the person you think is behind this handle, you should ping them and see what's up.

On Fri, Dec 12, 2014 at 8:27 PM, Robby Dermody notifications@github.com wrote:

@porcupine https://github.com/porcupine , can I get push access to the repo? I'm using this repo with a plugin I'm working on, and would like to be able to fix any bugs I come across (one is that configobj is lacking from pip-requirements.txt)

— Reply to this email directly or view it on GitHub https://github.com/CounterpartyXCP/counterblockd/pull/80#issuecomment-66864739 .

[ouziel-slama]

@robby-dermody, I have not a strong opinion about this, but my first impulse is to separate modules that write in the database and modules that need only a read access. In this way someone can populate the database with another way, and continue to use read only modules without changes (and vice versa).

[robby-d]

@porqupine, sorry about that typo! :)

[ghost]

@porqupine Sorry about the confusion (I had porqup1ne which was switched to porcupine). @robby-dermody Added you to the repo. As for the logic in messages.py I'd say it's maybe a good idea to keep all the core subscriptions together since that makes it easy to read - but the large functions like parse_balance_changes and parse_trade_book could probably be replaced with a call to the function when the if_statement is triggered the way betting.parse_broadcast(config.mongo_db, msg_data) currently is.

On another note, I assumed the stuff in messages.py, startup.py, caughtup.py would be "core" only, additional modules should import the appropriate processor and add their functions from there, without touching core functions, if someone is using a different type of parsing for their purposes (i.e. xcpfeeds parsed broadcasts and bets differently and didn't use the tradebook or market history) they can disable the core processes via the config file.

@ouziel-slama Maybe this could be done with two separate mongo_db connections (r_only, write) ?

[robby-d]

@porqupine and @ouziel-slama ... sure, that approach makes sense. the core stuff can be organized so that it's easier to enable and disable certain functionality. Plugins/3rd party modules themselves will be naturally split up and grouped, with all of their functionality in one (or more files)... but probably being able to spot and understand the coherency with the core code overrides any functional grouping..I can see that.

[robby-d]

why is there an api_handler.py and a lib/apihandler.py? seems kind of messy

also, is the vend.py an optional or required component?

I'd like to see the config files not located in /lib, but in something like /config or /conf instead, as well.