Remove all dependencies on Obelisk; Temporary solution using an existing project.

[msgilligan]

I'm creating this issue to start a discussion around a temporary alternative to Issue #1164 (OmniEngine implementation of BTC balances and UTXOs)

I spent some time looking at alternative implementations of arbitrary BTC address balance and UTXO tracking. If found two that are worth considering:

1. Bitpay Insight API - uses Node.js and LevelDB
2. Extended bitcoind There is an addrindex branch of bitcoind that has the basic features we need maintained by @jmcorgan and @dexX7 has a master-addrindex-extended branch that includes additional RPC functionality, notably a listallunspent method.

[msgilligan]

And it looks like @dexX7 wrote some Python code to talk to this modified bitcoind back in May.

Should we take a serious look at this as a temporary solution until #1164 is finished? If a few other folks supported this idea, I'd be willing to do some integration and testing to make it happen.

[msgilligan]

I should also mention that Counterparty is now requiring the addrindex version of bitcoind.

[dexX7]

Hehe, you already found the mastercoin-tools mod.. :)

I'm using the extended address indexed branch for quite a while and it's used by Chancecoin, a gambling meta-coin, for probably half a year a few thousand mainnet transactions. There is basically no difference to @jmcorgan's branch, except the extended RPC calls.

The original was recently also ported to 0.10~ by @reorder: https://github.com/viacoin/viacoin/commit/9eb949f6f4fa57d1ab22085c05fbd8fafb521995

The mastercoin-tools wrapper should work in general, but there are a few unimplemented functions which were not required to run mastercoin-tools without Omni, and very important, parse_vin(raw_input) must be adjusted, because I cheated and instead of fetching every input, I derived the sender based on the public key in the scriptSig. This doesn't work for P2SH.

There are a few web APIs which could be used as well, most notably probably chain's web API.

One way or the other it would probably be useful for Omniwallet to get away from very specific data formats of one single data provider and instead use one interface and wrappers, if that's possible and applicable.

What do you have in mind when you say "until #1164" is finished?

If the goal is only related to UTXO fetching and everything else is done by Omniwallet and Master Core, then there is also https://github.com/bitcoin/bitcoin/pull/5048.

Please let me know, if there is anything specific I could help with.

[msgilligan]

Thanks, @dexX7 -- I appreciate all the information.

So, @achamely's plan is to create a fully indexed BTC block chain using OmniEngine and PostgreSQL -- and thereby remove the dependency on SX/Obelisk. I created issue #1164 as a placeholder for that work. #1164 is a fairly involved project and may not be finished until mid-January or later and I'm considering alternative approaches to help retire SX/Obelisk earlier.

In general, it makes sense to have an interface in Omniwallet that can work with multiple Blockchain APIs.

I'm not sure how much time I'm going to have available this month, nor am I sure how big of a project this would be, but if @achamely is willing to help with integration and with the code and possible additional help from @dexX7 this might be a feasible short-term project.

[dexX7]

Hehe, don't reinvent the wheel.. ;)

But right now I believe an address indexed modification would satisfy the needs. It can provide directly or indirectly:

1. Transaction histories
2. Unspent outputs (filter transactions based on spent-ness)
3. Balances (sum unspent outputs)

It would probably be useful, if it's compatible with Master Core, to avoid running two daemons. This is something I can check. If you need anything else, say for example "try to find a public key based on a public key hash", I'd have some ideas, but since I only used Omniwallet as user, I'm not sure, what the actual requirements are for a transition away from SX/Obelisk.

[achamely]

@dexx just took a look and right now it looks like there are a few legacy tool functions which use sx/obelisk that would need to be replaced/updated: get_addr_from_key(key) [Private or pubkey] get_json_tx(raw_tx, tx_hash='unknown hash') get_pubkey(addr) get_utxo(addr, value) mktx(inputs_outputs) rawscript(script)

I know we have some different options available and will be looking into the best option we can use

[dexX7]

I rebased the addrindex onto Master Core and created https://github.com/mastercoin-MSC/mastercore/pull/248 to keep track of compatibility.

The listed calls are actually less than expected. Let me take a look at this later.. I'll update.