API caching layer

[robby-d]

EDIT: see https://github.com/CounterpartyXCP/counterwallet/issues/369#issuecomment-54640894

right now, for doing things like loading the exchange page, there are a ton of API calls made. even things like refreshing BTC balances can use optimization as well

let's give the API call some thought and refactor for better performance and simplicity. some thoughts here:

• we don't want to just group a bunch of stuff into a few giant API calls, for instance, as we want the API interface to be as useful as possible to other apps. maybe there is some other way we can pipeline/streamline API calls to reduce roundtrips and processing overhead
• enable the use of request keepalives between counterblockd and counterpartyd to improve performance, possibly (CLOSE_WAIT may be an issue with this, as it was before, but given that we're not using NR agent anymore with counterblockd, it's probably not anymore)

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/3810349-api-caching-layer?utm_campaign=plugin&utm_content=tracker%2F542579&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F542579&utm_medium=issues&utm_source=github).

[ouziel-slama]

I think the priority for optimization is to make a cache valid one block: the cache is generated at the first query, and erased next block. So almost all the time, counterblockd will return only static json. What I have in mind is to make a python decorator, something like @block_cached, and decorate all necessary functions with it.

[ghost]

Great ideas

[ghost]

@JahPowerBit , does the underlying mechanism prevent cache utilization if counterblockd falls behind?

[robby-d]

@JahPowerBit ...yes, that's very simple caching that can be implemented quickly....i.e. cached data is accumulated at the start of a new block reading, and then whatever is cached is invalidated at the time the next block rolls in... selective invalidation (which would allow data to hang around for as long as it doesn't change) is really slick, but is much more complex because you have to maintain dependency chains so you know what you have to invalidate when something changes.... I agree, this simple approach will work well at first at least.

let's try the simple caching first, and then see where that gets us. however, the one concern there is that with multiserver setups cache efficiency goes down. we could "fix" this by using a single redis server, but it becomes not highly available. or, redis has master/slave configs, we could use that. we can cross that bridge when we get there. counterblockd already has very basic redis caching support ...it just needs to be enhanced a bit and tested more fully

I'd still like to take a look and see if there are any commonly called API calls that can be a) realistically collapsed down, or b) made to work with multiple addresses, and then if there are counterblockd -> counterpartyd-level optimizations we could do (involving custom sql queries via sql(), for instance). we should just focus on the top 10% of used API calls here

[ouziel-slama]

but is much more complex because you have to maintain dependency chains

I think, there is no problems if we clear all the cache each block.

the one concern there is that with multiserver setups cache efficiency goes down

If the cache is down, the whole server should be considered down, and CW managed already this case (switch to another server)

[ouziel-slama]

Just did a push in Counterblockd with a decorator @block_cache for all DEX functions called by the API.

[robby-d]

as we discussed, the next steps would be:

• make this work with redis instead
• make is save the cached data with a key that is compatible with nginx redis querying from https://github.com/CounterpartyXCP/counterpartyd_build/blob/master/dist/linux/nginx/counterblock_api_cache.inc ...this will allow the cached data to be queried from nginx directly, without having to even hit python (fast as hell)
• decorate the rest of the API functions