List of auctions, bids and reveals suffers from the same issues as tx history did (#888). This is document describes the potential API and reindexing necessary to get the similar features as the tx history. The indexes are categorized two ways: time index and count index. Count index refers to the ordered list of items, where adding new entries from new blocks does not change the ordering of the previous ones. Wallet TX pagination uses Block height + tx index in the block - for this, and time index is references into this.
Names have four separate endpoints and entries in the db: namestate, auction, bid and reveal. The indexing and pagination should be consistent across these three. Different from txs, names are only indexed for the confirmed txs. All these have two separate indexes for the owned and non-owned. Names and auctions currently are the same, but should be separate.
Time index note: Because all entries are only indexed by block, their time will be same as the block median time. So instead of having separate time indexes, we can introduce new index for median time -> block height, which can be used for querying by time. (Note: Confirmed tx index for time could also be removed and use time -> height index)
TX Pagination uses tx hashes as the cursor, unfortunately same can't be done with the auctions. Same name can appear several, after names expire - which are essentially different auctions. We can instead use block height + tx index (returned via API/Seek) in order to iterate over auctions/bids/reveals. NOTE: TXs use block height + index internally.
Sorted index for the auctions can be fully expressed by combination of height + tx index (as names themselves are not unique and have multiple auctions, we can't use them as cursor). We will go into details per item below.
NOTE: layout keys not chosen for this document, it must be done on implementation step. Listed layouts will use descriptive names instead of one symbol prefixes.
Wherever height + tx-index occurs, it can be aliased to txhash and use TX pagination entries to look up height and index (TXCount). It makes no sense for some: E.g. auctions. But this could be useful for bids/reveals.
Names (Latest namestate)
Names are the latest state that wallet knows from the chain. This is not an auction, there can be multiple auctions per name. NOTE: This endpoint wont have account to it, as namestate is more general state.
uniqueIndex is there to make sure, next/prev page does not get shifted when new name is indexed.
This is used to iterate over all names in consistent manner.
Given a name cursor, we can internally resolve the index, this is used by owned/non-owned. Also ensures we don't have name with two indexes.
HTTP API:
GET /wallet/:id/name - get latest namestates, but with new parameters.
params
reverse - boolean - should we reverse the list.
limit - number - number of namestates to list at once.
after - string - list of namestates after the name.
owned - boolean - list of namestates that are owned.
Uses layout.namestateByIndex for iteration.
Uses layout.namestateOwnedByIndex if owned is set.
Uses layout.namestateIndexByName[nameHash] to get from cursor to internal iteration.
This endpoint does not have time param, as this refers to the latest state instead of auction.
Auctions are almost same as names, but they can happen several times if the name expires. They can be uniquely identified by the block height + name, but tx-index is there to ensure the ordering follows blockchains ordering in a block.
or layout.auctionsByNameAccount[account][nameHash][height] -> tx-index (tx-index does not matter for sorting)
HTTP API:
GET /wallet/:id/auction - get all wallet auctions. (Must no longer include bids/reveals)
params:
account - optional string - instead of whole wallet, do specific account.
reverse - boolean - should we reverse the list.
limit - number - number of auctions to list at once.
time - number - list auctions after timestamp(or before reverse)
Uses layout.heightByTime[time]
after - height:txindex - cursor is little bit tricky here. Cursor alternatives:
height <<< ceil(log2(MAX_TX_IN_BLOCK)) | tx-index - gives a single number.
name:height - this has advantage that the namestate json contains these two.
(Or afterName, afterHeight)
Uses layout.auctionsByCount[height][index] for iteration (with or w/o after).
Uses layout.auctionsByCountAccount[height][index] if account is set.
GET /wallet/:id/auction/:name - Get all wallet auctions for the name. (Must no longer include bids/reveals)
params
account - optional string - instead of whole wallet, do specific account lookup.
reverse - boolean - should we reverse the list.
limit - number - number of auctions to list at once.
time - number - list auctions after timestamp(or before reverse)
Uses layout.heightByTime[time]
after - height.
Uses layout.auctionsByName[name][height] to iterate over name auctions (with or w/o after).
Uses layout.auctionsByNameAccount[account][nameHash][height] if account is set.
GET /wallet/:id/auction/:name/:height - Get specific auction.
params
account - optional string - make sure we check specific account only.
Uses layout.auctionsByName[nameHash][height] to grab specific auction.
Uses layout.auctionsByNameAccount[account][nameHash][height] if account is set.
This also enables Names endpoint:
GET /wallet/:id/name/:name/:height - to get namestate for specific auction, not latest. Similar to historical proofs, but namestates.
Uses layout.auctionsByName[nameHash][height] to get indexes.
Bids
Bids are different from auctions. Block and TX can have multiple bids for the wallet, so if we want sorted set we need to include all this. Another important note: Not all bids for the same name are part of the same auction, so they need to be separated as well.
limit - number - number of auctions to list at once.
time - number - list bids after timestamp
Uses layout.heightByTime[time]
after - This could be built using height + tx-index + out-index.
e.g. ?height=200100&txIndex=10&outIndex=5, unfortunately, similar to Auctions API, it's not easy to specify these manually and only more sense when used on iteration, built from last entry.
owned - Filter the bids that are ours only.
Uses layout.bidByOutput[height][tx-index][out-index] index for iteration.
Uses layout.bidByOutputAccount[account][height][tx-index][out-index] if account is set.
Uses layout.bidByOutputOwned[height][tx-index][out-index] if owned is set.
Uses layout.bidByOutputOwnedAccount[account][height][tx-index][out-index] if account and owned is set.
GET /wallet/:id/bid/:name - get all bids specific to name. (Include all auctions)
params
account filter by account.
Uses Account Indexes instead of normal indexes for all operations.
reverse - boolean - should we reverse the list.
limit - number - number of auctions to list at once.
time - number - list bids after timestamp
Uses layout.heightByTime[time]
after - This is same as all bids: height + tx-index + out-index
limit - number - number of auctions to list at once.
time - number - list bids after timestamp
Uses layout.heightByTime[time]
after - This could be built using height + tx-index + out-index.
e.g. ?height=200100&txIndex=10&outIndex=5, unfortunately, similar to Auctions API, it's not easy to specify these manually and only more sense when used on iteration, built from last entry.
owned - Filter the reveals that are ours only.
Uses layout.revealByOutput[height][tx-index][out-index] index for iteration.
Uses layout.revealByOutputAccount[account][height][tx-index][out-index] if account is set.
Uses layout.revealByOutputOwned[height][tx-index][out-index] if owned is set.
Uses layout.revealByOutputOwnedAccount[account][height][tx-index][out-index] if account and owned is set.
GET /wallet/:id/reveal/:name - get all reveals specific to name. (Include all auctions)
params
account filter by account.
Uses Account Indexes instead of normal indexes for all operations.
reverse - boolean - should we reverse the list.
limit - number - number of auctions to list at once.
time - number - list reveals after timestamp
Uses layout.heightByTime[time]
after - This is same as all reveals: height + tx-index + out-index
Wallet pagination for auctions
List of auctions, bids and reveals suffers from the same issues as tx history did (#888). This is document describes the potential API and reindexing necessary to get the similar features as the tx history. The indexes are categorized two ways:
timeindex andcountindex. Count index refers to the ordered list of items, where adding new entries from new blocks does not change the ordering of the previous ones. Wallet TX pagination uses Block height + tx index in the block - for this, and time index is references into this. Names have four separate endpoints and entries in the db:namestate,auction,bidandreveal. The indexing and pagination should be consistent across these three. Different from txs, names are only indexed for the confirmed txs. All these have two separate indexes for the owned and non-owned. Names and auctions currently are the same, but should be separate.Time index note: Because all entries are only indexed by block, their time will be same as the block median time. So instead of having separate time indexes, we can introduce new index for median time -> block height, which can be used for querying by time. (Note: Confirmed tx index for time could also be removed and use time -> height index)
TX Pagination uses tx hashes as the cursor, unfortunately same can't be done with the auctions. Same name can appear several, after names expire - which are essentially different auctions. We can instead use block height + tx index (returned via API/Seek) in order to iterate over auctions/bids/reveals. NOTE: TXs use block height + index internally. Sorted index for the auctions can be fully expressed by combination of height + tx index (as names themselves are not unique and have multiple auctions, we can't use them as cursor). We will go into details per item below.
NOTE: layout keys not chosen for this document, it must be done on implementation step. Listed layouts will use descriptive names instead of one symbol prefixes.
General block time -> height index
layout.heightByTime[time] -> height- Uses monotonic time.General considerations
height + tx-indexoccurs, it can be aliased totxhashand useTX paginationentries to look upheightandindex(TXCount). It makes no sense for some: E.g. auctions. But this could be useful for bids/reveals.Names (Latest namestate)
Names are the latest state that wallet knows from the chain. This is not an auction, there can be multiple auctions per name. NOTE: This endpoint wont have account to it, as namestate is more general state.
Indexes:
layout.namestateUniqueIndex -> number.layout.namestateByIndex[uniqueNameIndex++] -> namestatelayout.namestateOwnedByIndex[uniqueNameIndex] -> dummylayout.namestateIndexByName[nameHash] -> uniqueNameIndexHTTP API:
GET /wallet/:id/name- get latest namestates, but with new parameters.paramsreverse- boolean - should we reverse the list.limit- number - number of namestates to list at once.after- string - list of namestates after thename.owned- boolean - list of namestates that areowned.layout.namestateByIndexfor iteration.layout.namestateOwnedByIndexifownedis set.layout.namestateIndexByName[nameHash]to get from cursor to internal iteration. This endpoint does not havetimeparam, as this refers to the latest state instead of auction.GET /wallet/:id/name/:name- same as it was.layout.namestateIndexByName[nameHash]->layout.namestateByIndex[index]Auctions
Auctions are almost same as names, but they can happen several times if the name expires. They can be uniquely identified by the block height + name, but tx-index is there to ensure the ordering follows blockchains ordering in a block.
Indexes:
layout.auctionsByCount[auction-height][tx-index][nameHash] -> namestatelayout.auctionsByName[nameHash][auction-height][tx-index] -> dummy(uses auctionsByCount to resolve)layout.auctionsByName[nameHash][auction-height] -> tx-index(tx-index does not matter for sorting)Account indexes:
layout.auctionsByCountAccount[account][height][tx-index][nameHash] -> dummylayout.auctionsByNameAccount[account][nameHash][height][tx-index] -> dummylayout.auctionsByNameAccount[account][nameHash][height] -> tx-index(tx-index does not matter for sorting)HTTP API:
GET /wallet/:id/auction- get all wallet auctions. (Must no longer include bids/reveals)params:account- optional string - instead of whole wallet, do specific account.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list auctions after timestamp(or before reverse)layout.heightByTime[time]after-height:txindex- cursor is little bit tricky here. Cursor alternatives:height <<< ceil(log2(MAX_TX_IN_BLOCK)) | tx-index- gives a single number.name:height- this has advantage that the namestate json contains these two.afterName,afterHeight)layout.auctionsByCount[height][index]for iteration (with or w/oafter).layout.auctionsByCountAccount[height][index]ifaccountis set.GET /wallet/:id/auction/:name- Get all wallet auctions for the name. (Must no longer include bids/reveals)paramsaccount- optional string - instead of whole wallet, do specific account lookup.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list auctions after timestamp(or before reverse)layout.heightByTime[time]after-height.layout.auctionsByName[name][height]to iterate over name auctions (with or w/oafter).layout.auctionsByNameAccount[account][nameHash][height]ifaccountis set.GET /wallet/:id/auction/:name/:height- Get specific auction.paramsaccount- optional string - make sure we check specific account only.layout.auctionsByName[nameHash][height]to grab specific auction.layout.auctionsByNameAccount[account][nameHash][height]ifaccountis set.This also enables Names endpoint:
GET /wallet/:id/name/:name/:height- to get namestate for specific auction, not latest. Similar to historical proofs, but namestates.layout.auctionsByName[nameHash][height]to get indexes.Bids
Bids are different from auctions. Block and TX can have multiple bids for the wallet, so if we want sorted set we need to include all this. Another important note: Not all bids for the same name are part of the same auction, so they need to be separated as well.
Indexes:
layout.bidByOutput[height][tx-index][out-index] -> BlindBidlayout.bidByOutputOwned[height][tx-index][out-index] -> dummylayout.bidByName[nameHash][height][tx-index][out-index] -> dummylayout.bidByNameOwned[nameHash][height][tx-index][out-index] -> dummyAccount indexes:
layout.bidByOutputAccount[account][height][tx-index][out-index] -> BlindBidlayout.bidByOutputOwnedAccount[account][height][tx-index][out-index] -> dummylayout.bidByNameAccount[account][nameHash][height][tx-index][out-index] -> dummylayout.bidByNameOwnedAccount[account][nameHash][height][tx-index][out-index] -> dummyHTTP API:
GET /wallet/:id/bid- get all wallet bids.paramsaccountfilter by account.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list bids after timestamplayout.heightByTime[time]after- This could be built usingheight+tx-index+out-index.?height=200100&txIndex=10&outIndex=5, unfortunately, similar to Auctions API, it's not easy to specify these manually and only more sense when used on iteration, built from last entry.owned- Filter the bids that are ours only.layout.bidByOutput[height][tx-index][out-index]index for iteration.layout.bidByOutputAccount[account][height][tx-index][out-index]ifaccountis set.layout.bidByOutputOwned[height][tx-index][out-index]ifownedis set.layout.bidByOutputOwnedAccount[account][height][tx-index][out-index]ifaccountandownedis set.GET /wallet/:id/bid/:name- get all bids specific to name. (Include all auctions)paramsaccountfilter by account.Account Indexesinstead of normal indexes for all operations.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list bids after timestamplayout.heightByTime[time]after- This is same as all bids:height+tx-index+out-indexowned- Filter the bids that are ours only.layout.bidByName[nameHash][height][tx-index][out-index]index.layout.bidByNameAccount[account][nameHash][height][tx-index][out-index]ifaccountis set.layout.bidByNameOwned[nameHash][height][tx-index][out-index]ifownedis set.layout.bidByNameOwnedAccount[account][nameHash][height][tx-index][out-index]ifownedandaccountis set.This also enables Auctions endpoint:
GET /wallet/:id/auction/:name/:height/bid- get all bids specific to single auction.paramsaccountfilter by account.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list bids after timestamplayout.heightByTime[time]after- This same as well:height+tx-index+out-indexowned- Filter the bids that are ours only.layout.auctionsByName[name][height]to get the start height, end height can be grabbed with treeInterval + biddingPeriodlayout.bidByName[nameHash][height][tx-index][out-index]layout.bidByNameAccount[account][nameHash][height][tx-index][out-index]ifaccountis set.layout.bidByNameOwned[nameHash][height][tx-index][out-index]ifownedis set.layout.bidByNameOwnedAccount[account][nameHash][height][tx-index][out-index]ifownedandaccountis set.Reveals
Reveals are similar to bids. There can be several reveals in a single block and in a single tx.
Indexes:
layout.revealByOutput[height][tx-index][out-index] -> BidReveallayout.revealByOutputOwned[height][tx-index][out-index] -> dummylayout.revealByName[nameHash][height][tx-index][out-index] -> dummylayout.revealByNameOwned[nameHash][height][tx-index][out-index] -> dummyAccount indexes:
layout.revealByOutputAccount[account][height][tx-index][out-index] -> BidReveallayout.revealByOutputOwnedAccount[account][height][tx-index][out-index] -> dummylayout.revealByNameAccount[account][nameHash][height][tx-index][out-index] -> dummylayout.revealByNameOwnedAccount[account][nameHash][height][tx-index][out-index] -> dummyHTTP API:
GET /wallet/:id/reveal- get all wallet reveals.paramsaccountfilter by account.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list bids after timestamplayout.heightByTime[time]after- This could be built usingheight+tx-index+out-index.?height=200100&txIndex=10&outIndex=5, unfortunately, similar to Auctions API, it's not easy to specify these manually and only more sense when used on iteration, built from last entry.owned- Filter the reveals that are ours only.layout.revealByOutput[height][tx-index][out-index]index for iteration.layout.revealByOutputAccount[account][height][tx-index][out-index]ifaccountis set.layout.revealByOutputOwned[height][tx-index][out-index]ifownedis set.layout.revealByOutputOwnedAccount[account][height][tx-index][out-index]ifaccountandownedis set.GET /wallet/:id/reveal/:name- get all reveals specific to name. (Include all auctions)paramsaccountfilter by account.Account Indexesinstead of normal indexes for all operations.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list reveals after timestamplayout.heightByTime[time]after- This is same as all reveals:height+tx-index+out-indexowned- Filter the reveals that are ours only.layout.revealByName[nameHash][height][tx-index][out-index]index.layout.revealByNameAccount[account][nameHash][height][tx-index][out-index]ifaccountis set.layout.revealByNameOwned[nameHash][height][tx-index][out-index]ifownedis set.layout.revealByNameOwnedAccount[account][nameHash][height][tx-index][out-index]ifownedandaccountis set.This also enables Auctions endpoint:
GET /wallet/:id/auction/:name/:height/reveal- get all reveals specific to single auction.paramsaccountfilter by account.reverse- boolean - should we reverse the list.limit- number - number of auctions to list at once.time- number - list reveals after timestamplayout.heightByTime[time]after- This same as well:height+tx-index+out-indexowned- Filter the reveals that are ours only.layout.auctionsByName[name][height]to get the start height, end height can be grabbed with treeInterval + revealdingPeriodlayout.revealByName[nameHash][height][tx-index][out-index]layout.revealByNameAccount[account][nameHash][height][tx-index][out-index]ifaccountis set.layout.revealByNameOwned[nameHash][height][tx-index][out-index]ifownedis set.layout.revealByNameOwnedAccount[account][nameHash][height][tx-index][out-index]ifownedandaccountis set.