Provide ability to paginate transactions similar to invoices and payments

[mrfelton]

Background

Payments and Invoices have standard controls for pagination that enable you to specify how many items to receive and what index to start from. Transactions have a different set of pagination controls that enable you to specify a start height and end height.

This makes it challenging to build a UI that has decent pagination of all core activity types (transactions, invoices and payments) since the mechanism to fetch the data varies depending on what data type is being fetched.

For Payments and Invoices I can easily fetch a certain 100 items. But for Transactions there is no way for me to fetch a certain block of 100 items - I don't know wether blocks X through Y contain any transactions at all, so finding the last 100 transactions can only be done through a process of trial and error.

The best I can do with the current api is to choose a particular start and end height to fetch transactions between and then count how many were fetched. If there were less than 100 then try fetching some more. Or, if there were more than 100, truncate the result. With an approach like this, I might need to make thousands of calls to GetTransactions just to find the last 100 transactions since I'd effectively have to keep calling GetTransactions with a lower and lower start and end height all the way back until the start of time.

Your environment

• version of lnd: 0.11
• which operating system (uname -a on *Nix): all
• version of btcd, bitcoind, or other backend: any
• any other relevant environment details

Steps to reproduce

Try building a paginator that pages through transactions in batches of 100.

Expected behaviour

I should be able to specifically ask for 100 transactions form a given offset.

Actual behaviour

I have to do a process of trial and error in order to fetch only the last 100 transactions, or transactions 100-200 etc.

[mikuhl-dev]

Need this, very hard to display transactions alongside invoices and payments, as transactions do not support pagination, and transactions use block height for filter instead of dates.

[alexbosworth]

This is important also to avoid timeout issues, although the workaround of block heights could help? Maybe there is also a documentation aspect to this on how to use block heights most efficiently when normal paging isn't available @saubyk

[Abdulkbk]

I will be working on addressing this issue

[Abdulkbk]

I have researched this issue, including how pagination controls were implemented in the payments here and invoices here RPCs. From what I have observed for listchaintxns RPC, transactions are retrieved from a node's wallet using an external package btcwallet.

I have a question: would it be necessary to open a pull request in btcwallet to modify the GetTransactions function to accommodate the new controls that will be added to the listchaintxns RPC? Or is there a better way to achieve this?

cc: @mrfelton @alexbosworth @Roasbeef

[Roasbeef]

Hi @Abdulkbk you're correct that in order to make this efficient for transactions, we'd need to modify GetTransaction in btcwallet. Right now it's based solely on start and end height, but we also want it to be aware of the number of transaction to return, and also an offset. You should be able to use a database cursor to handle that.

[Abdulkbk]

Hi @Abdulkbk you're correct that in order to make this efficient for transactions, we'd need to modify GetTransaction in btcwallet. Right now it's based solely on start and end height, but we also want it to be aware of the number of transaction to return, and also an offset. You should be able to use a database cursor to handle that.

Do you mean I should be able to use a database cursor to handle this without necessarily making any changes to GetTransactions in btcwallet? I want to make sure I understand this before proceeding. Thank you.

[Roasbeef]

So you can use the database interface used in btcwallet to obtain a database cursor.

[Abdulkbk]

I added a PR #8998 addressing this, in case anyone wants to review :)

[saubyk]

I added a PR #8998 addressing this, in case anyone wants to review :)

Hi @Abdulkbk will take it up for review once the dev work starts up for 0.19. It may take a 2-3 weeks