search

ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.81k stars 890 forks source link

Paginate rpc commands that return long lists #4687

Open mb300sd opened 3 years ago

mb300sd commented 3 years ago

Mentioned briefly in #4305, now coming up again over on RTL ( https://github.com/Ride-The-Lightning/RTL/issues/737 )

lnd has an index/count option, something similar would be very nice to have on c-lightning.

whitslack commented 3 years ago

Concept NACK.

Pagination against a live data source is inherently racy. If you need to paginate a large JSON array, capture the entire array into a file and use a JSON manipulation tool (I highly recommend jq) to select subranges from it as you desire.

mb300sd commented 3 years ago

Maybe a before x/after y timestamp option would be better then, basically anything to avoid having to return millions of results every time listforwards is called.

whitslack commented 3 years ago

@mb300sd: Hint: you can also just query the database directly. And if you're using PostgreSQL as your backend, you can add a trigger that notifies you of any insertions or updates to the forwarded_payments table as they happen.

zerofeerouting commented 2 years ago

I would really like to emphasize the importance of this. Especially with listforwards we're talking hundreds of thousands, if not millions of entries. Would also be great being able to filter results doing start_date=-2d or end_date=-12h.

cdecker commented 2 years ago

I think @rustyrussell was going to design a generic pagination system that could leverage the DB that is backing most of the queries to limit results, order by field and offset based on the ordering column. Maybe he can share the design in a quick doc and maybe we can get this unblocked a bit :-)