Display TLV data in listinvoices

[Sjors]

When receiving a keysend payment I can see it under listinvoices. But this doesn't show any custom TLV fields.

I'm particularly interested in the fields used by Podcasting 2.0: https://github.com/satoshisstream/satoshis.stream/blob/main/TLV_registry.md#field-7629169

[cdecker]

We have one issue with this, namely that despite the name, TLV fields are not typed as such. They are just blobs of binary data, that are interpreted as integers, hashes, signatures, etc by any program that has more knowledge about the T in the acronym.

So a generic implementation would just show type, length (redundant really in JSON results) and a hex-encoded binary value. Would that still be acceptable?

[Sjors]

It's a start. I could probably pipe that through JQ and then some tools to decipher the hex value. Maybe plugins can do that too.

[daveajones]

We have one issue with this, namely that despite the name, TLV fields are not typed as such. They are just blobs of binary data, that are interpreted as integers, hashes, signatures, etc by any program that has more knowledge about the T in the acronym.

So a generic implementation would just show type, length (redundant really in JSON results) and a hex-encoded binary value. Would that still be acceptable?

The hex encoded strings are what we are parsing when pulling out the tlv payload on incoming keysend transactions for the Podcast Index node. It’s not difficult (bin2hex —> json_decode) and provides tremendous listener stats to see what podcast/episode was being listened to, and exactly what time stamp within it that the payment was sent.

The podcast industry currently works on “download stats” which are highly inaccurate. It’s basically guesswork. This would revolutionize that as a standard for listener metrics.

[cdecker]

4610 is the first step towards this. It adds experimental support for extratlvs, i.e., hex-encoded TLVs that are not interpreted and stored internally yet, with opt-in to even TLV types on the command line. They are currently only being exposed in the invoice_payment hook for testing, but if the format looks ok to users we can add them to the corresponding notification, and start storing them in the DB as well, so listinvoices can retrieve and display them.

[cnixbtc]

a generic implementation would just show type, length (redundant really in JSON results) and a hex-encoded binary value. Would that still be acceptable?

I think that would be fine. It should be easy for consuming apps to interpret the TLV type and decode the hex values accordingly.

They are currently only being exposed in the invoice_payment hook for testing

That's already really helpful -- thanks for that! :)

I've written a small plugin for lightningd that watches for Podasting 2.0 TLVs in the invoice_payment hook and stores them in a local db so that they can be retrieved again later on: cln-podcast-payments.

It's not much but should be an acceptable workaround until Core Lightning can natively do that.

but if the format looks ok to users we can add them to the corresponding notification, and start storing them in the DB as well, so listinvoices can retrieve and display them.

That'd be great, imo! As I said, I feel like the binary format is fine enough. Having access to TLVs in listinvoices would make the use of a separate plugin obsolete and by that make everything easier for people to use.

[cdecker]

I wonder if this can now be considered as solved by https://github.com/ElementsProject/lightning/pull/5619. That PR adds the longest string extracted from TLVs into the invoice we use to track spontaneous payments.

[jb55]

yes I see podcast keysend data in listinvoices now which is pretty cool

[Sjors]

Nice, I'll give this a try after the final v22.11 release.

[dennisreimann]

Great news, looking forward to integrate this into our BTCPay Lightning lib.

[dennisreimann]

I wonder if this can now be considered as solved by https://github.com/ElementsProject/lightning/pull/5619

@cdecker As far as I see, the description with the keysend: part contains only the 133773310 TLV key. Is there a way to also get access to the other custom TLV records?

[Sjors]

This jq query gets a long list of podcast sats:

lightning-cli listinvoices | jq -c '[.invoices[] | select(.status == "paid") | select(.label | startswith("keysend-")) | select(.description | startswith("keysend:"))] | .[].description[9:] | fromjson' | jq --slurp

This query doesn't seem to find all boosts though, e.g. adding | select(.action == "boost") only returns one result, even though the episode in question has 6 boosts.

I'm guessing the other boosts live under 7629173.

[dennisreimann]

@Sjors Are you able to access TLV records other than 133773310 at all? Just checked again with CLN 22.11 and I only get those. Pointers appreciated :)

Also curious to know if the accept-htlc-tlv-types option has to list the types one wants to accept, e.g.

accept-htlc-tlv-types=696969,112111100

[rustyrussell]

Like, seriously... Should we just put every damn TLV in the invoice label? Standards FFS people. Or are there TLVs which are ACTUALLY used which we should just always include?

Otherwise, we may need an explicit new field for this garbage, and give you all the TLVs in hex and let you sort it out :(

Removing this from milestone.

[dennisreimann]

@rustyrussell It'd be nice to include the well known ones. I came across this testing the Podcasting 2.0 fields (7629173, 7629175, 133773310) + wallet integration (696969, 112111100) …

[Sjors]

every damn TLV

That's probably the easiest. Having a parameter to specify one or more is fine by me too, though perhaps that more suitable for a transaction filter (plugin?).

Also curious to know if the accept-htlc-tlv-types option has to list the types one wants to accept

Accepting or rejecting TLV types seems out of scope for here.

[dennisreimann]

every damn TLV

That's probably the easiest. Having a parameter to specify one or more is fine by me too

Maybe combining it with accept-htlc-tlv-types would be an option, I haven't looked at how it works internally though.

Unfortunately the current state of only listing certain TLV records isn't intuitive and as described above one cannot support e.g. the Podcasting 2.0 features fully as there is no API access to most of the records.

[dennisreimann]

Sorry to bump this, but including either the well-known TLV records or the ones listed in accept-htlc-tlv-types (see above) would give us a chance to implement Podcasting 2.0 features on Core Lightning.

If you don't see it that way that's also fine, but then let's manage expectations and close this issue. /cc @cdecker @rustyrussell

[Sjors]

As podcaster I like to stack sats, as the song goes...