addtobatch response + addtobatch/batchspend enhancements

[xsats]

Awesome project - I’ve been poking around and looking forward to exploring more!

I have a few questions relating to batching and spending transactions, which I’d really appreciate some insight on.

As far as I can tell, calling the addToBatch() api doesn’t currently return a response, so I’d like to know; what is the best way to confirm that a new tx is staged in an upcoming batch? Is there a way to query the transactions in (an) upcoming batch(es)? Alternatively, if there is no callback/api what’s the best way to view txs sitting in a pending batch?

On a related note, before calling batchSpend() is there currently a convenient way to return/validate txs in a batch before broadcasting?

I’m also interested in how fees are handled through the api. What are the best practices relating to tx fee estimation and configuration within cyphernode and where does one configure the fees? Sorry if this has already been detailed - my digging hasn’t yet successfully uncovered this info.

Thanks in advance!

[Kexkey]

Hi @xsats ! Thanks for the questions. Funny you're asking those because I was about to revisit and add features to the batching endpoints this week! Here is what I'm about to do, tell me if you think it makes sense and is useful to you.

• Add a useful response to the addtobatch call;
• Add a webhook Url to each addtobatch call, so Cyphernode will call them when the batchspend endpoint is called;
• Add a confTarget argument to the batchspend endpoint;
• Add a estimatesmartfee endpoint to Cyphernode;
• Add batching groups: a way to have multiple batches. For example, a batch with low fees executed hourly, a batch with medium fees executed daily and a batch with high fees executed weekly;
• Add an endpoint to dequeue an output from a batch;
• Add endpoints to get information about batches/pending batches/outputs.

My plan is to develop all that this week, so stay tuned. Let me know if you think there could be useful additions to all that.

BTW currently the fees are controlled by using the confTarget arg to the spend endpoint as well as by adding a txconfirmtarget in bitcoin.conf to set the default.

[xsats]

Thanks a lot for the detailed response @Kexkey !

You read my mind :) these additions are exactly the type of thing I was looking for, and more! Here are some of my thoughts and comments, keeping your same order:

• I'm curious about your thoughts on what the response would contain? On top of the success boolean, it could be useful to return some info on the particular batch (e.g. # of outputs currently in batch, timestamp of oldest output in batch (proxy for new batch creation timestamp), value of pending outputs in batch)
• Love it. It may be useful for addtobatch to have a urlLabel parameter to indicate which url the notification associated with that particular output would be sent to. in this scenario, I'm thinking that a list of labelled urls could be created separated in a config file. My reasoning here is there is likely (at least in the way I am thinking of making use of this feature - you probably have better ideas!) to be a lot of duplication in callback urls between outputs in a batch, so instead of sending multiple notifications to the same callback endpoint when a single batchspend endpoint is called, there would be a json payload containing info on each of the outputs within the batch, for each of the urls designated by the urlLabels referenced by outputs within the batch. The label could be an easy way of organising and referring to different urls. My thinking is that this would be more efficient than having duplicate notifications dispatched to the same url for outputs within the same batch. I'm sure there is probably a better way of doing this, but I thought I'd raise it for discussion as I'd like to hear your thoughts on whether this could be a good idea and whether it is an optimal approach. Alternatively, I suppose that the duplication prevention logic could also be associated with the batchspend endpoint. But anyway, I'll stop prattling.
• This sounds great. It might also be useful for there to be an option to specify an exact fee in terms of sats/byte, and so allow more flexibility in specifying the fee and also for custom fee calculation approaches.
• The estimatesmartfee endpoint also sounds great! I'm intrigued to learn more about what would go on under the hood there
• Really like the batching groups feature. The ability to construct and add to multiple batches with different 'priorities' simultaneously would be a huge plus. I'm curious how you plan to implement this. I guess there could be a few different ways to do it e.g. with a boolean parameter to the addtobatch endpoint called newbatch and a second options parameter that (1) if newbatch = true; specifies the properties of the new batch (e.g. below *); or (2) if newbatch = false; specifies the label/batchid of the existing batch that the output should be added to. Another way might be to add another endpoint createnewbatch that contains the logic for batch creation - maybe that way could be a bit cleaner/intuitive. I hope you don't mind my brainstorming here, I'm trying not to drivel too much!
• ability to remove an output from a batch would definitely be useful. Perhaps it could also be useful in the future to have the option to move an output to a different batch with a higher/lower priority or simply by 'batchid'
• Would definitely find a getbatchinfo endpoint very useful, that returns the contents of a particular queued or past batch using a batchid or otherwise. It may also be useful for each output in a batch to have an associated timestamp for when it was added to the batch, that could also be included in the response to getbatchinfo.

*One additional (lesser priority) feature I'd like to suggest is the ability to configure thresholds (e.g. in a config file) for batch age (and possibly also number of outputs in and/or value of particular batch), and add a separate webhook type that fires a callback to notify when the threshold is reached/exceeded. This could be useful to help keep track of batches and make sure that they remain within the desired bounds in terms of batch interval/value when broadcasted - "spend me, I'm getting old/fat". Batch could be checked on the addition of each new output before the response is sent. To attempt to improve privacy/reduce potential for chain snoopers, perhaps it may also be useful if there is an option to configure a randomly generated variance about the configured threshold e.g. set a batch age threshold of 30mins with variance 0.2; when the first output is added to a batch, a randomly generated threshold between 24-36mins is associated with that particular batch (notifierthreshold property or something like that). I suppose there could be the ability to set multiple thresholds in the config, to account for batches with different priorities/fee rates. Alternatively this logic could be included in a createnewbatch endpoint, which takes arguments like batchagethreshold, thresholdvariance - this method may be superior in terms of control and ease of use.

Thanks for the info on controlling fees, that makes sense :)

Sorry this turned into an essay! I hope you find some of this useful. I need to brush up on shell so I can try to start contributing something useful to cyphernode!

[xsats]

I see now that @FrancisPouliot has already written in some detail (#6) how the batching implementation might look - really like all of these ideas e.g. batch scheduling, frequency, batch status and index and a lovely comprehensive list of batch data types for lookup!

On top of that the PSBT endpoints raised in #7 sound ideal!

Are the specs outlined in #6 and #7 still roughly what you plan to implement? Very excited about all of this!

[FrancisPouliot]

Hey! The batching schedule idea was initially just a brainstorm. The route we ended up doing was having the client apps manage their own scheduling

CC @Kexkey and I have discussed this recently

But im still open with having the scheduler within cyphernode itself

[Kexkey]

@xsats unfortunately, there's no way to explicitly set the feerate on the sendmany RPC call in Bitcoin Core. There is a 3 year old PR for that though: https://github.com/bitcoin/bitcoin/pull/11413

For now, instead of using createrawtransaction, fundrawtransaction and sendrawtransaction to achieve that, I will wait for the PR to be merged in Bitcoin Core and will only support confTarget to set the transaction fee, at least for phase 1 of this feature,.

Yes, that means there will be a phase 2 for this feature :) because I want to have the basic stuff quickly released and the most advanced ideas later.

[Kexkey]

Little followup, things have been a little bit slow during July, but the features/batching branch is pretty stable. Here is what we decided to do: CN will have the basic batching functionalities and I'm developing a new CypherApp called Batcher that will hide the complexity of the batching in CN and add flexibility. Kind of a 2nd layer.

Example of added features in the Batcher: automatic merging of same outputs, handling of different webhooks when outputs merged, added data to the webhooks body, simplify the interface (batchRequestId + batchId vs batcherId + outputId + txid), batch scheduling, amount threshold, etc.

Also, the Batcher cypherapp is coded in TypeScript, which is easier to contribute to and clearer than JavaScript by typing everything. My first TypeScript experience!

We are slowly moving to using json-rpc everywhere, planning to have 100% json-rpc compliance in the CN v1 API. Right now we have a hybrid API which is kind of weird.

Anyway, Batcher v1 coming soon!

P.S.: also planning Cyphernode v0.5.0 that will include the enhanced batching stuff.

[xsats]

Closing this, now that the cn batcher is here!