[Design] API for Transaction Mining

[jansegre]

Phase 2 of our design to reduce costs involves creating an API for mining transactions that is independent from the main API.

This will require a change in the transaction execution flow on the wallet and ultimately require 3 major components: the tx mining server, new fullnode API calls, wallet update to use the new flow.

Wallet changes

Steps to build and publish a transaction on the wallett are currently:

• generate tx locally (outputs and inputs without signature)
• sign inputs
• [request to fullnode API]: push tx (without nonce and parents the fullnode API will take care of mining)

With the new tx-mining API:

• generate tx locally (outputs and inputs without signature)
• [request to fullnode API]: get parents (new API call)
• sign inputs
• [request to tx-mining API]: create tx mining job (since it returns a time estimate this can be used for UI feedback)
• wait for X time, where X is half the estimated time returned (the wallet can actually use any strategy but this is the recommended)
• [request to tx-mining API]: get job status, go back to previous step if not done (this is another opportunity to update the UI)
• [request to fullnode API]: push tx (now with nonce)

Changes to the fullnode API

We will need to create an API to get fresh parents to use on a new transaction. Alternatively and additionally the fullnode could pre-validate a transaction (and fill-in the parents if needed).

The Transaction Mining Server

To summarize, the layout is:

 +-------+
 |Wallets|
 +-------+
     |
     |      +------------------+
     v      |                  |
 HTTP API+--+ Tx Mining Server +---+Stratum
            |                  |       ^
            +------------------+       |
                                       |
                                   +------+
                                   |Miners|
                                   +------+

• Wallets will call the API to mine transactions, and then call push-tx with the nonce included. The push-tx API should continue to accept transactions without nonce for a while, and in the future only accept txs mined externally.
• We will connnect miners to the Tx mining server stratum, which will replace our only practical use for the Stratum built-in on the fullnode.

The HTTP API should look like this (specific formats, endpoint names, field names are up to change):

• /submit-job: endpoint for the wallet to submit the required data to be mined, the server will return a job-id, and an estimated mining time
• /job-status: the wallet, now in possession of a job-id, can request to check the status of a mining job at any time, these are the possible answers:
  • the job does not exist because either the job-id is incorrect or because it finished too long ago
  • the job is in progress, a new estimate will be returned
  • the job is finished, the nonce is returned, after a while (TBD) this job will be deleted

The reasons for this API to be modeled in a way that the client has to make multiple requests are:

• Simpler implementation
• No long polling requests: makes request throttoling make much more sense, fast responses are the norm, timed-out requests will always be an error (i.e. will never mean that mining could not finish in time)
• Time estimate: the server knows its hashrate and rate of incoming mining requests, which it can use to estimate how long it will take for the mining to finish, this has a lot of UX improvement potential

Future improvements

• This same design could be used to require a server fee
• This same design could also be used to make a mining marketplace (external miners can receive the fees or a portion of the fees)
• The tx-mining-server could push the transaction to the network, this might make sense if the proof-of-work ever becomes high enough that there is a significant fee/mining-time trade off, then it would make sense for a low-fee long-wait option, and it would be more practical for the tx-mining-server to push the transaction so a wallet actively waiting for a job result to push the transaction is not needed

[obiyankenobi]

On /submit-job, what data did you plan to send? We need to make sure people can't abuse or attack the tx miners, possibly by sending a malformed or invalid tx.

[msbrogli]

It seems good to me, but I think we should give at least one example of the requests and responses. We should also add a throttling to both endpoints.

[jansegre]

On /submit-job, what data did you plan to send? We need to make sure people can't abuse or attack the tx miners, possibly by sending a malformed or invalid tx.

Today we won't be any worse off by receiving the same data we receive on the push-tx endpoint

It seems good to me, but I think we should give at least one example of the requests and responses.

I'll work on adding and example.

We should also add a throttling to both endpoints.

We should use the same nginx throttling strategy as used on fullnode APIs.

[pedroferreira1]

For me it's good.

Only one thing was not 100% clear. After sending the first request (/submit-job) the wallets will need to check from time to time if the job is already finished, right?

If that's the case we should include this UI change in the wallet, so we can give a more precise feedback (Solving tx pow. ETA X. Then move to 'Propagate tx to the network'. Then success).

[pedroferreira1]

Another thing that should be discussed is what will be done with the current thin_wallet/send_tokens API. We should deprecate it but keep compatibility because old wallets might still be using it. And as @jansegre said

We will connnect miners to the Tx mining server stratum, which will replace our only practical use for the Stratum built-in on the fullnode.

When we decide that it's good to remove the compatibility with the old API we can think about removing the Stratum built-in on the fullnode.

[obiyankenobi]

We will connnect miners to the Tx mining server stratum, which will replace our only practical use for the Stratum built-in on the fullnode.

Nowadays we still have the stratum api available for miners. I'm not sure how many people are using it, but we need to check that before we can remove it.

[jansegre]

Nowadays we still have the stratum api available for miners. I'm not sure how many people are using it, but we need to check that before we can remove it.

@pedroferreira1 @obiyankenobi I created a card to track the deprecation of the built-in stratum

Another thing that should be discussed is what will be done with the current thin_wallet/send_tokens API.

@pedroferreira1 As currently proposed in HathorNetwork/hathor-core#67 self-mining should be deprecated after a while. I added checkboxes to help track the progress. But I think we can either discus specific detals there or on a new issue, and keep this one just for the design of this new system.

[obiyankenobi]

With the new tx-mining API:

• generate tx locally (outputs and inputs without signature)
• [request to fullnode API]: get parents (new API call)
• sign inputs
• [request to tx-mining API]: create tx mining job (since it returns a time estimate this can be used - for UI feedback)

Parents are not part of the signature, so we don't need to get them before contacting the mining server. The full node could add them as part of /push-tx.

[pedroferreira1]

Parents are not part of the signature, so we don't need to get them before contacting the mining server. The full node could add them as part of /push-tx.

Yes, we don't need the parents for the signature but the /push-tx expects the tx already mined (with the nonce). And to mine the tx we need the parents, so we need to add the parents before sending to the mining server, I think.

@pedroferreira1 As currently proposed in HathorNetwork/hathor-core#67 self-mining should be deprecated after a while. I added checkboxes to help track the progress. But I think we can either discus specific detals there or on a new issue, and keep this one just for the design of this new system.

Ok, I agree with you. We should discuss the deprecation in another issue

For me the design is good to close. Can you just confirm this question, please?

Only one thing was not 100% clear. After sending the first request (/submit-job) the wallets will need to check from time to time if the job is already finished, right?

[msbrogli]

I like the design, but I think we should get into the pooling strategy. Should we check the status every X seconds, where X is half the expected mining time?

I also think we can discuss the changes in both wallets. Today, after clicking on Send Tx, the user sees a "Sending..." screen. I think we should split it into two screens: "Signing tx...", then "Resolving tx...", and finally, "Sending tx...".

In the "Resolving tx..." we can use the expected time to fill a progress bar. The progress bar can grow from 0% to 100% linearly on max(2, 2*X seconds) (where X is the expected time). While it is growing, we should check for the mining status every max(0.5, X/2) seconds.

[obiyankenobi]

Yes, we don't need the parents for the signature but the /push-tx expects the tx already mined (with the nonce). And to mine the tx we need the parents, so we need to add the parents before sending to the mining server, I think.

We could use some other API instead of /push-tx, like the one we currently use for sending tokens. But why do we need the parents to mine a tx?

I think we should split it into two screens: "Signing tx...", then "Resolving tx...", and finally, "Sending tx...". What's each step? I understand signing and sending, but what's resolving?

Also, while I think we could have a progress bar, I'd leave it for future improvements.

[pedroferreira1]

We could use some other API instead of /push-tx, like the one we currently use for sending tokens. But why do we need the parents to mine a tx?

The parents are part of the graph hash used to calculate the tx hash. We could discuss about removing it from there but I think this should be done elsewhere. Right now we need the parents before sending to the mine API, so we can use our current /push-tx endpoint after returning from the mining.

Also, while I think we could have a progress bar, I'd leave it for future improvements.

I agree we could leave for the future but I think we should at least give a feedback of the steps (like we do when using the ledger).

[obiyankenobi]

The parents are part of the graph hash used to calculate the tx hash. We could discuss about removing it from there but I think this should be done elsewhere.

I completely mixed things up. They are not in the signature but we do need them for mining. My bad.

[jansegre]

Parents are not part of the signature, so we don't need to get them before contacting the mining server. The full node could add them as part of /push-tx.

Yes, as @pedroferreira1 said, they are needed for mining.

For me the design is good to close. Can you just confirm this question, please?

Only one thing was not 100% clear. After sending the first request (/submit-job) the wallets will need to check from time to time if the job is already finished, right?

Yes, the wallet will need to do some polling to get the result.

I like the design, but I think we should get into the pooling strategy. Should we check the status every X seconds, where X is half the expected mining time?

I also think we can discuss the changes in both wallets. Today, after clicking on Send Tx, the user sees a "Sending..." screen. I think we should split it into two screens: "Signing tx...", then "Resolving tx...", and finally, "Sending tx...".

In the "Resolving tx..." we can use the expected time to fill a progress bar. The progress bar can grow from 0% to 100% linearly on max(2, 2*X seconds) (where X is the expected time). While it is growing, we should check for the mining status every max(0.5, X/2) seconds.

I like discussing UI/UX, but I think we could do it directly on the pull request of the respective wallets. Mainly because we could opt for the faster implementation, and also because the current proposal seems to already supports a wide range of UI/UX options.

About the polling strategy, I'll add that as the recommended strategy.

[obiyankenobi]

This is good for me.

As Jan mentioned, we should debate wallet UI/UX on a separate issue on each wallet.

[pedroferreira1]

Good for me!

[msbrogli]

Good for me as well.