search

HathorNetwork / hathor-wallet-service-old

MIT License
4 stars 4 forks source link

[Design - old] General architecture, features, and API. #1

Closed msbrogli closed 4 years ago

msbrogli commented 4 years ago

Introduction

The hathor-wallet-service will be used as backend for all official Hathor's wallets.

This design is highly inspired in the Bitcore Wallet Service.

Sync between full node and service is being designed here: PR https://github.com/HathorNetwork/hathor-wallet-service/issues/8.

TODO LIST

TEST LIST

Requirements

Architecture

┌──────────────┐       ┌─────────────────┐       ┌───────────────┐
│              │       │                 │       │               │
│  Full node   │──────▶│  Tx Processor   │◀─────▶│   Database    │
│              │       │                 │       │               │
└──────────────┘       └─────────────────┘       └───────────────┘
                                │                        ▲        
                                │                        │        
                                │                        ▼        
                                │                ┌───────────────┐
                                │                │               │
                                └───────────────▶│  API Service  │
                                                 │               │
                                                 └───────────────┘

The service will be implemented using AWS services.

Services

Tx Processor

Receive transactions/blocks from the full node and update the database. It will receive two types of messages: NEW_TX and UPDATE_TX. The latter is used to notify a change in the tx status (from executed to voided or otherwise).

It must keep the balance and transaction history of all addresses, even the ones not designated for a wallet. In the future, we may include an API to query the address and tx history for any address. We would also easily know the richest addresses.

When a new tx arrives or a tx is updated, it must update the indexes of all affected wallets.

When the full node restarts, it should sent all notifications again. If the status of the transaction is the same, we can safely skip it.

API Service

Load from database and return to the wallet. The idea is to do the least number of calculations here. All things should be pre-computed by the Tx Processor.

Maintenance Tools

For each table, we should have a verification tool that checks its consistency. This tool should also have a parameter to recalculate the records of a given table. These tools will be integrated in our monitoring and alert system.

Database

Flow from wallets

  1. Client creates a wallet (wallet_id = sha256d(pubkey)).
  2. Client polls the wallet until it is ready.
  3. Get balance and tx history for a token.
  4. Create a tx proposal.
  5. Sign the inputs of the tx proposal.
  6. Resolve the tx proposal.
  7. Propagate the tx proposal.

Events from Full Node

API for the Wallet

Features:

Authentication

All requests must be signed by m/1/1 of the wallet's xpriv.

POST /wallet: Create a new wallet

When the wallet is created, the following tasks must be executed:

After requesting the creation of the wallet, the client must polling to verify when it is ready. It may take a few seconds to be ready. If an error occurs, the user must have the choice to check status again or reset the wallet.

Arguments:

Returns: { walletId, status }

GET /wallets: Get status of a wallet

Arguments:

Returns: All fields of the wallet, including its status.

GET /addresses: Get wallet's addresses

We can run one query in the Address table filtering by a given walletId.

Open question: Should we paginate?

Returns:

GET /balance: Get wallet's balance

Run one query in the Balance table filtering by walletId (and tokenId if the argument is passed).

Arguments:

Returns:

If TokenId is received, it will return only the requested token.

GET /txhistory: Get wallet's transaction history

Arguments:

Returns:

GET /tx: Get transaction's detail

Arguments:

Returns:

GET /txproposals: List of wallet's tx proposals

Arguments:

Returns:

POST /txproposals: Create a new tx proposal

When a tx proposal is created, it should mark the used UTXO as locked. They do it by assigning a TxProposalId to them. So, next tx proposal will skip them avoiding double spending. We can optionally create a scheduled job to cancel all tx proposal opened for more than X minutes.

Arguments:

Returns:

POST /txproposals/update: Update tx proposal

Arguments:

Returns:

POST /txproposal/close: Cancel tx proposal

When a tx proposal is closed, it should remove all locks from the UTXOs used by it.

Arguments:

Returns:

pedroferreira1 commented 4 years ago

I agree with @jansegre that it's okay to let the balance be outdated sometimes. I just think we shouldn't prevent the users from use the tokens only because we haven't updated it.

That's the reason I think we could go with the first solution proposed with the timelock updates when a new block arrives. My only concern is if we have a long time without blocks, we would have an outdated balance for a long time but this not exactly a big problem, since timelock is not used every time. So to allow the use of tokens even if we haven't received any blocks, I think we could update the wallet timelocks every time we request a tx proposal also, so the users would never have any problems when trying to spend an output that is not locked anymore. This is not ideal because it will show one thing on the balance and another thing is for real but I guess it's fine.

obiyankenobi commented 4 years ago

and then this is an event that should trigger the same update mechanism that a schedule would (which I assume is a solved problem, how to make this update arrive to the frontend).

That's true, hadn't thought about it this way.

I agree that it's ok to have the balance outdated for a little while. @pedroferreira1 I'm not sure that we necessarily need to handle this, as the user will see the balance as locked and the wallet currently will show an error if he tries to send more tokens than he has. So on the user's point of view, it'll be consistent.

But I'm also thinking more about this lazy evaluation proposal and it might not be much more complex to implement. Also, as @jansegre suggested, we should look at how BWS handles this.

obiyankenobi commented 4 years ago

From my research, it seems BWS does not handle height or time locks. From bitcore-wallet-service/src/lib/chain/btc/index.ts:

  totalizeUtxos(utxos) {
    const balance = {
      totalAmount: _.sumBy(utxos, 'satoshis'),
      lockedAmount: _.sumBy(_.filter(utxos, 'locked'), 'satoshis'),
      totalConfirmedAmount: _.sumBy(_.filter(utxos, 'confirmations'), 'satoshis'),
      lockedConfirmedAmount: _.sumBy(_.filter(_.filter(utxos, 'locked'), 'confirmations'), 'satoshis'),
      availableAmount: undefined,
      availableConfirmedAmount: undefined
    };
    balance.availableAmount = balance.totalAmount - balance.lockedAmount;
    balance.availableConfirmedAmount = balance.totalConfirmedAmount - balance.lockedConfirmedAmount;

    return balance;
  }

In this context, locked means UTXOs which are part of a tx proposal and confirmed means transactions which have at least one new block confirming them.

This print in one of their issues shows the explanation in the Copay app.

jansegre commented 4 years ago

To me this also raises the concern of confirmations. We aren't really doing anything, and I think it's more important time locks.

obiyankenobi commented 4 years ago

To me this also raises the concern of confirmations. We aren't really doing anything, and I think it's more important time locks.

I agree that it's an important topic to be discussed, maybe on a separate issue.

obiyankenobi commented 4 years ago

I started to think about tx proposals. Some stuff is not clear to me:

  1. I think the tx-proposal idea of BWS is to handle multisig. Is this also our goal? If it is, we need to handle “collecting” signatures from multiple users and storing that while the tx is only partially signed, before sending. There are no tables for this in the design;
  2. we could also accept a list of inputs on POST /txproposals, so the user can choose the inputs. It’s probably easier than creating the proposal and later updating with the inputs (with POST /txproposals/update);
  3. what’s the advantage of the wallet-service sending the tx? I’d say it’s better that it only deals with assembling the tx. Then the wallet can do the pow and propagate the tx;
  4. do we need to store old proposals? I think we can have on database only the open ones. In that case, not sure if it’s worth having a status, as they’re all open.
  5. also, if we want to be able to update the proposal, we need to store the original information, such as the outputs. Currently there’s no place for that;

I'm starting to wonder if it's not better to have a super simple flow. It'd be like this:

  1. wallet calls POST /txproposal with the outputs (or just the total amount needed);
  2. wallet-service selects the UTXOs, marks them as part of a proposal and returns them to the wallet;
  3. wallet assembles the final tx, does pow (with mining service) and propagates tx. If send is not successful or user cancels, tell wallet-service to close the proposal;
jansegre commented 4 years ago 
1. I think the tx-proposal idea of BWS is to handle multisig. Is this also our goal? If it is, we need to handle “collecting” signatures from multiple users and storing that while the tx is only partially signed, before sending. There are no tables for this in the design;

I'd say we don't need to support multisig right now. But ideally we make an API that is identical, or very close, or at the very least compatible, with the API that supports multisig. Unless supporting multisig is easy (like an extra day of work maybe), then I'd be in favor of supporting it from the get go.

2. we could also accept a list of inputs on `POST /txproposals`, so the user can choose the inputs. It’s probably easier than creating the proposal and later updating with the inputs (with `POST /txproposals/update`);

As long as the API can work without having to take a list of inputs. If the API requires explicit inputs, it won't improve the current situation of the wallet, and any updates on that logic will take much longer because they need to be deployed to the end-users. But manually choosing the inputs is something the current wallet can already do, it would be bad to take this feature away, even if only temporarily.

3. what’s the advantage of the wallet-service sending the tx? I’d say it’s better that it only deals with assembling the tx. Then the wallet can do the pow and propagate the tx;

The biggest advantage is not exposing any API of the fullnode to the wallet. And this is no small advantage. I'm not saying the proposal API has to take care of propagating, maybe it does. 

4. do we need to store old proposals? I think we can have on database only the open ones. In that case, not sure if it’s worth having a status, as they’re all open.

It probably makes sense for a couple of reasons. We can offload old proposals to a log db to help us identify issues (privacy wise it's debatable, but maybe all or most of the info stored on the proposal would end up anyway on the blockchain if it was successfully propagated) and the process would collect "done" or "cancelled" proposals and then delete them.

5. also, if we want to be able to update the proposal, we need to store the original information, such as the outputs. Currently there’s no place for that;

I'd say we can start with immutable proposals. There may be pitfalls in supporting updates, it might make sense for multisig, but don't know.

I'm starting to wonder if it's not better to have a super simple flow. It'd be like this:

1. wallet calls POST /txproposal with the outputs (or just the total amount needed);

2. wallet-service selects the UTXOs, marks them as part of a proposal and returns them to the wallet;

3. wallet assembles the final tx, does pow (with mining service) and propagates tx. If send is not successful or user cancels, tell wallet-service to close the proposal;

I'd prefer if the wallet-service was responsible for propagation. The only difference would be that after the tx is assembled the wallet-service marks it as needing POW (which would use the state or status field), the wallet chooses how to handle POW on its own, and when it has the final info (nonce and maybe timestamp), it asks the wallet-service to propagate the tx. The advantage of that is that the wallet service can keep locks for UTXOs with more confidence, whether if the tx was propagated outside it's knowledge it would have to wait for the node to detect that an UTXO was spent, or timeout the UTXO lock.

obiyankenobi commented 4 years ago

I'd say we don't need to support multisig right now. But ideally we make an API that is identical, or very close, or at the very least compatible, with the API that supports multisig.

To be honest, I'm not sure how much extra time that would be. But I think we didn't design any other part of the service thinking about multisig wallets. If we want to support it, we'll have to change all of them also, so I don't think it's much use designing this part thinking about multisig if the rest doesn't support it.

Expanding a bit on this point with a practical example, multisig wallets don't have xpubkeys like regular wallets. The service so far was designed relying on having the xpubkeys, so adapting to multisig will require a major overhaul.

As long as the API can work without having to take a list of inputs.

Of course, the inputs would be optional.

The biggest advantage is not exposing any API of the fullnode to the wallet.

That makes sense, but I worry if that would create a dependency in other services. We'll start propagating txs using the tx-mining-service, so the wallet-service would have to communicate with that one.

It probably makes sense for a couple of reasons.

It's still not clear to me the benefits of keeping old proposals. If it's just for debugging, I'd say just logging can take care of that.

The advantage of that is that the wallet service can keep locks for UTXOs with more confidence, whether if the tx was propagated outside it's knowledge it would have to wait for the node to detect that an UTXO was spent, or timeout the UTXO lock.

To be honest, even if we propagate from the wallet-service, I'd say we should wait for the confirmation from the full node. Since we'll "lock" UTXOs that are part of a transaction, I'd say it's not a problem to wait for this confirmation, there's no risk of trying to use the same UTXOs twice. And I think we'll have to have a "cleanup" process anyway.

pedroferreira1 commented 4 years ago

  1. I think we should discuss multisig later, no need to add support right now and we can discuss in a later design;

  2. I agree that we should add support to receive the inputs on /txproposals POST and not updating it. The input selection might change also the outputs (with a change output), so I would say we support it only in the creation. Also I think would be good to prepare this part of the code to accept the an input selection algorithm later (I am not saying to add support now, only to code this part thinking about it).

  3. I agree with Jan that this could be done by the wallet-service. At first I thought it would be better to leave it to the wallet because we could decide to resolve the pow of the transaction on the client later (or have another third party paid service that resolves tx pow, as we discussed sometimes). But with the reasoning of not exposing the API I don't see any problems in returning to the wallet the 'unresolved tx', resolve the pow and then send to the wallet service to propagate it.

That makes sense, but I worry if that would create a dependency in other services. We'll start propagating txs using the tx-mining-service, so the wallet-service would have to communicate with that one.

The tx-mining-service only solves the pow of the transaction, it's not propagating anything (it could but we decided not to), so nowadays the wallet is propagating the transaction even after the tx mining integration.

  1. I also don't think we should save old proposals but the status is important to know if it's creating the proposal, waiting for pow, waiting for propagation, cancelled, or completed. Thinking about this 'cancelled' status maybe it's good to have the proposals on the database (for a while at least) to get this stats.
jansegre commented 4 years ago

Expanding a bit on this point with a practical example, multisig wallets don't have xpubkeys like regular wallets. The service so far was designed relying on having the xpubkeys, so adapting to multisig will require a major overhaul.

Makes sense. I suppose it isn't of much practical use to support multisig transactions but not wallets, since it could be hard to use the utxo later. Unless this useful for the headless-wallet, we could skip considering multisig for now.

That makes sense, but I worry if that would create a dependency in other services. We'll start propagating txs using the tx-mining-service, so the wallet-service would have to communicate with that one.

The tx-mining-service only solves the pow of the transaction, it's not propagating anything (it could but we decided not to), so nowadays the wallet is propagating the transaction even after the tx mining integration.

Yeah. I agree that creating inter-service dependency is not good. And I think we can still do this without it. The wallet-service should not communicate with tx-mining-service whatsoever. The wallet is the one that communicates with both. And both communicate with a fullnode (or multiple fullnodes, it's up to each service).

msbrogli commented 4 years ago
  1. I think the tx-proposal idea of BWS is to handle multisig. Is this also our goal? If it is, we need to handle “collecting” signatures from multiple users and storing that while the tx is only partially signed, before sending. There are no tables for this in the design;

I agree that we can design multisig later. I don't think adapting the service will require a major overhaul, but we can dive deep into it in another design.

  1. we could also accept a list of inputs on POST /txproposals, so the user can choose the inputs. It’s probably easier than creating the proposal and later updating with the inputs (with POST /txproposals/update);

I agree that we can accept a list of inputs on POST /txproposals. I think the POST /txproposals/update is useful in other cases, such as Atomic Swap. When its just user A sending funds to user B, it doesn't seem relevant.

  1. what’s the advantage of the wallet-service sending the tx? I’d say it’s better that it only deals with assembling the tx. Then the wallet can do the pow and propagate the tx;

I also prefer to publicly expose only the wallet service API. The wallet wouldn't need to have access to the full node at all.

  1. do we need to store old proposals? I think we can have on database only the open ones. In that case, not sure if it’s worth having a status, as they’re all open.

The status may be important in other cases. For example, in a multisig transaction, the status may let the other parties that a transaction has been cancelled. Or that one of the signers declined to do it. In our case here, the status would indicate whether the transaction was successfully created and is just waiting for the correct nonce and parents.

  1. also, if we want to be able to update the proposal, we need to store the original information, such as the outputs. Currently there’s no place for that;

I didn't understand why we need to store the original information.

I'm starting to wonder if it's not better to have a super simple flow. It'd be like this:

  1. wallet calls POST /txproposal with the outputs (or just the total amount needed);
  2. wallet-service selects the UTXOs, marks them as part of a proposal and returns them to the wallet;
  3. wallet assembles the final tx, does pow (with mining service) and propagates tx. If send is not successful or user cancels, tell wallet-service to close the proposal;

I can't see the difference between this super simple flow to the one in the design.

msbrogli commented 4 years ago

Yeah. I agree that creating inter-service dependency is not good. And I think we can still do this without it. The wallet-service should not communicate with tx-mining-service whatsoever. The wallet is the one that communicates with both. And both communicate with a fullnode (or multiple fullnodes, it's up to each service).

@jansegre Another advantage of not propagating the transaction using the tx-mining-service is that to avoid the following case: (i) wallet sends the tx to the tx-mining-service, (ii) wallet's connection is down, so an error message shows up, and (iii) the tx-mining-service propagate the transaction. In this case, the user would see an error message while the tx-mining-service would have sent the transaction.

I know that this case may happen in the propagate request, but this is a very fast request. The request to the tx-mining-service is way slower and requires polling to check if the tx has already been resolved.

msbrogli commented 4 years ago

To be honest, even if we propagate from the wallet-service, I'd say we should wait for the confirmation from the full node. Since we'll "lock" UTXOs that are part of a transaction, I'd say it's not a problem to wait for this confirmation, there's no risk of trying to use the same UTXOs twice. And I think we'll have to have a "cleanup" process anyway.

I agree. We should mark a transaction as sent (and release its "UTXOs") only after the full node has responded that it was successfully propagated. In this case, we should also wait for the notification from the full node before we release the UTXOs. Otherwise we may be subject to a race condition.

obiyankenobi commented 4 years ago

Ok, from the comments we seem to have agreed on skipping any multisig design for now and that this service will be responsible for propagating the tx to the full-node. The flow I propose is this:

  1. wallet sends POST /txproposals to create a tx. Parameters:

    • outputs (we might complete it with the change output);
    • (optional) input selection algo;
    • (optional) inputs;

  2. wallet-service gets the utxos corresponding utxos and also creates a tx proposal in the DB, with a open status. Response:

    • inputs;
    • outputs;

  3. wallet does the PoW with the tx-mining-service;

  4. wallet uses POST /txproposals/update to send the PoW, parents and timestamp, also asking to propagate the tx. Parameters:

    • nonce;
    • parents;
    • outputs;
    • inputs;

  5. wallet-service assembles the final tx and propagates it;

On step 4, we need to receive both the outputs and inputs:

In the end, it might be easier to just receive the complete tx in step 4, instead of each piece. The wallet can assemble it and send the hex. The wallet-service wouldn't need to assemble it, just propagate.

jansegre commented 4 years ago 
2. wallet-service gets the utxos corresponding utxos and also creates a tx proposal in the DB, with a `open` status. Response:

* inputs;

* outputs;

It might be worth saving these input and outputs to the database. For the inputs, we can link them to the utxos which would help preventing selecting an utxo that is already in use by another open proposal, and also saving the order of that input. Outputs benefit from having the order stored and also being a cache so it doesn't need to be passed back by the wallet after the tx is mined.

I'm also not sure what is enough for the tx-mining-service to mine a transaction. Does it need a timestamp hint, tx version, parents, or anything else? If it does this would be the call that needs to return those params.

4. wallet uses `POST /txproposals/update` to send the PoW, parents and timestamp, also asking to propagate the tx. Parameters:

Not sure what other endpoints do but this could be a PUT /txproposals. No substantial benefit to this except being more RESTful (but not completely), but also no downsides either. If other endpoints follow these POST /x/update scheme, then I wouldn't bother with this alternative.

Otherwise this proposal seems great.

obiyankenobi commented 4 years ago

Not sure what other endpoints do but this could be a PUT /txproposals. No substantial benefit to this except being more RESTful (but not completely), but also no downsides either. If other endpoints follow these POST /x/update scheme, then I wouldn't bother with this alternative.

I agree. Also, I am wondering if we should use path parameters instead of query ones:

POST /wallets/       --> creates the wallet
GET /wallets/{walletId}
GET /balances/{walletId}
etc...

POST /txproposals/     --> creates the tx proposal
PUT /txproposals/{txProposalId}

The only reason I am not 100% convinced is that no other service of ours uses this standard, if I'm not mistaken. For example, fetching a tx in the full node uses GET /transaction?id=tx_id.

pedroferreira1 commented 4 years ago

In the end, it might be easier to just receive the complete tx in step 4, instead of each piece. The wallet can assemble it and send the hex. The wallet-service wouldn't need to assemble it, just propagate.

I do agree with this, it's probably easier to receive a hex tx and just propagate it to the full node. However I think we should have a validation here, that the inputs/outputs from the hex data are the same as the ones in the proposal. Otherwise we might be locking some outputs that were not used.

I'm also not sure what is enough for the tx-mining-service to mine a transaction. Does it need a timestamp hint, tx version, parents, or anything else? If it does this would be the call that needs to return those params.

tx-mining-service needs tx data without nonce and parents. It returns the parents, nonce and timestamp.

obiyankenobi commented 4 years ago

I do agree with this, it's probably easier to receive a hex tx and just propagate it to the full node. However I think we should have a validation here, that the inputs/outputs from the hex data are the same as the ones in the proposal. Otherwise we might be locking some outputs that were not used.

This will only happen if the wallet modifies the tx, which is not expected. Anyway, when the tx is sent (and confirmed by the full node), we can remove all proposal ids from the utxos, if there are any left. No utxos would be left locked, so there's no need to double-check this information.

Outputs benefit from having the order stored and also being a cache so it doesn't need to be passed back by the wallet after the tx is mined.

It's still not clear to me what's the advantage of saving them. In most cases, we have only a few outputs, so that would not make a big difference regarding bandwidth (receiving them again in the update call).

One other thing I forgot on step 4 is receiving the signature for each input. I think this also supports the idea of sending the full tx.

On the other hand, assembling the tx on the wallet-service might serve us if we decide to change its structure in the future. Instead of everyone having to update the wallet, we just need to update the wallet-service.

msbrogli commented 4 years ago

I agree with all points you made. Maybe we should put this design in another Design Issue and put the link in the description of this one. Or should we update this one?

obiyankenobi commented 4 years ago

Which design? For tx-proposals?

obiyankenobi commented 4 years ago

Points to decide:

msbrogli commented 4 years ago

Points to decide:

  • are we gonna save the inputs and outputs of a proposal on the wallet-service database?

I think we should mark the selected inputs as "used" in the UTXO table. It seems similar to saving the inputs, except for the input data, right?

Regarding the outputs, I think we must save them as well to know where the funds will be transferred to, and also which address was used for the change. Maybe I'm not seeing the whole picture.

I can't see how we can move forward without saving the inputs and outputs. We can talk tomorrow about it.

  • will we receive the complete tx or just the weight, nonce, etc on the tx-proposal update API?

I think the tx-proposal update API can have a bunch of optional fields. It will update only the received fields. It can accept weight, nonce, parents, inputs, input[0], outputs, output[0], and so on. If it gets too complicated, we can do the bare minimum to get the service working and extend it later.

obiyankenobi commented 4 years ago

I think we should mark the selected inputs as "used" in the UTXO table. It seems similar to saving the inputs, except for the input data, right?

Almost the same. But we'll have not only to mark the UTXO as used, but also its position in the input list of the new tx.

Regarding the outputs, I think we must save them as well to know where the funds will be transferred to, and also which address was used for the change. Maybe I'm not seeing the whole picture.

I can't see how we can move forward without saving the inputs and outputs. We can talk tomorrow about it.

We can if we require the user/wallet to send us the complete tx after it does the pow. The wallet will have all the information it needs to assemble the tx.

jansegre commented 4 years ago

I think we should mark the selected inputs as "used" in the UTXO table. It seems similar to saving the inputs, except for the input data, right?

Almost the same. But we'll have not only to mark the UTXO as used, but also its position in the input list of the new tx.

We could just have a table for inputs and outputs of open transaction proposals (which would include a column to indicate the position of the output/input on their respective tables). I think that's what I suggested, this should be fine, right?

Regarding the outputs, I think we must save them as well to know where the funds will be transferred to, and also which address was used for the change. Maybe I'm not seeing the whole picture. I can't see how we can move forward without saving the inputs and outputs. We can talk tomorrow about it.

We can if we require the user/wallet to send us the complete tx after it does the pow. The wallet will have all the information it needs to assemble the tx.

I think that accepting a whole tx for the sake of not having to assemble it is a bad idea. But accepting a whole tx for the sake of convenience (if that's what the tx-mining-service returns), is OK. We should at the very least verify if the transaction we're sending is of an open tx proposal, which would require breaking up an "assembled tx". Mostly to mitigate propagating transactions unrelated to the current wallets ("rogue transactions").

On the other hand, if the wallet service only receives the missing parts of the tx (instead of an "assembled tx"), there is no way to even propagate a rouge transactions, because there wouldn't even be a way to build one.

obiyankenobi commented 4 years ago

We should at the very least verify if the transaction we're sending is of an open tx proposal, which would require breaking up an "assembled tx". Mostly to mitigate propagating transactions unrelated to the current wallets ("rogue transactions").

The user can indicate the proposal-id when sending the tx, so we don't need to "break it up" to check. In the end, we'll only update most of the database tables when we receive the transaction from the full node, when it's been propagated and it's valid.

We'll need a "cleaner process" anyway, to clean up utxos that are marked as part of a tx proposal but were eventually not sent. That would also serve in case the user sends a tx with incorrect proposal id. Let's see how that would work:

  1. User creates the tx proposal. The wallet service selects utxo0 and utxo1 as inputs and mark with txProposalA;
  2. User decides to use a different input (utxo2), does the pow and asks to propagate the tx (tx1);
  3. Wallet service receives tx1 from the user and propagates it. At this point, it does not remove the proposal id from the utxo table;
  4. Wallet service receives tx1 from the full node. At this point, it'll remove the proposal id from the utxo table. In fact, it removes the utxo from the table, as it's been spent. utxo0 and utxo1 are left there marked as being part of txProposalA;
  5. After some time, the cleaner process will check all utxos that are marked as part of a proposal and see which ones should be unmarked, based on how "old" the proposal is. At this point, we'd remove txProposalA from utxo0 and utxo1;

Only one problem can happen as result of this mismatch: utxo2 is not marked as part of proposal and it will be available to be chosen as an input until we get tx1 from the full node. If the wallet tries sending another tx immediately after sending tx1, it might use utxo2 and it'll get an error when propagating, as utxo2 has been spent by now (we have this problem currently in our wallets). Given that this will only happen if the wallet misbehaves, I think it's acceptable.

jansegre commented 4 years ago

It still bothers me that the wallet service would be able to propagate any transaction where there is an open proposal-id, even if this transaction is totally unrelated to the transaction being proposed. That would make that API almost like a proxy to push-tx. Is there any other upside besides simplifying implementation a little? Even if we have to implement a little more to assemble txs (which really shouldn't be much), it could help catch bugs on the client end that could otherwise be completely silent.

I agree that we'll need a cleaner process for the sake of catching up what eventually passes through the cracks. But allowing the tx-proposal to transparently push transactions to the fullnode is relying too much on the cleaner process. Ideally the cleaner process would log an error or warn us every time it had to actually cleanup something, and that would indicate a bug.

That's not to say that accepting an encoded transaction is a bad format, maybe it's still more convenient, but I do think that some validation that would require inspecting the encoded transaction is needed.

obiyankenobi commented 4 years ago

It still bothers me that the wallet service would be able to propagate any transaction where there is an open proposal-id, even if this transaction is totally unrelated to the transaction being proposed. That would make that API almost like a proxy to push-tx. Is there any other upside besides simplifying implementation a little?

This only happens if the wallet misbehaves, so I don't see as a big problem.

Even if we have to implement a little more to assemble txs (which really shouldn't be much), it could help catch bugs on the client end that could otherwise be completely silent.

I'm not sure what you mean by that. Do you mean if the wallet assembles the tx incorrectly? The wallet service will only work in conjunction with the wallets, anyway. They'll have to send the correct data in the API, so whether it's the list of outputs or the complete tx.

Is there any other upside besides simplifying implementation a little?

I think simplifying the code seems like a very good reason, since it's not clear the benefits of the more complex solution. The benefit I see in assembling it on the wallet service is that, if we make changes to the tx structure, we only need to change the wallet service, not require all users to update the wallet.

jansegre commented 4 years ago

It still bothers me that the wallet service would be able to propagate any transaction where there is an open proposal-id, even if this transaction is totally unrelated to the transaction being proposed. That would make that API almost like a proxy to push-tx. Is there any other upside besides simplifying implementation a little?

This only happens if the wallet misbehaves, so I don't see as a big problem.

But that's exactly it. Misbehaving means bugs, and we're making that bug harder to find. (Edit: not just harder to find, harder to come up)

Even if we have to implement a little more to assemble txs (which really shouldn't be much), it could help catch bugs on the client end that could otherwise be completely silent.

I'm not sure what you mean by that. Do you mean if the wallet assembles the tx incorrectly? The wallet service will only work in conjunction with the wallets, anyway. They'll have to send the correct data in the API, so whether it's the list of outputs or the complete tx.

The wallet client or the tx-mining-service could have a bug, or there could be some exploit possible to sending a custom transaction through the wallet service, which could be cheaper than setting up a fullnode (but I'm not really sure on this).

Is there any other upside besides simplifying implementation a little?

I think simplifying the code seems like a very good reason, since it's not clear the benefits of the more complex solution. The benefit I see in assembling it on the wallet service is that, if we make changes to the tx structure, we only need to change the wallet service, not require all users to update the wallet.

I think my bias is that I'm assuming we make some validations that would require querying the database (which I still think are very important). Assuming we have to make these validations, assembling the transaction on the wallet service instead of the wallet isn't any more complex (we would have all the fields on hand already, it would just be a matter of serialization). Probably just more convenient.

If we don't make any verification, then we don't have to make any queries, then actually is less complex. So in the end it seems to be a matter of simplifying for the sake of skipping these checks. Which I can totally accept for a proof-of-conecept implementation, but I think it's bad for the design of what it should be.

obiyankenobi commented 4 years ago

The wallet client or the tx-mining-service could have a bug, or there could be some exploit possible to sending a custom transaction through the wallet service, which could be cheaper than setting up a fullnode (but I'm not really sure on this).

I think this is the situation we already have and no issue has ever been raised. People can just use our full node APIs to push any tx.

I think my bias is that I'm assuming we make some validations that would require querying the database (which I still think are very important). Assuming we have to make these validations, assembling the transaction on the wallet service instead of the wallet isn't any more complex (we would have all the fields on hand already, it would just be a matter of serialization). Probably just more convenient.

If we don't make any verification, then we don't have to make any queries, then actually is less complex. So in the end it seems to be a matter of simplifying for the sake of skipping these checks. Which I can totally accept for a proof-of-conecept implementation, but I think it's bad for the design of what it should be.

To be honest, I'm not sure what validations we need to make. Could would give some examples?

In the end, if the wallet wants to send some "weird" tx and it owns the UTXOs, it can do it. There's nothing the wallet-service can do to prevent this. That's why I don't think it's a big problem if the service just acts as a proxy in the final send, receiving the full tx. As long as this cannot leave the db in some inconsistent state, I really don't think it's a problem.

msbrogli commented 4 years ago

I agree with @jansegre that the wallet service should validate the transaction and only propagate if it matches the tx proposal. Otherwise, there might be possible attacks.

If a new transaction is received from the full node, and this transactions spends an output associate with a tx proposal, we should mark that tx proposal as invalid and log it, unless this transaction was sent by the wallet service.

If the user would like to change the chosen utxo, the user must update the tx proposal. Otherwise, the tx proposal will decline to propagate the tx. If the user propagates directly to a full node, the tx proposal will be marked as invalid.

I don't mind what is the best format to receive the transaction from the user. A JSON is always easy to read and debug but we can receive the serialized transaction as well.

obiyankenobi commented 4 years ago

I agree with @jansegre that the wallet service should validate the transaction and only propagate if it matches the tx proposal. Otherwise, there might be possible attacks.

Do you mean validating inputs and outputs? Or just that there's a valid proposal id? And what attacks are there?

jansegre commented 4 years ago

In the end, if the wallet wants to send some "weird" tx and it owns the UTXOs, it can do it. There's nothing the wallet-service can do to prevent this. That's why I don't think it's a big problem if the service just acts as a proxy in the final send, receiving the full tx. As long as this cannot leave the db in some inconsistent state, I really don't think it's a problem.

Why should the wallet propagate garbage txs? This could potentially open the door for some DoS if somebody finds a tx that takes long to validate on the node. If this isn't even possible in the first place, a whole class of possible attacks is prevented.

And what attacks are there?

Not any in particular is known right now. But I'd rather not leave the door open. Again, my point is not that this has to be the actual PoC implementation, but the target design. I'm OK taking a shortcut to implement it faster and get it tested.

For me we should have set a few higher level objectives and premises to the wallet service. One of them being shielding the fullnode as a security measure, currently we are exposed, and that is definitely not good. I'd rather have anyone that wants a fullnode API to run their own fullnode. I don't see why we should give up on this just because of one endpoint.

obiyankenobi commented 4 years ago

Why should the wallet propagate garbage txs?

But the wallet-service would only propagate txs with a valid proposal-id. Given that it'll be a random string (probably UUID), it's not easy to guess it.

Again, my point is not that this has to be the actual PoC implementation, but the target design. I'm OK taking a shortcut to implement it faster and get it tested.

This is a good point. I might actually go with this design right now since it's easier while we debate.

jansegre commented 4 years ago

But the wallet-service would only propagate txs with a valid proposal-id. Given that it'll be a random string (probably UUID), it's not easy to guess it.

That's exactly the same as a CSRF token. It's certainly better than no protection at all, but the content of a proposal submit would still be forwarded unmodified and most importantly, unverified to the fullnode. A 10MB transaction is invalid, but if we accept a request that large (there should be a max-body-size on the HTTP frontend, if we use one, hopefully we do) we would be forwarding that to the fullnode. Could it be crafted in a way that takes too long for the node to verify before rejecting? I don't know, if it could, this could lead to a DoS. Assuming we have more wallet-service workers than fullnodes, to improve scalability, this would impact the wallet service as a whole. This is a threat that can be avoided just by only forwarding well formed data to the fullnode. To me this is just as important as doing input sanitization when querying a database. Now a days most databases frameworks have some form of automatic sanitization, but we don't have anything like that for our fullnode.

obiyankenobi commented 4 years ago

What if we just parse the tx in the wallet service before sending, from the data sent by the user (parsing the hex data)? It's different than assembling it from scratch fetching info from the database. We'd have a basic confirmation that this is a valid transaction and make sure we're forwarding well formed data to the full node.

jansegre commented 4 years ago

What if we just parse the tx in the wallet service before sending, from the data sent by the user (parsing the hex data)? It's different than assembling it from scratch fetching info from the database. We'd have a basic confirmation that this is a valid transaction and make sure we're forwarding well formed data to the full node.

That only works for preventing bugs, which is good, but not exploits (which can lead to DoS opportunities).

obiyankenobi commented 4 years ago

The tx you're saying could be used in the future for an attack is:

If that's the case, it seems to me the problem is that we have such a tx and we'll need to correct our full node code so it doesn't happen anymore. I mean, no tx such take long to verify. Even if we prevent it from being done in the wallet-service, the same can be achieved by faking a synced node and sending this tx.

jansegre commented 4 years ago

The tx you're saying could be used in the future for an attack is:

* a valid tx, structurally;

Sure but this doesn't even have to be the case. Because the "passthrough" proposal doesn't even check for structural validity.

* takes the node a long amount of time to validate;

If that's the case, it seems to me the problem is that we have such a tx and we'll need to correct our full node code so it doesn't happen anymore. I mean, no tx such take long to verify. Even if we prevent it from being done in the wallet-service, the same can be achieved by faking a synced node and sending this tx.

That's all good until it happens. And we'll have a potentially unstable and DoS'd wallet service until we fix and deploy the node. I'm not arguing this isn't a bug that has to be fixed in the node. It definitely has to. I'm arguing a design that has this whole is inherently more vulnerable to attacks (and also silencing bugs) than one that doesn't.

obiyankenobi commented 4 years ago

Sure but this doesn't even have to be the case. Because the "passthrough" proposal doesn't even check for structural validity.

If we reassemble the tx in the wallet-service before sending, as I proposed, that'd do it.

And we'll have a potentially unstable and DoS'd wallet service until we fix and deploy the node.

Not necessarily. We can probably "understand" this type of tx and specifically block it on the wallet-service. But I do agree that it's one less attack surface.

jansegre commented 4 years ago

Sure but this doesn't even have to be the case. Because the "passthrough" proposal doesn't even check for structural validity.

If we reassemble the tx in the wallet-service before sending, as I proposed, that'd do it.

I think I missed that part. But what would be the meaning of reassembling without using the fields from the database? If it's just checking the structural validity, then "reassembling" just means validating (but only the structure) in this case, no? And why not validate more than just the structure, the cost is an additional query (or requesting additional data on the "proposal id" query).

obiyankenobi commented 4 years ago

But what would be the meaning of reassembling without using the fields from the database? If it's just checking the structural validity, then "reassembling" just means validating (but only the structure) in this case, no?

Yes, that would only be making sure we're sending a tx that is structurally valid. That means no one can attack by sending "garbage".

And why not validate more than just the structure, the cost is an additional query (or requesting additional data on the "proposal id" query).

It was not clear to me the benefits of doing it. The cost might seem just an additional query, but there's always more work behind, making sure the databases are consistent. Therefore, the simpler design seemed more attractive to me.

However, the suggestion I gave to reassemble the tx is probably very close to doing the full tx assembly on the service, as you've been saying. So I'm starting to agree with your point of view, although not yet 100% convinced 😆

pedroferreira1 commented 4 years ago

I am just trying to understand what are the downside (besides more work but not too much and an extra query) of validating the the inputs/outputs of the tx being sent matches the ones from the tx proposal, which is something I said in my last comment I think we should do.

After reading all other comments I've seen good arguments to do this and the only reason not to do is that it would be more work. I think we should do it and in the first version of the tx-proposal code, I don't think it would increase a week of work, so it's pretty okay.

obiyankenobi commented 4 years ago

the only reason not to do is that it would be more work

My point is not more work, is more complexity in the code for a benefit that's not clear to me.

obiyankenobi commented 4 years ago

Ok, so after all the debate, I'll implement it as Jan suggested. I'll add fields tx_proposal and tx_proposal_index to the UTXO table and create a new tx_proposal_outputs table, to keep track of the outputs.

obiyankenobi commented 4 years ago

Introduction

The hathor-wallet-service will be used as backend for all official Hathor's wallets.

Sync between full node and service is being designed here: PR https://github.com/HathorNetwork/hathor-wallet-service/issues/8.

Architecture

┌──────────────┐       ┌─────────────────┐       ┌───────────────┐
│              │       │                 │       │               │
│  Full node   │──────▶│  Tx Processor   │◀─────▶│   Database    │
│              │       │                 │       │               │
└──────────────┘       └─────────────────┘       └───────────────┘
                                                         ▲        
                                                         │        
                                                         ▼        
                                                 ┌───────────────┐
                                                 │               │
                                                 │  API Service  │
                                                 │               │
                                                 └───────────────┘

The service will be implemented using AWS services.

Services

Tx Processor

Receive transactions/blocks from the full node and update the database. It will receive two types of messages: NEW_TX and UPDATE_TX. The latter is used to notify a change in the tx status (from executed to voided or otherwise).

It must keep the balance and transaction history of all addresses, even the ones not designated for a wallet. In the future, we may include an API to query the address and tx history for any address. We would also easily know the richest addresses.

When a new tx arrives or a tx is updated, it must update the database for all affected wallets.

API Service

Load from database and return to the wallet. The idea is to do the least number of calculations here. All things should be pre-computed by the TxProcessor. Only locked balances/authorities may trigger computation on the APIs.

Maintenance Tools

For each table, we should have a verification tool that checks its consistency. This tool should also have a parameter to recalculate the records of a given table. These tools will be integrated in our monitoring and alert system.

Database

- Wallet
  - Id
  - XPubKey
  - MaxGap (default: 20)
  - Status (creating, ready, error)
  - CreatedAt
  - ReadyAt
- WalletBalance
  - WalletId
  - TokenId
  - UnlockedBalance
  - LockedBalance
  - UnlockedAuthorities
  - LockedAuthorities
  - TimelockExpires
  - Transactions

TimelockExpires contains the earliest timestamp at which any locked balance or authority will become unlocked.

- WalletTxHistory (for all transactions)
  - WalletId
  - TokenId
  - TxId
  - Balance (positive: received / negative: sent)
  - Timestamp
- Address (for all addresses, even if WalletId is null)
  - Address
  - Index
  - WalletId
  - NumberOfTransactions

Index indicates the derivation path index used for this address. Both Index and WalletId will be null until the corresponding wallet is initialized in the service.

- AddressBalance
  - Address
  - TokenId
  - UnlockedBalance
  - LockedBalance
  - UnlockedAuthorities
  - LockedAuthorities
  - TimelockExpires
  - Transactions

Same idea as WalletBalance table.

- AddressTxHistory
  - Address
  - TxId
  - TokenId
  - Balance (positive: received / negative: sent)
  - Timestamp
- UTXOs
  - TxId
  - Index
  - TokenId
  - Address
  - Value
  - Authorities
  - Timelock: at which time this becomes unlocked
  - Heightlock: at which height this becomes unlocked (for blocks)
  - Locked: boolean indicating if it is locked
  - TxProposalId: the tx proposal id, if applicable
  - TxProposalIndex: the input index on the tx proposal for this UTXO, if applicable
- TxProposal
  - Id
  - WalletId
  - Status (open, sent, cancelled)
  - CreatedAt
  - UpdatedAt
- TxProposalOutputs
  - TxProposalId
  - Index
  - Address
  - TokenId
  - Value
  - Timelock

This table is used to store the suggested outputs for a tx proposal. The index indicates the output index.

- Token
  - Id
  - Name
  - Symbol

Flow from wallets

  1. Client creates a wallet (wallet_id = sha256d(pubkey)).
  2. Client polls the wallet until it is ready.
  3. Get balance and tx history for a token.
  4. Create a tx proposal.
  5. Sign the inputs of the tx proposal.
  6. Resolve the tx proposal.
  7. Propagate the tx proposal.

Events from Full Node

API for the Wallet

Authentication

All requests must be signed by m/1/1 of the wallet's xpriv.

Websocket:

POST /wallet: Create a new wallet

When the wallet is created, the following tasks must be executed:

After requesting the creation of the wallet, the client must polling to verify when it is ready. It may take a few seconds to be ready. If an error occurs, the user must have the choice to check status again or reset the wallet.

Arguments:

Returns: { walletId, status }

GET /wallet: Get status of a wallet

Arguments:

Returns: All fields of the wallet, including its status.

GET /addresses: Get wallet's addresses

We can run one query in the Address table filtering by a given walletId.

Open question: Should we paginate?

Arguments:

Returns:

GET /balance: Get wallet's balance

Run one query in the Balance table filtering by walletId (and tokenId if the argument is passed).

Arguments:

Returns:

If TokenId is received, it will return only the requested token (list with 1 element).

GET /txhistory: Get wallet's transaction history

Arguments:

Returns:

POST /txproposal: Create a new tx proposal

When a tx proposal is created, it should mark the used UTXOs. They do it by assigning a TxProposalId and TxProposalIndex to them. So, next tx proposal will skip them avoiding double spending. We can optionally create a scheduled job to cancel all tx proposal opened for more than X minutes.

Arguments:

Returns:

The wallet can use this info to assemble the tx and mine it, probably using the tx-mining-service, and propagate the tx using the API bellow.

PUT /txproposal/{txProposalId}/: Send the tx

Arguments:

Returns:

DELETE /txproposal/{txProposalId}: Cancel a tx proposal

When a tx proposal is closed, it should remove all locks from the UTXOs used by it.

Arguments:

Returns:

obiyankenobi commented 4 years ago

As we discussed, closing this issue in favor of #10, where the updated design was posted. This design here is already too long and we'll keep it for records.