search

NebulousLabs / Sia

Blockchain-based marketplace for file storage. Project has moved to GitLab: https://gitlab.com/NebulousLabs/Sia
https://sia.tech
MIT License
2.71k stars 440 forks source link

API Functions for Exchange Implementation #576

Closed poloniex closed 9 years ago

poloniex commented 9 years ago

Some additional features are needed for exchange implementation to be possible:

  1. An API method that lists all transactions the wallet is involved with, including both sending and receiving. Each transaction listed should include at minimum the amount, the number of confirmations, the destination address, the transaction ID/hash, and the timestamp.
  2. Methods for wallet encryption. This could be like bitcoin, where there is an unlock call and a lock call, or the encryption key/passphrase could be supplied as a param of the /wallet/send method.
DavidVorick commented 9 years ago

@poloniex

Sorry it took so long to get here, but I'm working on the wallet right now. The new wallet should be coded up by Aug. 1st, and the first release with the changes should be on Aug. 15th.

Wanted to run the api calls by you though.

/wallet (GET) :: display the balance of the wallet, both siacoins and siafunds, potentially with additional information

I wasn't planning on having this call list all transactions related to the wallet, it seemed unwieldy.

/wallet/address (GET) :: list all active addresses (and maybe their respective balances)

/wallet/address (POST) :: returns a new address that can receive coins

/wallet/$(ADDR) :: display the balance of an address, and return every txn related to the address. Each txn will also have a number of confirmations. The receiver will need to parse the transactions to figure out how many confirmations the balance as a whole has (or some subset of the balance). In the simplest case, this just means looking at the # of confirmations on each transaction and then taking the lowest number - that will give you the number of confirmations on the balance.

From the transaction you will be able to parse the amount, the confirmations, the source(s), the destination, the ids (both of the outputs and of the transaction), and the timestamp, as well as the actual raw transaction.

/wallet/siacoins/send (POST) :: requires a password, which will be established by the user when the wallet is created. The wallet will temporarily decrypt the private key, send the coins, and then wipe the decrypted key from memory.

/wallet/siafunds/send (POST) :: same as /wallet/siacoins/send, except it sends siafunds.

There will be a few other calls related to backups, setting passwords, and similar functions, but the above are the ones related to sending and receiving coins.

poloniex commented 9 years ago

If /wallet/siacoins/send automatically selects inputs from all addresses in the wallet (rather than requiring a single source address to be specified), then an exchange will need a method to list all transactions in the wallet. If you provide only the methods proposed above, I would need to construct this list using /wallet/$(ADDR) by querying every deposit address, collecting the txns, and (if /wallet/$(ADDR) provides only txns and no further transaction information) fetching the amount, destination, and confirmations of each transaction. As more deposit addresses are generated, this could end up becoming very slow; it is preferable to minimize the number of API queries needed.

Unless sending funds must be done by specifying both a source and a destination address, knowing the balances of each address in the wallet is actually of little use to an exchange; because trades happen off the blockchain, the balance of a user's deposit address rarely matches that user's balance on the exchange or even the sum of that user's deposits.

DavidVorick commented 9 years ago

I am concerned about the size of a /wallet/transactions call, it could potentially exceed the amount that you could safely send over an API.

I guess we could add pagination, so that if the size exceeds 1mb or something, it'd be broken up into several pages, and you could specify a page with each call. I think that's reasonable.

poloniex commented 9 years ago

Most wallet APIs provide a way to limit the number of transactions returned. Bitcoin's allows you to specify such a number, and will return the latest n transactions. Others allow you to specify a starting block, sometimes an ending block as well. The latter is ideal -- specifying a starting block would return all transactions between that block and the latest or a specified ending block.

DavidVorick commented 9 years ago

Working on it now, I think I can have everything you've request ready in the next release.

Timestamps on the transactions are vague. Do you want the time that the transaction was first seen, or the timestamp on the block it appeared in? Transactions in Sia don't have a timestamp on their own. First block appearance would be pretty easy, but timestamp first seen would be harder.

poloniex commented 9 years ago

First block appearance is fine.

DavidVorick commented 9 years ago

For the most part, the implementation is complete. I still need to update the actually http api, but the backend is ready.

All wallet keys and seeds are encrypted on disk. After loading, the wallet needs to be unlocked. The wallet won't start until it has been unlocked. The wallet is unlocked using a password/key. The wallet can be relocked by calling 'Close', which will wipe all of the keys from memory (to the extent possible. Because of context switching, swap space, and other confounding variables, it's possible that keys will be in memory unreachable to siad) You can call unlock and close as many times as you like.

When requesting transaction history, you can either ask for all transactions, or you can ask for transactions between two block heights [start, end]. Instead of actual transactions being returned, a 'Wallet Transaction' will be returned, which contains a single siacoin or siafund input or output. It also contains the full transaction (which means there's high redundancy for large transactions - this could perhaps be replaced by using transaction ids).

As an example, transaction A might have 3 inputs, 1 output, and 1 refund. 5 wallet transactions will be created. One for each input, and one for each output. Each will specify the type of movement (input vs. output, and siacoin vs. siafund), the address involved (generally only useful for incoming coins (appearing as outputs in a transaction), as the wallet arbitrarily selects addresses when creating inputs to a transaction), and the value. Each wallet transaction contains a timestamp and a confirmation height (to get the number of confirmations, you'll need to compare to the current height).

All 5 are guaranteed to appear consecutively in the transaction history.

When requesting a balance, you can request 'ConfirmedBalance' and 'UnconfirmedBalance'. Confirmed balance currently will only look at 1 confirmation, other than parsing the transaction history yourself you can't get 6 confirmation balance right now. Unconfirmed balance will return the number of incoming and outgoing siacoins, which will include refunds.

For example, if you create an unconfirmed transaction that sends 5 coins, and a 500 coin input is used, the result reported will be 500 outgoing ,and 495 incoming.

Hope this is all okay. Let me know if there are changes you think should be made.

DavidVorick commented 9 years ago

Queries:

/wallet [GET]

Function: Returns basic information about the wallet, such as whether the wallet is locked or unlocked.

Parameters: none

Response:

struct {
    encrypted bool
    unlocked  bool
}

'encrypted' indicates whether the wallet has been encrypted or not. If the wallet has not been encrypted, then no data has been generated at all, and the first time the wallet is unlocked, the password given will be used as the password for encrypting all of the data. Encrypted will only be set to false if the wallet has never been unlocked before (the unlocked wallet is still encryped - but the encryption key is in memory).

'unlocked' indicates whether the wallet is currently locked or unlocked. Some calls become unavailable when the wallet is locked.

/wallet/close [PUT]

Function: Locks and closes the wallet, preparing for shutdown. After being closed, the keys are encrypted. Queries for the seed, to send siafunds, and related queries become unavailable. Queries concerning transaction history and balance are still available.

Parameters: none

Response: Standard.

/wallet/history [GET]

Function: Return a list of transactions related to the wallet.

Parameters:

struct {
    start int
    end   int
}

Response:

struct {
    transactions []WalletTransaction
}

'transactions' is a list of 'WalletTransactions'. Wallet transactions are transactions that have been processed by the wallet and given more information, such as a confirmation height and a timestamp. Each wallet transaction contins information about only a single input or output. One network transaction with many inputs an outputs can result in many wallet transactions. All of the wallet transactions created by a network transaction are guaranteed to be consecutive in history. Wallet transactions will always be returned in chronological order.

A wallet transaction takes the following form:

struct WalletTransaction {
    TransactionID         string
    ConfirmationHeight    int
    ConfirmationTimestamp uint64

    FundType       string
    OutputID       string
    RelatedAddress string
    Value          int
}

'TransactionID' is the id of the transaction from which the wallet transaction was derived. The full transaction can be obtaied by calling '/wallet/transaction/$(id)'

'ConfirmationHeight' is the height at which the transaction was confirmed. The height will be set to 'uint64MAX' if the transaction has not been confirmed.

'ConfirmationTimestamp' is the time at which a transaction was confirmed. The timestamp is a 64bit unix timestamp, and will be set to uint64MAX if the transaction is unconfirmed.

'FundType' indicates what type of fund is represented by the wallet transaction. The options are 'Siacoin Input', 'Siacoin Output', 'Siafund Input', and 'Siafund Output', corresponding to whether the fund is related to siacoins or siafunds, and to whether the fund is an input to the transaction or an output of the transaction. When looking at transactions, it should be noted that a 'Siacoin Input' actually represents outgoing siacoins, as they are an input to the transaction; they are being spent. A 'Siacoin Output' represents incoming coins, because the output is being created by the transaction and made available to the wallet.

'OutputID' is the id of the output. OutputIDs will always be unique.

'RelatedAddress' is the address that is affected. For inputs (outgoing money), the related address is usually not important because the wallet arbitrarily selects which addresses will fund a transaction. For outputs (incoming money), the related address field can be used to determine who has sent money to the wallet.

'Value' indicates how much money has been moved in the input or output.

/wallet/history/$(addr) [GET]

Function: Return all of the transaction related to a specific address.

Parameters: none

Response:

struct {
    transactions []WalletTransaction
}

'transactions' is a list of 'WalletTransactions' that affect the input address. See the documentation for '/wallet/history' for more information.

/wallet/seed [GET]

Function: Return the seed that is being used to generate addresses. This seed can be used to derive secret keys, and must be kept safe. This call is unavailable when the wallet is locked or closed.

Parameters:

struct {
    dictionary string
}

'dictionary' is the name of the dicitionary that should be used when encoding the seed.

Response:

struct {
    primarySeed        string
    addressesRemaining int
    backupSeeds        []string
}

'primarySeed' is the seed that is actively being used to generate new addresses for the wallet.

'addressesRemaining' is the number of addresses that remain in the primary seed until exhaustion has been reached and no more addresses will be generated.

'backupSeeds' is a list of seeds that are no longer used to generate new addresses, but are still tracked can have spendable outputs.

A seed is an encoded version of a 128 bit random seed. The output is 15 words chosen from a small dictionary as indicated by the input. The most common choice for the dictionary is going to be 'english'. The underlying seed is the same no matter what dictionary is used for the encoding. The encoding also contains a small checksum of the seed, to help catch simple mistakes when copying. The library entropy-mnemonics is used when encoding.

/wallet/seed [PUT]

Function: Get a new address from the wallet generated by the primary seed. An error will be returned if the seed has been exhausted (there is a limit on the number of addresses that can be generated from one seed), or if the wallet is locked/closed.

Parameters: none

Response:

struct {
    address string
}

'address' is the address that cna

/wallet/seed [POST]

Function: Fetch a new seed for the wallet. The old seed will be added to the list of backup seeds. In general, this call should be avoided unless address exhaustion is reached. Because making a new seed is considered a significant action, there is a verification field that must be set to true. This is to prevent mistakes e.g. calling POST instead of GET or PUT.

Parameters:

struct {
    verification bool
    dictionary   string
}

'verification' must be set to true, or an error is returned. This is to prevent an implementation mistake where POST is called instead of GET or PUT.

'dictionary' is the name of the dictionary that should be used when encoding the newly created seed. 'english' is the most common choice.

Response:

struct {
    newSeed string
}

'newSeed' is the new seed that will be used as the seed for generating new addresses.

/wallet/siacoins [GET]

Function: get the siacoin balance (confirmed and unconfirmed) of the wallet.

Parameters: none

Response:

struct {
    confirmedSiacoinBalance     int
    unconfirmedOutgoingSiacoins int
    unconfirmedIncomingSiacoins int
}

'confirmedSiacoinBalance' is the number of siacoins available to the wallet as of the most recent block in the blockchain.

'unconfirmedOutgoingSiacoins' is the number of siacoins that are leaving the wallet according to the set of unconfirmed transactions. Often this number appears inflated, because outputs are frequently larger than the number of coins being sent, and there is a refund. These coins are counted as outgoing, and the refund is counted as incoming. The difference in balance can be claculated using 'unconfirmedIncomingSiacoins' - 'unconfirmedOutgoingSiacoins'

'unconfirmedIncomingSiacoins' is the number of siacoins are entering the wallet according to the set of unconfirmed transactions. This number is often inflated by outgoing siacoins, because outputs are frequently larger than the amount being sent. The refund will be included in the unconfirmed incoming siacoins balance.

/wallet/siacoins [PUT]

Function: Send siacoins to an address. The outputs are arbitrarily selected from addresses in the wallet.

Parameters:

struct {
    amount      int
    destination string
}

'amount' is the number of siacoins being sent.

'destination' is the address that is receiving the coins.

Response: standard

/wallet/siafunds [GET]

Function: get the siafund balance of the wallet. Only the confirmed balance is displayed for siafunds.

Parameters: none

Response:

struct {
    siafundBalance      int
    siacoinClaimBalance int
}

'siafundBalance' is the number of siafunds available to the wallet as of the most recent block in the blockchain.

'siacoinClaimBalance' is the number of siacoins that can be claimed from the siafunds as of the most recent block. Because the claim balance increases every time a file contract is created, it is possible that the balance will increase before any claim transaction is confirmed.

/wallet/siafunds [PUT]

Function: Send siafunds to an address. The outputs are arbitrarily selected from addresses in the wallet. Any siacoins available in the siafunds being sent (as well as the siacoins available in any siafunds that end up in a refund address) will become available to the wallet as siacoins after 144 confirmations. To access all of the siacoins in the siacoin claim balance, send all of the siafunds to an address in your control (this will give you all the siacoins, while still letting you control the siafunds).

Parameters:

struct {
    amount      int
    destination string
}

'amount' is the number of siafunds being sent.

'destination' is the address that is receiving the funds.

Response: standard

/wallet/transaction/$(id) [GET]

Function: Get the transaction associated with a specific transaction id.

Parameters: none

Response:

struct {
    transaction Transaction
}

'transaction' is a 'types.Transaction'. The full transaction can be seen in types.transaction.go. All hashes in the transaction are encoded as strings.

/wallet/unlock [PUT]

Function: Unlock the wallet. The wallet is capable of knowing whether the correct password was provided. The first time that the wallet is unlocked ever, the password used will become the permanent password. Any string can be used as the key, though it is generally recommended that the encoded primary seed be used.

Parameters:

struct {
    key string
}

Response: standard

DavidVorick commented 9 years ago

@poloniex @agundy @Mingling94 @lukechampine

the above comment contains the rough draft of the specification for the wallet api. If there is anything that is unusual, unusable, or you think could be improved, please let me know.

@poloniex, let me know if this adds all of the features that you will need to add Sia to an exchange.

The majority of this has been implemented already.

poloniex commented 9 years ago

The wallet transaction thing is a little cumbersome for exchange purposes. The API described is sufficient for an exchange implementation, but I will have to construct the list of transactions I described myself -- either by adding up all the wallet transactions, or fetching transactions individually by id (probably the former).

Mingling94 commented 9 years ago

I think the API could be less repetitious and more logically consistent by combining /wallet/siacoins [PUT] and /wallet/siafunds [PUT] into one

/wallet/transaction [POST]

struct {
    amount      int
    destination string
    currency    string
}

while also combining /wallet/siacoins [GET] and /wallet/siafunds [GET] into

/wallet/balance [GET]

struct {
    confirmedSiacoinBalance     int
    unconfirmedOutgoingSiacoins int
    unconfirmedIncomingSiacoins int
    siafundBalance              int
    siacoinClaimBalance         int
}
DavidVorick commented 9 years ago

People should not make their own passwords. People make grossly insecure passwords, and dictionary attacks are extremely powerful. While they could make their own password when locking the wallet, the vast majority of people make passwords (even at 24+ characters!) that can be guessed in a relatively short amount of time using a dictionary attack. That's why we recommend you use the seed password - it's got a full 128 bits of entropy and is immune to dictionary attacks.

I could put all of the balance stuff into the /wallet [GET] call.

I'm not sure that it makes sense to combine the /siacoins and /siafunds calls when sending money.

Mingling94 commented 9 years ago
DavidVorick commented 9 years ago

The new api functions have been implemented and merged into master. Documentation on the calls can be found in doc/API.md.

Further discussion/requests can be continued by opening a new issue.