Prepare roll out of API for January or February release

chimp1984 commented 3 years ago

_This is a Bisq Network project. Please familiarize yourself with the project management process._

Description

Rationale

API seems to be pretty complete now, so lets prepare the last steps for deployment.

Criteria for delivery

API usage is sufficiently documented and announced. Protection tools are in place.

Measures of success

Users are using the API, no major problems.

Risks

Scripts could cause heavy load for network if incorrectly used.
Trade peers can suffer in case of bugs.
Hidden bugs can be revealed by using the network and protocols in a different manner as from the UI app.
Security risks if a malicious actor gets access to API users wallet.

Tasks

Protection

[ ] Add a flag to preferences to deny API takers
[ ] Add a enum entry to AvailabilityResult in case the maker denied API takers
[ ] Add "disable API" flag to filter, which would prevent trading with API users
[ ] API checks if API is disabled by filter
[ ] API handles AvailabilityResult.MAKER_DENIED_API_USER response
[ ] Add more fields to filter for network wide filtering of certain data

@ghubstan Can you add the protection mechanism which are in place from the API side and which are planned but not implemented yet? E.g. protect against endless loops,...

Track API usage

[ ] Add referral ID to offer so we can track how much API is used in offers
[ ] Maybe show icon in offer book (not sure if that is needed?)
[ ] Add a field to trade statistics so we can track how much API is used in trades

Documentation

[ ] Add descition about security model (authentication,...)
[ ] Add wiki/docs page for overview (maybe @ghubstan can provide basic content and @m52go finalizes it?)

Scripts:

[ ] Add a few simple example scripts for usage. Some use cases for added value what cannot be done in the UI would be good like "create offer if price > x"
[ ] Find some tech savvy traders who are willing to Beta test and provide input to scripts
[ ] Find some traders experienced with APIs to provide some sample scripts

Estimates

I will work on the app related protection tasks. I estimate 2000 USD. @sqrrm , @ghubstan , @m52go Could you add your estimates as well?

Notes

@ghubstan @sqrrm @cbeams @ripcurlx @m52go @pazza83 Do you have any further input?

Edited some dev tasks...

m52go commented 3 years ago

For documentation, I estimate 250-1000 USD depending on quantity and quality of the starting point.

Is this for the full, public rollout? Will there be some kind of intermediate pre-release phase before a big rollout? Asking mainly for messaging/communication purposes.

chimp1984 commented 3 years ago

Is this for the full, public rollout? Will there be some kind of intermediate pre-release phase before a big rollout? Asking mainly for messaging/communication purposes.

I think its more safe to make a soft roll out for at least one releaze cycle to not risk too much. So PR should be help back until we are sure that all works smoothly.

chimp1984 commented 3 years ago

Started implementation at: https://github.com/bisq-network/bisq/pull/5053

pazza83 commented 3 years ago

This sounds great. Happy to help with testing if I can. Where is the best place to get a background of what has been developed to date?

ghubstan commented 3 years ago

Where is the best place to get a background of what has been developed to date?

@pazza83 Sorry, no api documentation yet, but the code explains a lot. See :cli supbroject's Method class for a list of currently available CLI commands:

public enum Method {
    canceloffer,
    confirmpaymentreceived,
    confirmpaymentstarted,
    createoffer,
    createpaymentacct,
    getaddressbalance,
    getbalance,
    getfundingaddresses,
    getmyoffer,
    getmyoffers,
    getoffer,
    getoffers,
    getpaymentacctform,
    getpaymentaccts,
    getpaymentmethods,
    gettrade,
    gettransaction,
    gettxfeerate,
    getunusedbsqaddress,
    getversion,
    keepfunds,
    lockwallet,
    registerdisputeagent,
    removewalletpassword,
    sendbsq,
    sendbtc,
    settxfeerate,
    setwalletpassword,
    takeoffer,
    unlockwallet,
    unsettxfeerate,
    withdrawfunds
}

This won't work on Windows (not meant to). It's meant for *nix only. Seemed to work OK when I tried it on OSX some months ago.

To see (linux) CLI help, get latest master, and build with options needed for running api tests later:

./gradlew clean build :apitest:installDaoSetup

CLI help: ./bisq-cli --help

To see server side implementation, you can drill down starting from the :daemon subproject's bisq.daemon.grpc.GrpcServer.

If you want (and you have bitcoin-core v19 or v20 in your $PATH) you can run the :apitest suite (almost 5 minutes to complete):

./gradlew :apitest:test -DrunApiTests=true

The test cases are in the :apitest src/test folder, the src/main (ignore it) folder is where the test harness lives. If you want to run individual method tests you'll need to comment out the @Disabled annotation. And you'll need to add the JVM argument -DrunApiTests=true to your IDE's Gradle launcher template.

Again, this test harness only works on *nix, and depends on bash. Windows support is not in the scope of the api project at this stage.

I am working on a simple trading script that simulates Bob & Alice using the CLI to buy and sell BTC with EUR. But I haven't checked it in yet -- it needs more testing, load limits are being found as I test, and some api protections have been implemented, but not clearly defined and applied to specific api methods yet.

ghubstan commented 3 years ago

Can you add the protection mechanism which are in place from the API side and which are planned but not implemented yet? E.g. protect against endless loops,...

@chimp1984 The feature is implemented, using a configured CallRateMeteringInterceptor on a GrpcOffersService, for example, but has not been applied. We discussed on 3-Jan in keybase, and you mentioned you will take a look at the related :daemon classes.

We just need to decide how many X calls per Y should be defined, where X = api method, and Y = time-window (second, 10 seconds, 10 minutes, 1 day, 10 days, etc.).

ghubstan commented 3 years ago

@ghubstan Can you add the protection mechanism which are in place from the API side and which are planned but not implemented yet? E.g. protect against endless loops,...

@chimp1984, PR https://github.com/bisq-network/bisq/pull/5103 defines the rate meter constraints on all api methods.

ghubstan commented 3 years ago

Add wiki/docs page for overview (maybe @ghubstan can provide basic content and @m52go finalizes it?)

I intend to create an api overview doc this week, for @m52go.

Since this issue was posted, method level help has been merged into the main branch.

Method Help Usage: bisq-cli [options] <method> --help

Examples:

./bisq-cli --password=xyz --port=9998 getbtcprice --help

./bisq-cli --password=xyz --port=9998 createoffer --help

ghubstan commented 3 years ago

Add a few simple example scripts for usage. Some use cases for added value what cannot be done in the UI would be good like "create offer if price > x"

There are two example scripts in the main branch (apitest/scripts):

trade-simulation.sh -- emulates Bob & Alice trading regtest BTC, using a country based F2F account
limit-order-simulation.sh -- emulates creating regtest buy or sell offer when a limit price has been reached

ghubstan commented 3 years ago

My estimate for docs + trading scripts is 4500-5500 USD.

Conza88 commented 3 years ago

Q - the Android app, after this API release - it makes it possible for it to take offers, make offers etc. on that?

bisq-network / projects