Gateway Architecture Redesign

[phbrgnomo]

Current Gateway architecture might not be good enough to support others than Ethereum protocols.

For gateway to be more "universal", it must have a code architecture that allows:

• Easy connection to different blockchains
• Have a proper interface to allow communication between Python strategies and JS protocols interactions
• Allow "universal" point of connection between blockchain protocols and any program written with Python.

The end goal is to make Gateway a universal tool to be used not only with Hummingbot but work as a "translator" between blockchain protocols/smart contracts and any program based on Python.

[fengtality]

I will write a design doc and then pass to @Nullably and @martinkou for review

[evseevnn]

@mifeng Any updates with this? Actually I had to take a look on gateway-api in last 2 days and I see a lot of problem in code. Look like code written by Python developers.

So, if Hummingbot planned some plan about new architecture of gateway-api maybe I can see it and help in some cases?

[evseevnn]

@mifeng @martinkou

I looked inside the current api design and I have some idea about building the next implementation.

I can offer a few ways for realization of next gen gateway-api.

1. Standard REST API + Swagger client generation
2. Using GraphQL to make our API more flexible
3. Using gRPC

1. Standard REST API + Swagger client generation

This old school way is easy for understanding and can be used by any external/internal service.

In this case really important made API by strong standards because in other cases this can become trash. We need to keep high quality standards and keep api scalable.   In this case our users can use client SDK which will be generated from Swagger data.

I think this is possible by using any Typescript backend framework with Swagger generator. A lot of frameworks can provide this feature, TSED or NestJS as examples.

Props

Good:

• Easy to understand
• Auto generated client SDK based on Swagger for every language
• Scalable

Bad:

• Old school way
• Important keep REST API standards of project

2. Using GraphQL for make our API more flexible

GraphQL is gool query language for api.

GraphQL api can be really flexible and scalable, but this can require a little bit more effort for understanding and using.

We also can use the generated library for access to API from many programming languages like with REST + Swagger.

In this case we also can use many backend frameworks with Typescript.

So, that can be a good choice for gateway api, but that makes our team read more about GraphQL and about how to use GraphQL API in Python.

Props

Good:

• Modern way
• Easy to extends
• Can return many data by one query
• Strong typesation and validation out of box
• A lot of tools for client SDK generation or using based on schema
• No need make POST instead GET only for hide data from client =) (I see this way in code)

Bad:

• Difficult to start without any experience
• Need make some changes in backend of bot
• Not extremely fast (but very often this not depend on GraphQL but depend on developer)

Using gRPC

Good way for using gateway as microservice only, like microservice for backend of bot.

In this case it looks like the best way to start it as part of the system in docker. So, in this way this service will be just one more component of the backend for the bot.

Because we use gRPC this should be easy to extends and safety. Communication between bot and gateway-api will fast. But in this case we need to implement support of this protocol for bot and nodejs application.

I'm sure what we also can and should use Typescript for this service.

Good way for using this microservice only like microservice for backend of bot.

Props

Good:

• Safety
• Good way for makes communications between parts of system

Bad:

• Difficult to start without any experience
• Need make some changes in backend of bot
• Not easy find community support

Final thoughts

So, actually I see a way with GraphQL more profitable, but in case if nobody in the team wanna use it I think the best way will be standard REST API.

We can have additional call with yours for speaking about that.

[evseevnn]

After looking at the current design, I realized that you need to develop a standardized api for exchange transactions across decentralized exchanges.

So, looks like REST API that what you need. I on vacation, so, I started working on that.

[Nullably]

https://www.notion.so/hummingbot/Gateway-API-Improvements-31cb6fdbf8354ec2a4bd27448b1c7d76#180ba9cd753546faa9bb5b1d705f9597

[james-hummingbot]

Based on the notion document, here are few things that can be turned into tasks for the Gateway API, some are pulled directly from the document. I'll make another comment for hummingbot specifically.

gateway-api tasks

1. Introduce a linter and apply it to all of the current JS code. I don't have strong opinions, we could use ESLint unless someone has a reason not to. Then enforce a pre-commit check. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/63

2. Add typescript development tools to the repo and convert all files to .ts. Fix only what is necessary in the initial PR and then gradually add types. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/64

3. After number 2, build a base API class that is typed. It represents the generic routes that every exchange should implement. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/65

4. Refactor the default error catching for API functions into a TypeScript function decorator, or simply make it a function wrapper. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/72

5. Implement a custom winston logging transport to allow error logging to a WebSocket server - which Hummingbot can listen on and forward gateway errors to its own CLI. (would be nice if this can also be redirected to stdout/stderr for dev purposes). https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/68

6. Create mocks for each DEX, then create a few basic unit tests based on the most common operations, and the most common failures (e.g. not enough ETH to pay gas, insufficient balance for trading, timeouts, etc.). Make a PR per set of DEX API routes. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/69

7. Add Github workflow to automate unit testing into a CI process. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/70

8. Add JSdoc and document the main functionality. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/73

9. Add swagger to document the APIs. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/71

10. Add a tool that measures test coverage. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/66

11. Add a tool that measure documentation coverage. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/67

[evseevnn]

Nice points @james-hummingbot, @Nullably, @phbrgnomo

Actually I already implemented some part of new version gateway-api. Pls, take a look and I ready for make discuss about that.

https://github.com/evseevnn/hummingbot-gateway-api

[james-hummingbot]

Thanks @evseevnn for sharing your repo. I think there are a lot of good ideas in there on how to structure gateway-api better and in a more type safe way. Let me think a bit more about how to incorporate this into our repo.

[evseevnn]

Thanks @evseevnn for sharing your repo. I think there are a lot of good ideas in there on how to structure gateway-api better and in a more type safe way. Let me think a bit more about how to incorporate this into our repo.

@phbrgnomo actually I planned reimplement current api and then we can just replace current implementation without changing anythings on Hummingbot side. Then next step will working on protocol of connection to gateway and another refactoring and improvments.

I think this is good start for our redesign task.

[james-hummingbot]

hummingbot client tasks

I think it makes sense to start on these after the gateway-api tasks are completed. Create the following commands for hummingbot. We will likely need some new corresponding routes in gateway-api.

1. gateway up - Sets up SSL certificates and asks for initial configurations if not already set up. Starts the gateway. Prompts for the SSL certificates passphrase for start up if needed. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/74

2. gateway status - Prints the current gateway API status, including an overview of its configurations. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/75

3. gateway config - Modifies the gateway configuration. Restarts the gateway if any configurations has been modified. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/76

4. gateway down - Stops the gateway. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/77

5. gateway reset - Deletes the gateway container, and any gateway related configurations including the SSL certificates. https://app.zenhub.com/workspace/o/coinalpha/gateway-api/issues/78

6. For developers, we should allow a special config for Hummingbot to connect to a pre-defined gateway API URL. gateway up will simply instruct Hummingbot to connect to the pre-defined gateway. All other gateway commands become no-op. https://app.zenhub.com/workspaces/hummingbot/issues/coinalpha/hummingbot/3375

[james-hummingbot]

Redesign doc https://www.notion.so/hummingbot/Gateway-architecture-redesign-e768c4208207497cbbb9be035ae9b4da

[james-hummingbot]

@phbrgnomo This issue itself was turned into a discussion issue. Then I split it into many issues in the epic that this issue belongs to.

[phbrgnomo]

@james-hummingbot ok, I will then close this issue and move the discussion to the related Epic (#44)