websocket/gateio: Multiple connection management and gateio integration.

shazbert commented 1 month ago

WebSocket

Multiple Connections for Asset Endpoints

Feature: Implemented the ability to spawn multiple WebSocket connections for different asset endpoints.
Design Decision:
- Opted for a separate subscription holder per connection due to import cycle issues with a unified subscription manager.
- This allows for future scalability when exchange constraints limit subscriptions per connection.

Connection-Coupled Functionality

Approach: Coupled specific functionality with the connection itself, along with its own subscription holder.
Benefit: Enables tracking multiple connections efficiently, facilitating scaling when exchanges restrict subscriptions per connection.

Reader Routine in Streaming Package

Implementation: Created a reader routine in the streaming package to avoid code duplication.
Future Work: This will initiate a purge across wrappers to standardize this routine.

Context Awareness

Progress: Began integrating context awareness throughout the WebSocket functionality.
Enhancement: Added a DialContext method to WebsocketConnection to leverage the supported context.

Gate.io Enhancements

WebSocket Support

Added Support:
- Futures
- Dated Futures
- Options
Note: Margin, cross margin, and BTC-margined dated futures not included due to PR size and complexity.

Bug Fixes

Delivery Futures Handling:
- Fixed issue where all available pairs were fetched instead of only enabled pairs.
Payload Generation:
- Addressed issues in generateDeliveryFuturesPayload where different settlements were handled incorrectly.
- Separated connections for different settlements to flatten functions for consistency.

🐛 fix:

Type of change

Please delete options that are not relevant and add an x in [] as item is complete.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How has this been tested

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration and also consider improving test coverage whilst working on a certain feature or package.

[I ran it 🏃 instead] go test ./... -race
[ ] golangci-lint run
[ ] Test X

Checklist

[ ] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation and regenerated documentation via the documentation tool
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally and on Github Actions with my changes
[ ] Any dependent changes have been merged and published in downstream modules

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 57.55167% with 267 lines in your changes missing coverage. Please review.

Project coverage is 38.41%. Comparing base (cb6b342) to head (06acaac).

Files with missing lines	Patch %	Lines
exchanges/stream/websocket.go	80.06%	38 Missing and 22 partials :warning:
exchanges/gateio/gateio_ws_futures.go	18.84%	54 Missing and 2 partials :warning:
exchanges/gateio/gateio_ws_delivery_futures.go	2.27%	43 Missing :warning:
exchanges/gateio/gateio_ws_option.go	5.00%	18 Missing and 1 partial :warning:
exchanges/gateio/gateio_websocket.go	10.52%	16 Missing and 1 partial :warning:
exchanges/gateio/gateio_wrapper.go	71.18%	11 Missing and 6 partials :warning:
exchanges/kraken/kraken_websocket.go	0.00%	9 Missing :warning:
exchanges/stream/websocket_connection.go	52.94%	7 Missing and 1 partial :warning:
exchanges/okx/okx_websocket.go	0.00%	6 Missing :warning:
internal/testing/websocket/mock.go	84.00%	2 Missing and 2 partials :warning:
... and 14 more

Additional details and impacted files

[![Impacted file tree graph](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580/graphs/tree.svg?width=650&height=150&src=pr&token=41784B23TS&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp)](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp) ```diff @@ Coverage Diff @@ ## master #1580 +/- ## ========================================== + Coverage 36.50% 38.41% +1.91% ========================================== Files 414 415 +1 Lines 179353 149258 -30095 ========================================== - Hits 65477 57342 -8135 + Misses 106006 84021 -21985 - Partials 7870 7895 +25 ``` | [Files with missing lines](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp) | Coverage Δ | | |---|---|---| | [exchanges/binance/binance\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree&filepath=exchanges%2Fbinance%2Fbinance_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2JpbmFuY2UvYmluYW5jZV93cmFwcGVyLmdv) | `41.22% <100.00%> (+2.81%)` | :arrow_up: | | [exchanges/binanceus/binanceus\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree&filepath=exchanges%2Fbinanceus%2Fbinanceus_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2JpbmFuY2V1cy9iaW5hbmNldXNfd3JhcHBlci5nbw==) | `44.89% <100.00%> (+3.99%)` | :arrow_up: | | [exchanges/bitfinex/bitfinex\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree&filepath=exchanges%2Fbitfinex%2Fbitfinex_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2JpdGZpbmV4L2JpdGZpbmV4X3dyYXBwZXIuZ28=) | `41.75% <100.00%> (+4.39%)` | :arrow_up: | | [exchanges/bithumb/bithumb\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree&filepath=exchanges%2Fbithumb%2Fbithumb_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2JpdGh1bWIvYml0aHVtYl93cmFwcGVyLmdv) | `33.08% <100.00%> (+3.40%)` | :arrow_up: | | [exchanges/bitmex/bitmex\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree&filepath=exchanges%2Fbitmex%2Fbitmex_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2JpdG1leC9iaXRtZXhfd3JhcHBlci5nbw==) | `41.75% <100.00%> (+2.90%)` | :arrow_up: | | [exchanges/bitstamp/bitstamp\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree&filepath=exchanges%2Fbitstamp%2Fbitstamp_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2JpdHN0YW1wL2JpdHN0YW1wX3dyYXBwZXIuZ28=) | `59.20% <100.00%> (+5.97%)` | :arrow_up: | | [exchanges/btcmarkets/btcmarkets\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree&filepath=exchanges%2Fbtcmarkets%2Fbtcmarkets_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2J0Y21hcmtldHMvYnRjbWFya2V0c193cmFwcGVyLmdv) | `36.05% <100.00%> (+3.28%)` | :arrow_up: | | [exchanges/btse/btse\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree&filepath=exchanges%2Fbtse%2Fbtse_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2J0c2UvYnRzZV93cmFwcGVyLmdv) | `47.62% <100.00%> (+3.71%)` | :arrow_up: | | [exchanges/bybit/bybit\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree&filepath=exchanges%2Fbybit%2Fbybit_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2J5Yml0L2J5Yml0X3dyYXBwZXIuZ28=) | `47.05% <100.00%> (+3.85%)` | :arrow_up: | | [exchanges/coinbasepro/coinbasepro\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree&filepath=exchanges%2Fcoinbasepro%2Fcoinbasepro_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2NvaW5iYXNlcHJvL2NvaW5iYXNlcHJvX3dyYXBwZXIuZ28=) | `43.86% <100.00%> (+3.80%)` | :arrow_up: | | ... and [36 more](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp) | | ... and [346 files with indirect coverage changes](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1580/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp)

shazbert commented 1 month ago

This first comment from me covers the overall architecture.

Thanks for the suggestions. I appreciate the thought you've put into the overall architecture and the proposed changes you outlined should help provide a more logical structured framework.

I'd like to suggest for now, we proceed with the current implementation @thrasher- . Just so we can merge the PR and make it usable right away and can get Sam to start rolling that across the exchanges straight away. In the meantime we can start planning iterations to gradually evolve the architecture towards the model you've outlined?

So plan is, after this is merged I want to:

Make connections and subscriptions concurrent
Have N+ subs per "connection/websocket/whatever name its going to be"

I'd like to defer localizing the interface for the subscription package, just for now. I don't want to start blowing out complexity for this PR, there's already a bit of moving parts. Perhaps you could open a supplementary PR to address this when you have the time?

Thanks again for your input!

gbjk commented 1 month ago

Sounds good buddy 👍

samuael commented 1 month ago

https://github.com/shazbert/gocryptotrader/blob/6376a12b18b7db8ab6d7ab68864af89e8ab06342/exchanges/okx/okx_websocket.go#L254

This function is very similar to WsConnect, so it might be a good idea to treat WsAuth as a single connection and use the SetupConnection method call to initialize it. Additionally, you could consider moving the dialer and context information inside the function.

samuael commented 1 month ago

https://github.com/shazbert/gocryptotrader/blob/6376a12b18b7db8ab6d7ab68864af89e8ab06342/exchanges/stream/websocket.go#L901

Just something to consider for the next round of improvements: we can eliminate these methods by treating each connection as a single instance. This instance can be appended to the slice of ConnectionWrapper, thereby reducing the complexities created by Conn and AuthConn

shazbert commented 1 month ago

https://github.com/shazbert/gocryptotrader/blob/6376a12b18b7db8ab6d7ab68864af89e8ab06342/exchanges/okx/okx_websocket.go#L254

This function is very similar to WsConnect, so it might be a good idea to treat WsAuth as a single connection and use the SetupConnection method call to initialize it. Additionally, you could consider moving the dialer and context information inside the function.

Good suggestion, I think that would be the direction to go when implementing. I have started a proof of concept to grow connections based off the max subscriptions per connection and I think I have already done what you have suggested. Will send you a link to it here when it's ready for your look over.

shazbert commented 1 month ago

https://github.com/shazbert/gocryptotrader/blob/6376a12b18b7db8ab6d7ab68864af89e8ab06342/exchanges/stream/websocket.go#L901

Just something to consider for the next round of improvements: we can eliminate these methods by treating each connection as a single instance. This instance can be appended to the slice of ConnectionWrapper, thereby reducing the complexities created by Conn and AuthConn

Is this specifically pointing to the change in SetWebsocketURL?

thrasher-corp / gocryptotrader