subscriptions: Encapsulate, replace Pair with Pairs and refactor; improve exchange support

gbjk commented 4 months ago

Move Subscriptions to their own encapsulated store, list, and package This work starts the process of disentangling subscriptions from websocket, since a subscription is an exchange concept, and we want the collection of subscriptions separate from each websocket so we can start to separate auth, non-auth, assets, private and multiplexed websockets
Move Subscription Pair to Pairs For some exchanges we've immediately made full use of this. For others we've ported over the existing one-pair method, and will revist it when we tackle subscription configuration
Subscription methods all use pointers, and mutexes exist on the store and on the subscriptions to allow this
Subscription configuration support for Bitmex and CoinbasePro
Fixes and improvements to many exchanges' subscription logic as a driveby
Linter: Disable testify.Len
Tests: Fix WsAuth turned off by config checking
Tests: TestFixtureToDataHandler preserve WS
Websocket: Log actual sent message when Verbose

Type of change

[x] Non-backwards compatible feature

How has this been tested

[x] go test ./... -race
[x] golangci-lint run

codecov[bot] commented 4 months ago

Codecov Report

Attention: Patch coverage is 37.95380% with 752 lines in your changes are missing coverage. Please review.

Project coverage is 36.02%. Comparing base (c967d8a) to head (c45b65e). Report is 23 commits behind head on master.

:exclamation: Current head c45b65e differs from pull request most recent head 913fb2d

Please upload reports for the commit 913fb2d to get more accurate results.

Additional details and impacted files

[![Impacted file tree graph](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501/graphs/tree.svg?width=650&height=150&src=pr&token=41784B23TS&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp)](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp) ```diff @@ Coverage Diff @@ ## master #1501 +/- ## ========================================== - Coverage 36.04% 36.02% -0.03% ========================================== Files 411 415 +4 Lines 176906 177039 +133 ========================================== Hits 63774 63774 - Misses 105351 105475 +124 - Partials 7781 7790 +9 ``` | [Files](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp) | Coverage Δ | | |---|---|---| | [currency/pair.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=currency%2Fpair.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-Y3VycmVuY3kvcGFpci5nbw==) | `86.40% <ø> (ø)` | | | [currency/pairs.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=currency%2Fpairs.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-Y3VycmVuY3kvcGFpcnMuZ28=) | `98.14% <100.00%> (+0.04%)` | :arrow_up: | | [exchanges/bitfinex/bitfinex\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fbitfinex%2Fbitfinex_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2JpdGZpbmV4L2JpdGZpbmV4X3dyYXBwZXIuZ28=) | `37.16% <100.00%> (ø)` | | | [exchanges/bithumb/bithumb\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fbithumb%2Fbithumb_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2JpdGh1bWIvYml0aHVtYl93cmFwcGVyLmdv) | `29.38% <100.00%> (ø)` | | | [exchanges/btcmarkets/btcmarkets.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fbtcmarkets%2Fbtcmarkets.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2J0Y21hcmtldHMvYnRjbWFya2V0cy5nbw==) | `24.14% <ø> (ø)` | | | [exchanges/exchange.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fexchange.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2V4Y2hhbmdlLmdv) | `77.08% <100.00%> (ø)` | | | [exchanges/kucoin/kucoin\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fkucoin%2Fkucoin_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2t1Y29pbi9rdWNvaW5fd3JhcHBlci5nbw==) | `34.59% <100.00%> (+0.05%)` | :arrow_up: | | [exchanges/sharedtestvalues/sharedtestvalues.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fsharedtestvalues%2Fsharedtestvalues.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL3NoYXJlZHRlc3R2YWx1ZXMvc2hhcmVkdGVzdHZhbHVlcy5nbw==) | `71.31% <100.00%> (-0.66%)` | :arrow_down: | | [exchanges/subscription/keys.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fsubscription%2Fkeys.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL3N1YnNjcmlwdGlvbi9rZXlzLmdv) | `100.00% <100.00%> (ø)` | | | [exchanges/subscription/list.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fsubscription%2Flist.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL3N1YnNjcmlwdGlvbi9saXN0Lmdv) | `100.00% <100.00%> (ø)` | | | ... and [39 more](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp) | | ... and [14 files with indirect coverage changes](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp)

codecov-commenter commented 1 month ago

Codecov Report

Attention: Patch coverage is 37.97468% with 833 lines in your changes missing coverage. Please review.

Project coverage is 36.26%. Comparing base (afb6f75) to head (ccf98f4).

Additional details and impacted files

[![Impacted file tree graph](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501/graphs/tree.svg?width=650&height=150&src=pr&token=41784B23TS&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp)](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp) ```diff @@ Coverage Diff @@ ## master #1501 +/- ## ========================================== + Coverage 36.21% 36.26% +0.05% ========================================== Files 415 419 +4 Lines 182979 183167 +188 ========================================== + Hits 66261 66432 +171 + Misses 108637 108624 -13 - Partials 8081 8111 +30 ``` | [Files](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp) | Coverage Δ | | |---|---|---| | [exchanges/binance/binance\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fbinance%2Fbinance_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2JpbmFuY2UvYmluYW5jZV93cmFwcGVyLmdv) | `38.45% <100.00%> (ø)` | | | [exchanges/bithumb/bithumb\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fbithumb%2Fbithumb_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2JpdGh1bWIvYml0aHVtYl93cmFwcGVyLmdv) | `29.68% <100.00%> (ø)` | | | [exchanges/bitmex/bitmex\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fbitmex%2Fbitmex_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2JpdG1leC9iaXRtZXhfd3JhcHBlci5nbw==) | `44.85% <100.00%> (+0.66%)` | :arrow_up: | | [exchanges/btcmarkets/btcmarkets.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fbtcmarkets%2Fbtcmarkets.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2J0Y21hcmtldHMvYnRjbWFya2V0cy5nbw==) | `24.14% <ø> (ø)` | | | [exchanges/coinbasepro/coinbasepro\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fcoinbasepro%2Fcoinbasepro_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2NvaW5iYXNlcHJvL2NvaW5iYXNlcHJvX3dyYXBwZXIuZ28=) | `40.05% <100.00%> (+0.59%)` | :arrow_up: | | [exchanges/exchange.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fexchange.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2V4Y2hhbmdlLmdv) | `76.54% <100.00%> (ø)` | | | [exchanges/kucoin/kucoin\_wrapper.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fkucoin%2Fkucoin_wrapper.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL2t1Y29pbi9rdWNvaW5fd3JhcHBlci5nbw==) | `34.85% <100.00%> (+0.05%)` | :arrow_up: | | [exchanges/sharedtestvalues/sharedtestvalues.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fsharedtestvalues%2Fsharedtestvalues.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL3NoYXJlZHRlc3R2YWx1ZXMvc2hhcmVkdGVzdHZhbHVlcy5nbw==) | `63.00% <100.00%> (-2.10%)` | :arrow_down: | | [exchanges/subscription/list.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=exchanges%2Fsubscription%2Flist.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-ZXhjaGFuZ2VzL3N1YnNjcmlwdGlvbi9saXN0Lmdv) | `100.00% <100.00%> (ø)` | | | [gctrpc/rpc.pb.go](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree&filepath=gctrpc%2Frpc.pb.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp#diff-Z2N0cnBjL3JwYy5wYi5nbw==) | `0.64% <ø> (ø)` | | | ... and [39 more](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp) | | ... and [14 files with indirect coverage changes](https://app.codecov.io/gh/thrasher-corp/gocryptotrader/pull/1501/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thrasher-corp)

gbjk commented 1 month ago

One weird test failure: https://github.com/thrasher-corp/gocryptotrader/actions/runs/9265643096/job/25488105275?pr=1501#step:8:1134 I haven't noted it before so thought I'd raise it.If this is fine/unrelated, then this PR is good for me

I can replicate this issue on master, so it's not related to this PR.

Looking at it, I think that fetchBookViaREST was triggered from applyBufferUpdate during TestPushData, and that raced. I can replicate this occasionally locally using go test -race.

gbjk commented 1 month ago

Deribit support for subscriptions.Pairs added

thrasher-corp / gocryptotrader