soulmachine
commented
1 year ago Can you remove line numbers from your code snipet above? I'd like to run you code on my machine
Open panicfarm opened 1 year ago
soulmachine
commented
1 year ago Can you remove line numbers from your code snipet above? I'd like to run you code on my machine
panicfarm
commented
1 year ago Can you remove line numbers from your code snipet above? I'd like to run you code on my machine
Here you go
use crypto_crawler::{crawl_trade, MarketType};
#[tokio::main(flavor = "multi_thread")]
async fn main() {
let (tx, rx) = std::sync::mpsc::channel();
tokio::task::spawn(async move {
crawl_trade(
"kucoin",
MarketType::LinearSwap,
Some(&["SOLUSDTM".to_string()]),
/*
MarketType::Spot,
Some(&["SOL-USDT".to_string()]),
*/
tx,
)
.await
});
for msg in rx {
//https://github.com/crypto-crawler/crypto-crawler-rs/blob/main/crypto-crawler/src/msg.rs
let trades =
crypto_msg_parser::parse_trade(&msg.exchange, msg.market_type, &msg.json).unwrap();
let trade = &trades[0];
println!("{:#?}", trade);
}
}and
[package]
name = "qk"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
crypto-crawler = { git = "https://github.com/crypto-crawler/crypto-crawler-rs", branch = "main" }
crypto-msg-parser = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main" }
crypto-message = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main"}
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
foonsun
commented
1 year ago Can you remove line numbers from your code snipet above? I'd like to run you code on my machine
Here you go
use crypto_crawler::{crawl_trade, MarketType}; #[tokio::main(flavor = "multi_thread")] async fn main() { let (tx, rx) = std::sync::mpsc::channel(); tokio::task::spawn(async move { crawl_trade( "kucoin", MarketType::LinearSwap, Some(&["SOLUSDTM".to_string()]), /* MarketType::Spot, Some(&["SOL-USDT".to_string()]), */ tx, ) .await }); for msg in rx { //https://github.com/crypto-crawler/crypto-crawler-rs/blob/main/crypto-crawler/src/msg.rs let trades = crypto_msg_parser::parse_trade(&msg.exchange, msg.market_type, &msg.json).unwrap(); let trade = &trades[0]; println!("{:#?}", trade); } }and
[package] name = "qk" version = "0.1.0" edition = "2021" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] crypto-crawler = { git = "https://github.com/crypto-crawler/crypto-crawler-rs", branch = "main" } crypto-msg-parser = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main" } crypto-message = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main"} tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
I have the same issue.any resolutions? @panicfarm
soulmachine
commented
1 year ago Looking into this issue, will update
panicfarm
commented
1 year ago I have the same issue.any resolutions? @panicfarm
@soulmachine I believe the issue is here:
https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L152
fn fetch_linear_multipliers() is blocking
https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L178 but your example that I used is async (because your websocket is async), so this function (and thus crypto_msg_parser::parse_trade()) cannot be called from an asynchronous context.
The function is blocking because inside it contains blocking reqwest: https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/utils.rs#L7
You could make the entire crypto_msg_parser::parse_trade() async, although I think it's prefererrable to "instantly" parse the websocket messages, leaving this function sync, so maybe this blocking fetch_linear_multipliers reqwest can be avoided or asynchronously done before the websocket connection is established.
foonsun
commented
1 year ago I have the same issue.any resolutions? @panicfarm
@soulmachine I believe the issue is here: https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L152
fn fetch_linear_multipliers()is blocking https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L178 but your example that I used is async (because your websocket is async), so this function (and thuscrypto_msg_parser::parse_trade()) cannot be called from an asynchronous context.The function is blocking because inside it contains blocking reqwest: https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/utils.rs#L7
You could make the entire
crypto_msg_parser::parse_trade()async, although I think it's prefererrable to "instantly" parse the websocket messages, leaving this function sync, so maybe this blockingfetch_linear_multipliersreqwest can be avoided or asynchronously done before the websocket connection is established.
I think whether fetch_linear_multipliers is blocking or async is not important because it's only called once when called. static LINEAR_CONTRACT_VALUES: Lazy<HashMap<String, f64>> = Lazy::new(|| only request once. I prefer the async resolution because the app may use the async functions.
foonsun
commented
1 year ago any update? @soulmachine
soulmachine
commented
1 year ago Not yet, still working on it
I am using the following code (collected from your examples):
Only for MarketType::LinearSwap (but not for MarketType::Spot because https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L198 never gets called for Spot) I get the following panic with the debug build (this never happens with a release build, i think it's something debug-specific? with blocking
http_get("https://api-futures.kucoin.com/api/v1/contracts/active")reqwest in crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:181 ).PS. With the release build, i let it run for several days, but the websocket stopped getting new messages after 2 days. I rebuilt as a debug and ran into the panic above. Should I file a separate Issue on the stuck websocket (unsure how to debug that, strace showed waiting for some FUTEX)?