Closed jarrodjay closed 3 years ago
What quantity was purchased? I'm guessing it was about 0.55? The program is creating market orders, not limit, so if you have it set for $40 it's supposed to buy X amount of the coin at the current market price, which in your case was $72.10, wherein the total amount paid is $40.
As for the coin price itself, it looks like it spiked super high then almost immediately sold off. This is pretty much what this script is supposed to catch and the price wave of (once these dumb bugs are fixed).
I also had that error, always something 🙄 (I forgot to convert the strings to numbers) lol. Just pushed a fix.
1.03 was purchased, of 70, equalling $72.1. $72USDT was taken from my spot wallet. Any other ideas why?
I think I figured out what happened, here's an explanation. A TLDR how I'll fix it is at the bottom.
Currently this is what happens:
I checked CyberPunkMetalHead's repo and he's placing orders in the same way I am (he also needs to make the changes below).
So basically I think if the price changes rapidly enough (as it appears to have in this case), the actual market price multiplied by the size calculated from (2.) could potentially be more than the price grabbed from Binance's API from (1), which is how you ended up with this size discrepancy...
Now, Binance's website allows users to make a market order based on a total dollar amount rather than quantity (with this message Place a market order based on the amount of assets you want to spend. The final executed quantity and price will depend on the actual transaction result.
).
(also TLDR) After looking through the API documentation again I see I need to change how I place market orders so they're placed based on total dollar amount like how Binance's site offers. Going forward, the total size purchased will be based solely on the dollar amount specified in your settings rather than calculated in the program.
The needed changes are in the quantity-fix
branch - I'm going to wait until the next coin release to ensure this actually works before merging in the main branch but feel free to pull this branch now if you don't mind taking the risk that it'll error out.
That makes sense! Legend. I've checked out that branch to test it too. So in looking at:
[ERROR] 2021-10-21 12:00:00,278: [BINANCE] Preparing to buy LAZIOUSDT. [ERROR] 2021-10-21 12:00:01,106: [BINANCE] Placing [LIVE] Order...
There was almost a second between the preparation, fetching the price and working out the quantity, to actually placing the order. Does that mean you've removed the need to check the current price, as it's an additional unnecessary step?
Furthermore- do you happen to know where the binance API server(s) reside? I'm running it on a server in Canada, but am thinking my Australia based server might be better?
Yeah, so the good news is with this change none of the dollar amount -> quantity calculations are needed anymore (for Binance, not FTX) so the time between capture and purchase should be a bit shorter. I should've noticed this correct setting to begin with, but the setting name is kinda odd and didn't grab my attention.... oh well.
Are you running the program locally through an AU computer connected to a Canadian VPN, or it's literally on a VPS or server hosted in Canada? If the former, I'd try to find a VPN in a closer country that's "compatible" with Binance. If the latter, according to this Binance may be hosted in Tokyo... moving your VPS to that region would probably be faster.
Tokyo EC2 instance finishes it's loop in 50-70ms, much better than AU and CA.
Now, new coins are released within the first few seconds of a minute. It would be ideal if the bot could spam the calls in the first few seconds of each minute, instead of spreading them over the minute. Thoughts?
I don't know how to do a pull request, but I made the following changes. It now spams the API for a set number of seconds, and with a reduced frequency in between. It also schedules the next run to be on the minute exactly (give or take a few milliseconds. Not sure if this is helpful to you.
config.yml
FREQUENCY_SECONDS: 5
SPAMMING_PERIOD_SECONDS: 5
FREQUENCY_SPAMMING_SECONDS: 0
main.py:37
from datetime import datetime
milliseconds = int(datetime.utcnow().strftime("%f"))/1000000
second = int(datetime.utcnow().strftime("%S"))
second = second+milliseconds
secondsremaining = 60-second
if secondsremaining < Config.FREQUENCY_SECONDS:
#change wait time to start on the minute exactly
Config.NOTIFICATION_SERVICE.debug("Not long left - Sleeping for [{}]".format(secondsremaining))
await asyncio.sleep(secondsremaining)
continue
if second < Config.SPAMMING_PERIOD_SECONDS:
Config.NOTIFICATION_SERVICE.debug(
"SPAMMING - Only sleeping for [{}] seconds".format(Config.FREQUENCY_SPAMMING_SECONDS)
)
await asyncio.sleep(Config.FREQUENCY_SPAMMING_SECONDS)
continue
if second >= Config.SPAMMING_PERIOD_SECONDS:
Config.NOTIFICATION_SERVICE.debug(
"Sleeping for [{}] seconds".format(Config.FREQUENCY_SECONDS)
)
await asyncio.sleep(Config.FREQUENCY_SECONDS)
Neat idea - I like it. I implemented it slightly differently but it's in the quantity-fix
branch - let me know what you think!
I've got both running side by side- the quantity-fix branch is taking between 250-300ms to complete a loop. The main branch is taking 50-70ms to complete a loop. Something is amiss.
I added a few lines to get the average times for each branch... I ran each branch for 5 minutes on my computer and had these average times per loop:
Main Branch: Avg of 0.38325 seconds per loop, 322 total iterations quantity-fix Branch: Avg of 0.5232 seconds per loop, 310 total iterations
It's indeed a little slower, about 36%, on my end as well, but certainly not the 358-400% you're seeing. The change that I believe is causing this slowdown is that I now parse the Binance API return results into objects using Pydantic (see broker.py and models.py) - this greatly reduces bugs due to object and parameter typing issues (the very issue that caused the str error in this ticket, in fact) so I'm hesitant to go back on this though.
As for your system, I would try testing separately instead of comparing when they're both running together at the same time. Depending on your VPS tier something could be getting throttled, I can't really say. If you test separately and still have these issues def send your logs and anything that may be helpful in finding the problem my way.
If you want to get averages you can add these lines to main.py and calculate/print the average at the end (I don't want to push these temporary changes):
current_second = datetime.now().second

time_taken = time.time() - t

Config.NOTIFICATION_SERVICE.debug(
"Loop finished in [{}] seconds".format(time_taken)
)

Config.total_time += time_taken

Config.total_iter += 1
When I say side-by-side, I meant on two different identical EC2 instances. To ensure it's equal testing, I ran them on the same instance, consecutively. As you can see below, dramatic difference on my setup. I've tested it on my own dedicated servers in CA and AU and found similar results.
main branch:
[DEBUG] 2021-10-26 07:46:50,041: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56184.
[DEBUG] 2021-10-26 07:46:50,082: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:50,108: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56184.
[DEBUG] 2021-10-26 07:46:50,158: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:50,159: Loop finished in [0.07739067077636719] seconds.
[DEBUG] 2021-10-26 07:46:50,360: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:50,388: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56182.
[DEBUG] 2021-10-26 07:46:50,420: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:50,421: Loop finished in [0.0611720085144043] seconds.
[DEBUG] 2021-10-26 07:46:50,622: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:50,648: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56179.
[DEBUG] 2021-10-26 07:46:50,681: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:50,682: Loop finished in [0.060219764709472656] seconds.
[DEBUG] 2021-10-26 07:46:50,882: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:50,909: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56189.
[DEBUG] 2021-10-26 07:46:50,942: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:50,942: Loop finished in [0.059925079345703125] seconds.
[DEBUG] 2021-10-26 07:46:51,143: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:51,169: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56186.
[DEBUG] 2021-10-26 07:46:51,215: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:51,216: Loop finished in [0.07336187362670898] seconds.
[DEBUG] 2021-10-26 07:46:51,417: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:51,444: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56176.
[DEBUG] 2021-10-26 07:46:51,477: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:51,478: Loop finished in [0.06127429008483887] seconds.
[DEBUG] 2021-10-26 07:46:51,679: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:51,704: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56179.
[DEBUG] 2021-10-26 07:46:51,737: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:51,737: Loop finished in [0.05878138542175293] seconds.
[DEBUG] 2021-10-26 07:46:51,938: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:51,963: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:51,997: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:51,998: Loop finished in [0.059548139572143555] seconds.
[DEBUG] 2021-10-26 07:46:52,198: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:52,222: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:52,274: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:52,275: Loop finished in [0.07694625854492188] seconds.
[DEBUG] 2021-10-26 07:46:52,476: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:52,499: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56179.
[DEBUG] 2021-10-26 07:46:52,532: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:52,532: Loop finished in [0.056650400161743164] seconds.
[DEBUG] 2021-10-26 07:46:52,733: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:52,762: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:52,795: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:52,796: Loop finished in [0.06314373016357422] seconds.
[DEBUG] 2021-10-26 07:46:52,997: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:53,023: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:53,055: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:53,056: Loop finished in [0.05953073501586914] seconds.
[DEBUG] 2021-10-26 07:46:53,257: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:53,281: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:53,314: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:53,315: Loop finished in [0.05832314491271973] seconds.
[DEBUG] 2021-10-26 07:46:53,516: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:53,539: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:53,587: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:53,588: Loop finished in [0.07183551788330078] seconds.
[DEBUG] 2021-10-26 07:46:53,788: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:53,814: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:53,847: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:53,848: Loop finished in [0.05967569351196289] seconds.
[DEBUG] 2021-10-26 07:46:54,049: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:54,074: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:54,109: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:54,109: Loop finished in [0.06101727485656738] seconds.
[DEBUG] 2021-10-26 07:46:54,310: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:54,335: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:54,368: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:54,369: Loop finished in [0.05912041664123535] seconds.
[DEBUG] 2021-10-26 07:46:54,570: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:54,597: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:54,648: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:54,648: Loop finished in [0.07850217819213867] seconds.
[DEBUG] 2021-10-26 07:46:54,849: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:54,873: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:54,907: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:54,908: Loop finished in [0.058767080307006836] seconds.
[DEBUG] 2021-10-26 07:46:55,108: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:55,132: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:55,166: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:55,167: Loop finished in [0.05872631072998047] seconds.
[DEBUG] 2021-10-26 07:46:55,368: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:55,392: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:55,427: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:55,428: Loop finished in [0.060320138931274414] seconds.
[DEBUG] 2021-10-26 07:46:55,629: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:46:55,655: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56178.
[DEBUG] 2021-10-26 07:46:55,688: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:46:55,689: Loop finished in [0.06015777587890625] seconds.
quantity-fix branch:
[DEBUG] 2021-10-26 07:49:23,981: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:24,005: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:24,246: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:24,247: Loop finished in [0.26651763916015625] seconds.
[DEBUG] 2021-10-26 07:49:24,448: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:24,472: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:24,739: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:24,739: Loop finished in [0.29181599617004395] seconds.
[DEBUG] 2021-10-26 07:49:24,940: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:24,967: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:25,207: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:25,207: Loop finished in [0.26753854751586914] seconds.
[DEBUG] 2021-10-26 07:49:25,408: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:25,432: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:25,690: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:25,690: Loop finished in [0.282102108001709] seconds.
[DEBUG] 2021-10-26 07:49:25,891: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:25,918: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:26,158: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:26,158: Loop finished in [0.26764845848083496] seconds.
[DEBUG] 2021-10-26 07:49:26,359: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:26,383: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:26,642: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:26,642: Loop finished in [0.28319382667541504] seconds.
[DEBUG] 2021-10-26 07:49:26,843: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:26,868: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:27,106: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:27,107: Loop finished in [0.2639775276184082] seconds.
[DEBUG] 2021-10-26 07:49:27,308: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:27,332: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:27,598: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:27,599: Loop finished in [0.29138851165771484] seconds.
[DEBUG] 2021-10-26 07:49:27,800: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:27,843: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:28,082: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:28,083: Loop finished in [0.28351879119873047] seconds.
[DEBUG] 2021-10-26 07:49:28,284: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:28,314: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:28,571: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:28,572: Loop finished in [0.28829288482666016] seconds.
[DEBUG] 2021-10-26 07:49:28,773: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:28,797: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:29,033: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:29,034: Loop finished in [0.2611711025238037] seconds.
[DEBUG] 2021-10-26 07:49:29,235: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:29,260: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:29,519: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:29,520: Loop finished in [0.2853677272796631] seconds.
[DEBUG] 2021-10-26 07:49:29,720: [BINANCE] Getting all tickers...
[DEBUG] 2021-10-26 07:49:29,750: https://api.binance.com:443 "GET /api/v3/exchangeInfo HTTP/1.1" 200 56195.
[DEBUG] 2021-10-26 07:49:29,992: [BINANCE] No new tickers found...
[DEBUG] 2021-10-26 07:49:29,993: Loop finished in [0.27227115631103516] seconds.
I just pushed a speed-compare
branch, please compare main
to this one (instead of quantity-fix
). This branch is identical to quantity-fix
, except I removed the Pydantic changes. No other changes in the git history grabbed my attention as being problematic... if this version isn't faster I'm not entire sure what the issue is.
The speed-compare branch getting 80-98ms (57-78ms with just Binance) The quantity-fix branch getting 260-330ms (250-280ms with just Binance) The main branch getting 80-98ms (57-78ms with just Binance)
Interesting! That's quite a significant difference to use the Pydantic changes.
Removed Pydantic features since it slowed things down a bit.
Something weird happened with LAZIO. I'm set to purchase 40 USDT. It instead grabbed 72.1USDT (price at 70USDT, flat). Candle stick looks weird too- L of 1, flat, and H of 100, flat. So not only did it purchase higher than my set limit, it also looks odd with the initial price. It also errored out.