search

alpacahq / Alpaca-API

The Alpaca API is a developer interface for trading operations and market data reception through the Alpaca platform.
https://alpaca.markets/
143 stars 13 forks source link

internal server error occurred #104

Closed itajaja closed 3 years ago

itajaja commented 4 years ago 
{"code":50010000,"message":"internal server error occurred"}

while the status dashboard keeps happily showing all green. This has been happening since markets opened, at least on paper, across all the order APIs. Sometimes, instead of a 500, the orders get submitted, but never get filled, or they get cancelled, but remained cancel_pending. I even created a new paper account, same problems showed up

mingchang commented 4 years ago

Yup, I was getting the same thing yesterday morning as well (2020-02-18 before noon EST). Happened 4-5 times, but seems to not have happened in the afternoon. Definitely a little bit worrying what exactly caused this to happen.

itajaja commented 4 years ago

downtimes can happen, what I find really disconcerting is the lack of visibility. The platform was unable to process orders for hours and the status dashboard had no downtime recorded. For what's worth, the dashboard has no downtime recorded since the beginning of time. Congrats to the alpaca team for 100% uptime, I guess 👏

pratttrader commented 4 years ago

Is it just me, or does this system suck?

itajaja commented 4 years ago

I love the idea of alpaca, but the amount of down time (and the lack of transparency about it) is embarrassing. The fact that nobody from alpaca cared to chime in in this conversation is also telling.

pratttrader commented 4 years ago

Yes, nobody from the company chimes in about anything...good or bad. Are you only on Paper or are you Live? I'm currently trading with IAB, but still pay commissions because I run through their API and I'm kinda pissed about it. So I'm looking for alternatives. I haven't moved to Alpaca yet...I at this point, seems doubtful unless something changes with the stability of their backend. Their fills seem fair, but the stuff doesn't work about 30% of the time.

lvca commented 4 years ago

Hi guys, working with Alpaca for 1y now and trading every day since then. I have to say no broker is perfect, especially with API. If you try them you see that actually Alpaca is pretty stable. I found a lot of problems with the Paper Account, but on Live Trading everything is much better. I suggest you try with small amounts and see how it goes. If you find something better than Alpaca (zero commissions) and with a nice API, please let us know. AFAIK Alpaca is the best so far. I hope to help with my comment. (No, I don't work for Alpaca)

pratttrader commented 4 years ago

I've traded with IAB for 8 years, tens of thousands of trades and hundreds of thousands in commissions, and Alpaca does not compare...on stability & performance.....not even close..at least paper. I've had more issue using the "paper" environment in one month than I've had at IAB in 8 years. Here's my issue: the paper environment MUST be a duplicate of LIVE...in performance. This is what we test on. Period. No short cuts. They can't be using paper to test their own systems themselves(which is what I feel is happening)...they must have a third, private platform where they test their stuff..BEFORE it goes to paper/live. Paper must be as important as live. They need to try harder...they need to add/upgrade their network and back end systems..if only for paper. Traders know; they know what's right or wrong. This isn't rocket science. I will move some money over to expand my live trading as a more comprehensive test...but I will know within 2-3 days. Thanks for the input

d-e-s-o commented 4 years ago

Yes, paper trading is horrible with Alpaca. I have a test suite checking my functionality and part of that acts as integration tests against the Alpaca paper trading API. Every month or so I have a test failure because behavior changed randomly upstream and have to adjust my code (I am not testing anything esoteric; it's very basic stuff). Sometimes there are sporadic failures as well. From the Alpaca side nobody really seems to care or have a vested interest in really addressing issues and getting to the root of them. You mostly get one comment on an open issue and it's either left open without further interaction ever or closed saying "was probably this and that for which we rolled out a change some time back". In general the GitHub issue list is borderline unmaintained (there are at least two issues that have been addressed by now and nobody closes them). Of course, that's minor, but it fits the picture.

I vote for Alpaca open sourcing the paper trading logic. That would give everybody a chance to improve it (to the degree I can tell development is very slow) and allow more visibility into what is happening in general. I don't believe that the paper trading in its current form is a differentiating factor or any kind of secret sauce. So why not make it public? With more openness I would consider opening my stuff as well.

On the topic of testing: Honestly, I never ever got the impression that they tested their stuff properly, which is a huge concern for me. But at the same time, there is no alternative from my point of view. That is to say, the way I see it Alpaca lucks out because they fill a niche and are the only one doing so (in a way that is still somewhat acceptable for me). The moment somebody else opens a broker that fits the bill for me I will strongly consider switching.

pratttrader commented 4 years ago

So Daniel, thanks for the in depth response, I do appreciate it. I do have a couple follow up questions:

1) are you using the live platform, and if so, does it have the same problems as paper? I'm not going "all-in" with Alpaca unit I know that the live version is much more stable.

2) you say Alpaca fills a niche, but I'm curious what that niche is, you think? I know they are not a "full service" broker, but what niche do they fill that couldn't be filled by IB, or TD Ameritrade's API?

d-e-s-o commented 4 years ago
  1. are you using the live platform, and if so, does it have the same problems as paper? I'm not going "all-in" with Alpaca unit I know that the live version is much more stable.

I have used the live trading facility, but to a much lesser extent. Basically I've submitted "a few" orders manually through a CLI. And everything worked fine. That (live trading working better) was also my impression reading through the issue list, which gives me some hope.

I admittedly don't understand enough of how the backend works in both cases, but my assumption is that the paper trading simulator is more complex to build than "just" accepting requests and forwarding them (simplified) as they would for the live environment.

  1. you say Alpaca fills a niche, but I'm curious what that niche is, you think? I know they are not a "full service" broker, but what niche do they fill that couldn't be filled by IB, or TD Ameritrade's API?

I left that very vague because it's presumably specific to myself. What I can say is that I looked at a few more APIs before going with Alpaca and again later after being dissatisfied. And nothing else fit the bill. Important for me: 1) No commissions. 2) No OAuth. Everything I use is written from scratch and I don't want to spend my time and effort on such a complicated beast (I certainly heard/read a lot of horror stories about it, enough to make me vary). The language I use doesn't (didn't?) have much on the OAuth front that I could reuse and there was little point in having it. So if it's optional, fine, I may add support later, but making it a prerequisite to doing anything (live or paper) is a no-go. That requirement alone actually rules out a lot of alternatives from what I recall. So if I ever scratched that I believe many more alternatives pop up and the "niche" suddenly no longer is. 3) Free of charge. I am mostly still playing around and working at my own pace based on whatever time allows (and it is an open question how far I take my efforts), I don't want a monthly subscription fee or anything of that sort. 4) Related to 3): (free) market data availability. Alpaca's own market data is bad from what I understand (and they admit that), but through the partnership with Polygon there is at least something (now, based on my experience Polygon is worse in many points than Alpaca [to the degree they are comparable]; development is even more closed, the API design is much worse in my opinion, their API is flakier [though that may or may not apply to the market data stream, I can't really tell with confidence at this point], but I guess it's still acceptable for me based on what I am doing right now. And in general their support is responsive and acknowledges problems and seems to actually react on them [which on the flipside I am not sure gives me confidence in whether more people are actually using it...I just can't believe that I am the only one encountering all the issues]; they even appreciate feedback) 5) An HTTP based API, stable and "reasonable". The Alpaca API is actually pretty nice from what I can tell. I know it has its flaws and is not as well thought out as it could be. And more and more stuff is getting stitched on top. But it's still very concise and does everything I want it to.

This list does not seem like something that is hard to satisfy (and I may have forgotten one or two requirements, my notes aren't very comprehensive here), but it's still strict enough to leave Alpaca as the only entity that I am aware of checking all boxes.

pratttrader commented 4 years ago

Ok thanks...all good points and answers

I agree with the OAuth assessment...I haven't been willing to jump through that hoop, mainly because I haven't had to. I currently us IB..and have for years...they use two factor authentication by default so it isn't necessary. I would implement Oauth if I must, because I had an account hacked by Russians about 10 years ago and take security VERY VERY serious. Luckily my firm didn't make me pay the price, but I can't say that for any other firm, including Alpaca.

TD Ameritrade has zero commissions and an API almost identical to Alpaca,...it's almost as if Alpaca is just copying what they do...but TDA does require the use of OAuth. And from what I can tell(I don't have an account), there are no fees.

All my stuff is self programmed in C#, The C# SDK from Alpaca is pretty good...the guy in charge is very good and responsive...but it doesn't do enough error handling for me so I'm working on building my own. I just need to enter orders, manage orders and get account info...I don't need data...so it shouldn't be super hard.

The last point for me, relating to your last comment...if their system fails 30% of the time, which has been my experience(stuck order, orders just disapper etc)...even if they check all the boxes, none of it matters. It might as well be 100% failure. I trade all day, every day..and I can't have even 5% failure. So I like the idea: good executions, zero commissions, bare bones(although options would be nice for hedging), but if my software breaks at random points because orders disappear, or other errors, it's all worthless. Thanks for your time.

d-e-s-o commented 4 years ago

The last point for me, relating to your last comment...if their system fails 30% of the time, which has been my experience(stuck order, orders just disapper etc)...even if they check all the boxes, none of it matters. It might as well be 100% failure. I trade all day, every day..and I can't have even 5% failure. So I like the idea: good executions, zero commissions, bare bones(although options would be nice for hedging), but if my software breaks at random points because orders disappear, or other errors, it's all worthless. Thanks for your time.

Yes, ack. I am still clinging on to the hope that live trading "just works" (and so far I have not been proven wrong) and I intend to test the waters with more low amount trades. But I am having a very hard time accepting any flakiness on the backend side of things (i.e., out of my control) as well.

That being said, there are many failures to be anticipated regardless, and it's probably wise to be able to handle them (in one form or another). In fact, stuck orders and orders disappearing (though I have no seen that; not in live trading) are failure scenarios that I argue client software needs to be able to deal with (May there be losses because of that? Yes, risk exists, no argument. Should Alpaca be held accountable for it? Absolutely. Will they stand up for their wrongs? Not likely.) Even in a well tested backend there is always a possibility of some race being triggered that one time resulting in some weird artifact and client software should not be utterly confused by that.

So in a way, I write the software as well as possible and it should be able to deal with many classes of errors (again, that's not meant to be an excuse for sloppiness on Alpaca's side, but given the constraints I have imposed it's what I am rolling with anyway). Meanwhile, as Alpaca improves (and I have not yet completely lost hope on that) we will see less of that.

That's the justification I've made up in my mind anyway. Haha.

umitanuki commented 4 years ago

Hi all,

Thanks for all feedbacks and opinions. It's great to know that our service is being expected with a lot and this is a great problem for us to solve as well to grow even faster.

We understand the importance of transparency in addition to the service stability. We have internally discussed in the team over the weekend intensively what we can do despite of many things going on as a fast moving startup. Here is what we say now.

And to answer some of the questions in this thread, the issue around the paper trading is not about the different code (in fact, the business logic and application code are almost identical between paper and live except the siumlator part), but rather the trading volume and infrastructure setting differences are causing the issue. Thanks to the ease of use of the service, trading volume are 10x or more than the live trading with the API activities even bigger, and for that reson we have a slight different setup for backend messaging queues which will eventually go live for the live trading side as the live volume grows. The recent incidents were caused by this setup for the paper but live were less impacted due to this difference. The issue started only last month, after it was introduced many months ago, and that was due to the higher activities on the paper side. We are addressing the issue permanently right now by reviewing and reconfiguring the backend H/W resources as well as queue configurations.

Live trading has been a lot more stable and we are very very careful about introducing something new for the live side. It's been also helpful to do beta testing on paper side before introducing something new in the live trading side, but nonetheless we take the paper trading as seriously as the live trading.

Thanks again for all useful feedback and it's very helpful for us to solve the problems. We would never exist without each of you and we work hard to keep as many of you happy as possible.

pratttrader commented 4 years ago

Ok, thank you for the update and the insight. It's good to know that you aware of the issues with paper and have a grasp on it. I can see how the paper side could have more traffic vs live(users don't need to put money on the line)...thus putting stress on your backend. And this is what I assumed.

My only additional comments are this:
1) have you considered making the paper environment available ONLY to customers who have deposited money? Or maybe even have on "internal" paper environment,(as in a third platform), for actual customers....so that the general public who sign up with only an email are not co-mingled with actual customers who are trying to get some work done.

2) as indicated in #1, it's really really important to traders to have a very robust test platform..as robust as live...literally an identical copy in all aspects. If volume is 10x that of live, then it has to get more resources assigned to it, or as I said, spun off onto an internal version to cut down on the volume.

3) to summarize, to be a truly legit trading platform, give your deposited customers a robust and identical environment to test and build against, and give the "email" customers, who are just curious about who you are, a dumbed down version so that they can get an idea of what you're offering. This will all become even more important as more and more people come on board.

Thanks for your input.

d-e-s-o commented 4 years ago

As as it stands your entire status page is green including paper trading, which is what I am looking at here. And yet NOTHING is working.

GET uri: https://paper-api.alpaca.markets/v2/account
Unexpected HTTP status 504 Gateway Timeout:
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.16.1</center>
</body>
</html>

That has been the case for the last five minutes or so. Meanwhile, live trading seems to be reachable, so it does not look like a problem on my side.

d-e-s-o commented 4 years ago

I also tried subscribing to the status updates. Perhaps I don't get the "subscribe to updates" button. To me, it implies being able to subscribe to updates. Yet, I can't see that option on either that page or the support site being linked.

May I ask why you are actively misleading like that?

blubb

d-e-s-o commented 4 years ago

In case you haven't noticed because your status reporting is broken: the same issue exists today. 

Unexpected HTTP status 504 Gateway Timeout: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.16.1</center>
</body>
</html>

Status board is unchanged green.

zoakes commented 4 years ago

I’ve been having this issue a lot lately — it’s definitely concerning when trying to validate a strategy prior to going live. I’m a big fan of the API, but yeah you guys gotta scale up your server it sounds like :)

zoakes commented 4 years ago

Also — I think I have a temporary fix (hack) — I’m added a try except block that reconnects to self.api = api.REST .... and resets the connection / variable within my trailing stop loop in this case. Not exactly revolutionary, but it seems to have resolved this by not depending on a stable connection.

MrRedbloX commented 4 years ago

@zoakes I mean in my opinion this is not a "temporary fix" but a mandatory one. Especially when you are dealing with high frequency transactions, there are so many factors that could come into consideration. You have to handle those issues.

Anyway, I just wanted to report to this date another internal server error while trying to retrieve barsets. I am using the Paper Trading REST API.

zoakes commented 4 years ago

I agree it would ideally be resolved — but I mean this is a testing network — it’s unfortunate but it’s not going to cost any money. What I’ve done is simply trade micro positions if I’m testing something that requires a connection beyond the Paper server capabilities. My reconnection block has worked flawlessly through a handful of these disconnects, so if it resolves the issue long enough to validate something to get to Live, who cares how ?

Get Outlook for iOShttps://aka.ms/o0ukef

From: MrRedbloX notifications@github.com Sent: Saturday, June 27, 2020 1:48:30 PM To: alpacahq/Alpaca-API Alpaca-API@noreply.github.com Cc: Zach Mazz zach_oakes@outlook.com; Mention mention@noreply.github.com Subject: Re: [alpacahq/Alpaca-API] internal server error occurred (#104)

@zoakeshttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fzoakes&data=02%7C01%7C%7C9a655175fa9c43c8e01d08d81acaaffc%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637288805117178689&sdata=p67g2Bx35msp9UpvlPIliW3k3ChBLnVCPc8X08XzPZw%3D&reserved=0 I mean in my opinion this is not a "temporary fix" but a mandatory one. Especially when you are dealing with high frequency transactions, there are some many factors that could come into consideration. You have to handle those issues.

Anyway, I just wanted to report to this date another internal server error while trying to retrieve barsets. I am using the Paper Trading REST API.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Falpacahq%2FAlpaca-API%2Fissues%2F104%23issuecomment-650601279&data=02%7C01%7C%7C9a655175fa9c43c8e01d08d81acaaffc%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637288805117188690&sdata=YoIt7NZzUydQROLZQGGOw65U3wVilIUpg1pgpbFGBQY%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALPUCIDLEEAJ7KGXNAFHTFDRYY5H5ANCNFSM4KXHDJFQ&data=02%7C01%7C%7C9a655175fa9c43c8e01d08d81acaaffc%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637288805117198678&sdata=bmDxAF8JTWiOEPIK1gsDozQlQY1O1aFiok%2BQtuze9n4%3D&reserved=0.