search

tedchou12 / webull

Unofficial APIs for Webull.
MIT License
597 stars 181 forks source link

get_options has stopped working today, returns a json error #305

Closed dovega closed 2 years ago

dovega commented 2 years ago

Hello,

Looks like the endpoint for get_options has stopped working today. It now returns this json error when trying to unpack the response using json(): requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0

The response code itself is 200, so it seems like it's getting something back. Also, the other endpoints work just fine still(at least the ones I've tried). Has this one changed at all?

The specific endpoint that is causing trouble is this one: def options(self, stock): return f'{self.base_options_url}/quote/option/{stock}/list'

Can anyone else verify that the endpoint is not working as expected?

grm083 commented 2 years ago

I opened and closed issue 306 as I've also encountered this. I've noticed that the API call is returning a success code of 200 but the content is empty. I believe this may be an issue with the endpoint itself, they may have updated or deprecated it.

(Edit to include link to post: https://github.com/tedchou12/webull/issues/306#issue-1214876931)

ryanblenis commented 2 years ago

API endpoints have been updated in various places. They've also enabled cert pinning in their apps, making it harder to find the calls to update

grm083 commented 2 years ago

It does appear that they POST them through https://quotes-gw.webullfintech.com/api/quote/option/strategy/list instead, with a payload of:

I can probably play with this a little later and see if I can make any headway.

dovega commented 2 years ago

It does appear that they POST them through https://quotes-gw.webullfintech.com/api/quote/option/strategy/list instead, with a payload of:

  • Ticker Id (obtained through the get_ticker call)
  • Count (Number of options to return in the call)
  • Direction
  • expireCycle (this one may need some deciphering, looks like it requires a list of integers)
  • Type (0, making an assumption that this will be defaulted to 0)
  • quoteMultiplier (100, how many shares are obtained when the contract is redeemed)

I can probably play with this a little later and see if I can make any headway.

Been fiddling with this using the following params:

params = {'tickerId': self.get_ticker(stock),
                  'count': -1,  #the value being used by the web browser client
                  'direction': 'all',
                  'expireCycle': [3, 2, 4], #also the value being used by the web browser client
                  'type': 0,
                  'quoteMultiplier': 100}

Getting an error message though, 'Parameters type miss match'. Hmm.

jjqqkk commented 2 years ago

I have got some workaround.

  1. Query option list. (POST)

curl -XPOST https://quotes-gw.webullfintech.com/api/quote/option/strategy/list -H "content-type:application/json" -d '{"tickerId":913244796,"count":-1,"direction":"all","expireCycle":[3,2,4],"type":0,"quoteMultiplier":100}'

The result is a HUGE json in which we can find the complete option expire date list and each option's tickerID. We need the tickerID in the next HTTPS request.

{ "strikePrice": "27", "volume": "0", "latestPriceVol": "0", "expireDate": "2022-06-03", "tickerId": 1032363507, "belongTickerId": 913244796, "activeLevel": 27, "cycle": 2, "weekly": 1, "executionType": "A", "direction": "call", "derivativeStatus": 0, "currencyId": 247, "regionId": 6, "exchangeId": 189, "symbol": "SQQQ220603C00027000", "unSymbol": "SQQQ", "quoteMultiplier": 100, "quoteLotSize": 100 }

  1. Get the option quotes. (GET)

GET this endpoint with a list of option ticker IDs.

curl "https://quotes-gw.webullfintech.com/api/quote/option/quotes/queryBatch?derivativeIds=1032283511,1032363508,1032363507"

grm083 commented 2 years ago

Awesome stuff! I saw the batch query string as well but the derivative Id was eluding me. I'll give this a shot this morning as well! You should be able to control the size of the response by specifying count as well. If you're tracking a ticker like SPY where there are hundreds it could be alot to sift through, but I would imagine a count of 10 would account for 5 above and 5 below the current trading price.

jjqqkk commented 2 years ago

The first request https://quotes-gw.webullfintech.com/api/quote/option/strategy/list must be called with http/2, otherwise webull server returns 500 error.

The python code in this library is not able to make http/2 connection coz requests is not ready for http/2 yet. We may import httpx for that POST request.

Curl makes http/1.1 POST and changes to http/2 seemlessly (curl -v to verify). I need a very quick fixup and let my bot work today. So I am using subprocess to fork curl and get the JSON data through stdout.

dovega commented 2 years ago

Thanks for the r&d on the endpoint jjqqkk and grm083!

I'm trying to make the post request using httpx and http2. It is still returning an error(this time a 500 error, despite making the request in http2). Here's the code:

headers = self.build_req_headers()

params = {'tickerId': self.get_ticker(stock),
                  'count': -1,
                  'direction': 'all',
                  'expireCycle': [3, 2, 4],
                  'type': 0,
                  'quoteMultiplier': 100}

import httpx
client = httpx.Client(http2=True)

data = client.post('https://quotes-gw.webullfintech.com/api/quote/option/strategy/list', params=params, headers=headers).json()

A .http_version check reveals that the post request is indeed using HTTP/2. Also, for sanity, I've tried the same thing with a get request, but that just gives me the 400 error again.

Are you guys getting something different?

dovega commented 2 years ago

Just to update, cURL gets the data just fine. Still having no luck with httpx.

BobLiu20 commented 2 years ago

It is working well for me.

import requests
json_data = {
    'tickerId': 913244796,
    'count': 1,
    'direction': 'all',
    'expireCycle': [
        3,
        2,
        4,
    ],
    'type': 0,
    'quoteMultiplier': 100,
}

response = requests.post('https://quotes-gw.webullfintech.com/api/quote/option/strategy/list', json=json_data)
print(response.json())
dovega commented 2 years ago

That's working for me too. I think I was missing the "json" field and that's why it was failing. Thanks BobLiu20, jjqqkk, and grm083.

jjqqkk commented 2 years ago

Verified, thank every one working on this issue.

dovega commented 2 years ago

Closing this since we've found a workaround.

zenhorace commented 2 years ago

Thanks everyone for the work done here. I'm reopening this until the fix is integrated into this package and doesn't require a custom workaround by users of the package.

fredflynn commented 2 years ago

Anyone else having trouble with the work around? It worked for me yesterday and is now returning an empty response.

grm083 commented 2 years ago

Anyone else having trouble with the work around? It worked for me yesterday and is now returning an empty response.

Yep, same issue with POST request now "200" OK but no body content; was working yesterday.

I actually am still having success with this, but there are some catches that don't follow the paradigms of the other callouts:

Another note is that the data should be passed as "json=" not as "params=". I was caught up with that shortly. I've consistently gotten 200 success responses by failing to provide the expire cycle and by labeling my parameters as params rather than JSON.

grm083 commented 2 years ago

Oh jeeze sorry! An additional important note. It does seem that this query is all or nothing. If you try to specify a count parameter as anything other than -1 it seems to break. So depending on the ticker you could be getting a really big response.

There is a mismatch between what you'd see on the UI and what you see in the payload (UI selection + 10), but modifying based on selections made there doesn't seem to make much of a difference.

image

mrazum commented 2 years ago

https://chat.whatsapp.com/BdYPQv7yJwtEL7AiteceL0 guys join the group we share idea faster

Do you have similar group in Telegram?

dovega commented 2 years ago

I can confirm that this has stopped working for me as well. Here's my code. It was working well yesterday.

params = {'tickerId': self.get_ticker(self, 'AAPL'),
          'count': -1,
          'direction': 'all',
          'expireCycle': [3, 2, 4],
          'type': 0,
          'quoteMultiplier': 100}

import httpx
client = httpx.Client(http2=True)

response = client.post('https://quotes-gw.webullfintech.com/api/quote/option/strategy/list', json=params)

Gives a json decode error:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

EDIT: it seems to be intermittent. I can query it 10 times and it may work 1-2 times out of the 10. Sometimes it goes long stretches of not working. Then works a couple times. Very peculiar.

Jakezheng commented 2 years ago

telegram group guys join up and help each other https://t.me/+tXp2V59FAIUyYzkx

gbtota commented 2 years ago

ok solved, you need almost all of the headers in the request device-type,os,ph,hl,platform,reqid,osv all of there are required now

Jakezheng commented 2 years ago

ok solved, you need almost all of the headers in the request device-type,os,ph,hl,platform,reqid,osv all of there are required now

yeah man , i copy everything from the chrome devtool lol

jjqqkk commented 2 years ago

yeah, my firefox version, update webull.py 

self._headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0',
            'Accept': '*/*',
            'Accept-Encoding': 'gzip, deflate',
            'Accept-Language': 'en-US,en;q=0.5',
            'Content-Type': 'application/json',
            'platform': 'web',
            'hl': 'en',
            'os': 'web',
            'osv': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0',
            'app': 'global',
            'appid': 'webull-webapp',
            'ver': '3.39.18',
            'lzone': 'dc_core_r001',
            'ph': 'MacOS Firefox',
            'locale': 'eng',
            'did': self._get_did()
        }
dovega commented 2 years ago

yeah, my firefox version, update webull.py

self._headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0',
            'Accept': '*/*',
            'Accept-Encoding': 'gzip, deflate',
            'Accept-Language': 'en-US,en;q=0.5',
            'Content-Type': 'application/json',
            'platform': 'web',
            'hl': 'en',
            'os': 'web',
            'osv': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0',
            'app': 'global',
            'appid': 'webull-webapp',
            'ver': '3.39.18',
            'lzone': 'dc_core_r001',
            'ph': 'MacOS Firefox',
            'locale': 'eng',
            'did': self._get_did()
        }

Unfortunately, this is not working for me. Still using httpx with http2. Maybe I'll play around with those things and see if your header changes start to work.

jjqqkk commented 2 years ago

Try this one:

self._headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0',
            'Accept': '*/*',
            'Accept-Encoding': 'gzip, deflate',
            'Accept-Language': 'en-US,en;q=0.5',
            'Content-Type': 'application/json',
            'platform': 'web',
            'hl': 'en',
            'os': 'web',
            'osv': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0',
            'app': 'global',
            'appid': 'webull-webapp',
            'ver': '3.39.18',
            'lzone': 'dc_core_r001',
            'ph': 'MacOS Firefox',
            'locale': 'eng',
            'reqid': 'feb46a9524574a59b8f121c8eab62023',
            'device-type': 'Web',
            'did': self._get_did()
        }

You may change reqid to any 32-byte random hash value.

More testing needed in today's trading session, lets see.

dovega commented 2 years ago

Try this one:

self._headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0',
            'Accept': '*/*',
            'Accept-Encoding': 'gzip, deflate',
            'Accept-Language': 'en-US,en;q=0.5',
            'Content-Type': 'application/json',
            'platform': 'web',
            'hl': 'en',
            'os': 'web',
            'osv': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0',
            'app': 'global',
            'appid': 'webull-webapp',
            'ver': '3.39.18',
            'lzone': 'dc_core_r001',
            'ph': 'MacOS Firefox',
            'locale': 'eng',
            'reqid': 'feb46a9524574a59b8f121c8eab62023',
            'device-type': 'Web',
            'did': self._get_did()
        }

You may change reqid to any 32-byte random hash value.

More testing needed in today's trading session, lets see.

This returns an unauthorized user error:

'msg': 'unauthorized user', 'code': '417'

EDIT: Nevermind, I got this to work by placing it in the headers area of the webull class. Prior to that I was doing a hacky job just for testing. Placing it in the correct spot seems to apply the trade token. Currently this is working for me. Thanks.

Per your suggestion I'm using a random value for the reqid. Perhaps this will avoid unwanted eyeballs if we don't all use the same reqid for every request. Here's my code - there's probably a better way of doing it though.


import random
req_id = str(random.randint(10000000000000000000000000000000, 99999999999999999999999999999999))
self._headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0',
            'Accept': '*/*',
            'Accept-Encoding': 'gzip, deflate',
            'Accept-Language': 'en-US,en;q=0.5',
            'Content-Type': 'application/json',
            'platform': 'web',
            'hl': 'en',
            'os': 'web',
            'osv': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0',
            'app': 'global',
            'appid': 'webull-webapp',
            'ver': '3.39.18',
            'lzone': 'dc_core_r001',
            'ph': 'MacOS Firefox',
            'locale': 'eng',
            'reqid': req_id,
            'device-type': 'Web',
            'did': self._get_did()
            }
tedchou12 commented 2 years ago

Sorry about the ignorance. The random reqid does fix the issue. It seems like Webull is demanding the headers as well. I think the package has been updated, and pushed to pypi as well. Thank you for the investigation!

nirmalc83 commented 2 years ago

@tedchou12 Still doesnt seem to work, getting the same error after upgrading the package. Any thoughts?

File "C:\Users\nirma\anaconda3\lib\site-packages\webull\webull.py", line 683, in get_options_expiration_dates return requests.get(self._urls.options_exp_date(self.get_ticker(stock)), params=data, headers=headers, timeout=self.timeout).json()['expireDateList']

File "C:\Users\nirma\anaconda3\lib\site-packages\requests\models.py", line 910, in json return complexjson.loads(self.text, **kwargs)

File "C:\Users\nirma\anaconda3\lib\json__init__.py", line 346, in loads return _default_decoder.decode(s)

File "C:\Users\nirma\anaconda3\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "C:\Users\nirma\anaconda3\lib\json\decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting value

UPDATE: Nevermind, pls ignore this comment.. Firefox version was a build older. Fixed it after browser update. Thanks for the fix everyone and module update!

JonECG commented 2 years ago

I'm still getting this issue... sometimes...

I feel like its 50:50

Half the time, after the update, get_options works as expected, the other time I get an OK in the response, but it's empty. Causing the original decode exception when trying to parse it as JSON.

JonECG commented 2 years ago

@tedchou12

Could you try running

for lp in range(100):
    options = wb.get_options("MMM")
    print(len(options))
    time.sleep(1) # maybe rate limiting? idk

and see if you get through a good chunk? I can't get past 5 or 10 before getting an exception

JonECG commented 2 years ago

After further testing, it seems it's not related to reqid. Fixing it to a specific reqid can work for a bit, but then it will eventually fail.

JonECG commented 2 years ago

It looks like it also happens in get_options_expiration_dates

JonECG commented 2 years ago

On the bright side, it looks like get_options_bars works flawlessly. So, I've just been silently retrying the other two functions until I get the tickers I need as a workaround:

def get_options_s(stock=None, count=-1, includeWeekly=1, direction='all', expireDate=None, queryAll=0):
    result = None
    while result is None:
        try:
            result = wb.get_options(stock, count, includeWeekly, direction, expireDate, queryAll)
        except:
            print("get_options failed, retrying")
    return result

def get_options_expiration_dates_s(stock=None, count=-1):
    result = None
    while result is None:
        try:
            result = wb.get_options_expiration_dates(stock, count)
        except:
            print("get_options_expiration_dates failed, retrying")
    return result
tedchou12 commented 2 years ago

you are right, it is very intermittent. 🤔 There are two possibilities

grm083 commented 2 years ago

The updated endpoint doesn't use immediate refresh or streaming when accessing through the web app, it seems to update at set intervals (every 3 seconds for example)

When wrapping the callout in a try/catch loop, I've noticed it usually doesn't error out more than 2 times before succeeding. On the source side you may consider a while loop, not to exceed 10 retries, to combat it. Otherwise handling this client side isn't a huge effort.

grm083 commented 2 years ago

On the bright side, it looks like get_options_bars works flawlessly. So, I've just been silently retrying the other two functions until I get the tickers I need as a workaround:

I added an iterator, not to exceed count 10, just to ensure it doesn't get stuck if something changes again in the near future.

tedchou12 commented 2 years ago

I gave it a try:

200
Success
200
Success
200

Failed
200
Success
200
Success
200
Success
200
Success
200
Success
200
Success
200

Failed
200
Success
200
Success
200
Success
200

Failed
200
Success
200

Failed
200
Success
200
Success
200
Success
200
Success
200

Failed
200
Success
200
Success
200

Failed
200
Success
200
Success

The response code is always 200, but the result is empty compared to 60000+bytes of json string.

@grm083 That's one work around, but it makes the code a bit more messy, we can try that temporarily.

I am pretty sure this would affect Webull's own WebApp, so there might be other ways to make this work.

tedchou12 commented 2 years ago

Dear All.

Thanks for the discussion here.

After some time investigating, I realized that the webapp endpoint for option is updated. This is probably a good reason for intermittent success on the old endpoint. My guess is that they did not stop it completely, but some servers have it stopped working, so with some load balancing, if you are responded by servers that have deprecated the endpoint would result in the empty response or the json error.

It is likely that the old endpoint will be stopped completely, so I updated the endpoint. The response is not completely the same some some variables have been changed. I tried to respond it in the same format so the code might look a bit messy. I tried calling the new endpoint 30+ times and found it working 100%. The new endpoint is a bit fat though, so it's taking webull's server a bit longer for each request.

I am happy it's solved. Sorry for the delayed response. I am a bit busy with job interviews and I try to shy away from my investing recently since the market is doing so bad. Thanks to Yashwant who has been encouraging me on LinkedIn and on the buymecoffee app. If you like I do, you can post some kind words to me too. 😂🙏🙏🙏