Requests to the stats endpoints are timing out/hanging

[kashav]

Requests to the stats endpoints seem to be timing out on specific servers. Tested on a Digital Ocean instance, request does resolve, but takes significantly longer (as expected, kind of at least), can see why this is a problem for slower connections.

Would love some input on how to fix this, current course of action is to investigate/research this a little bit more, got has a timeout option that might help with this.

Originally reported on Twitter by Justin Conklin.

[eyev]

Thanks for recording the issue, I originally discovered this last night when attempting to host an express app that uses nba.js on an ec2 instance. My application was loading all the data endpoints fine but, none of the stats endpoints resolved. I tried heroku as well and had the same results.

The original endpoints failing for me were scoreboard and allPlayers but I soon discovered that all of them were failing. So i picked one with a very small return object to eliminate any chance of it being the size of the returns.

2017-03-18T04:17:53.225127+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/test" host=murmuring-beyond-xxxxx.herokuapp.com request_id=38a4d20d-1515-4b95-aa19-fa10efe9bf85 fwd="24.165.xxx.xxx" dyno=web.1 connect=0ms service=30001ms status=503 bytes=0 protocol=https

here is the /test endpoint

app.get('/test', (req, res) => { 
    nba.stats.franchiseHistory({LeagueID: 00 })
        .then(response => res.json(response))
        .catch(err => console.log(err));
})

[colbywhite]

I believe this is with the API in general, right? Not just slow connections?

Doing the following command hangs forever:

curl -X GET \
  -H "Referer: http://stats.nba.com/scores/" \
  -H "User-Agent: curl/7.43.0" \
  "http://stats.nba.com/stats/scoreboardV2?GameDate=2017-03-18&LeagueID=00&DayOffset=7"

That may not be the exact equivalent of what the JS lib is doing under the covers. But close? If so, something's going on with the NBA's servers unfortunately. From what I can tell, this has been happening with the NBA's servers since Mar. 15th.

EDIT: My User-Agent is much diff than yours. That could be it.

[eyev]

I did further testing today and I was able to get my express app working on Azure just fine. The stats endpoints do take a bit longer than data, but I believe that is normal. Regardless, it's still completely failing on AWS and Heroku for me.

[kashav]

@colbywhite Good catch! I tested your script with the user-agent used in nba.js and the request also seems to be hanging. The odd thing is, it doesn't hang when used with nba.js itself. I'm convinced that the API is looking for one of the other request headers, but will need to do a little more testing for that.

Accessing the URL directly from the browsers results in a 403 (I'm fairly sure this wasn't always the case, but I'm not certain). The page that makes the request to /scoreboardV2 is still up and running (and still retrieving data from that endpoint).

@eyev I'm glad it's working on Azure!

Do you think you could test the following on AWS/Heroku as well? I doubt Express is the problem here, but just in case --

const nba = require('nba.js').default;

nba.stats.franchiseHistory({ LeagueID: "00" })
  .then(response => console.log(response))
  .catch(err => console.log(err));

If that doesn't work, can you try editing (in the project root) node_modules/nba.js/lib/api/stats/utils/fetch.js by adding the following (I don't think you'll be able to test this on Heroku):

opts = Object.assign({
  headers: {
    host: 'stats.nba.com',
    connection: 'keep-alive',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36',
    referer: 'http://stats.nba.com/'
  },
  json: false,
+ timeout: 100000,
}, opts || {});

[eyev]

@kshvmdn not getting a response or error on heroku. I don't have the time right now but I will try to fork add the timeout later... (i think i can do this on heroku if i npm install a git repo rather than the npm package)

[Dolevco]

Adding this to the header in node_modules/nba.js/lib/api/stats/utils/fetch.js file worked for me: opts = Object.assign({ headers: { host: 'stats.nba.com', "cache-control":"max-age=0", connection: 'keep-alive', "accept-encoding" : "Accepflate, sdch", 'accept-language':'he-IL,he;q=0.8,en-US;q=0.6,en;q=0.4', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' }, json: false }, opts || {});

[kashav]

@Dolevco were you running this locally or on some server? If locally, was it not working before? Which endpoint(s)?

[Dolevco]

Im running this locally. the teamLineups at nba stats endpoint didnt repond. It worked fine just month ago, but when I tried to run the same code last week it waited forever.

changing the header like I mentioned did the work for me.

[kashav]

@Dolevco just tested, looks like local requests are broken for me too!

Tried with your changes to the header fields and requests are resolving again, will push a change for this!

Edit: looks like its only the accept-language field that's required to make the requests work, but I think I'll add all 3 anyways.

[kashav]

Published in v0.4.0!

Looks like we're still having the problem with running this on servers though. Build is failing on Travis, but running completely fine locally. Will have to look into this a bit more.

There are similar problems going on at https://github.com/bttmly/nba/issues/41.

[andr3w321]

I'm not using the library, but I found using simple curl or python requests module that my requests were being blocked locally unless I included a user-agent string.

[gin-hell]

any update on running this on a digital ocean server? all promises are returned as { pending } even when they are resolved when i run on my personal machine. can i trick the api into seeing my digital ocean server as my machine (or something? i'm pretty new to node//linux//js in general...) i am running my app with systemd on ubuntu 16.04

[kashav]

@gin-hell Just got around to testing on my droplet, looks like requests are hanging on Digital Ocean now as well. ☹️

I'm not entirely sure if it's possible to mask the request, but the first step would be to figure out how the API detects when a request is coming from a VPS. I think it's just through the client's IP address, but I'm not certain. I'll look into this when I get the chance, maybe somebody in this thread has a better answer.

[TylerTaylor]

FWIW, I'm able to get some of the stats endpoints working, like leagueLeaders returns the expected results. But other endpoints, like playerBioStats or franchiseHistory are timing out.

This is in an express app on heroku.

[eyev]

Just an update on my situation, the stats endpoints were originally working fine on azure... However, after routing my DNS through Cloudflare they stopped working, so I gave Azure my DNS management and they continued to fail... Prior to that I was using namecheap to manage my DNS and it was working - although very slow (to the point I was opting to not use stats endpoints in production). This makes me feel like nba is blocking the requests or it's just so slow the cloud providers are assuming a timeout.

[jasonroman]

I am getting blocked requests on DigitalOcean as well as a DreamHost server - not even using nba.js, but simple cURL requests. This happens for both https:// and http:// and on most stats.nba.com/stats endpoints

[larryworm1127]

I can't seem to grab the data using the API. It keeps telling me this "requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://stats.nba.com/stats/teamgamelog?TeamID=1610612737&Season=2016-2017&SeasonType=Regular+Season" Is this normal?

[kashav]

@larryworm1127 the Season parameter should be of the form YYYY-YY, so 2016-17 in your case.

[kashav]

@eyev, @jasonroman, @TylerTaylor, thanks for the updates.

Unfortunately, it looks like there's not much we can do here to make this work... especially since cURL requests aren't resolving. I'll play around with this when I get the chance to see if I can find some workaround, but I'm not sure how likely that is. 😢

[jasonroman]

I just did a test on the 104 stats.nba.com/stats endpoints that I know of. Two of them work:

• stats.nba.com/stats/leagueleaders
• stats.nba.com/stats/assisttracker

The other 102 do not.

[skalero01]

Does this problem continue? Is there any alternative to use?

[Reagan666]

If you are using python nba_py package, you need to set proper headers. I added headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'} to nba_py/init.py line 40 get function, it's working now.

[stache100]

Has anyone found a fix to this yet? Endpoints: playercareerstats for one still doesn't request.

[skalero01]

I found another api that can be found on here: https://data.nba.com/ that works great and doesnt block any source. But doesnt have a documentation, more info here: https://www.quora.com/Is-there-an-NBA-API-for-free-that-has-live-stats

[SamCreamer]

Anyone find a solution to this? Or are there any cloud providers that do not block these requests? I'm using the nba-stats python package. Any workarounds? Spent quite a while developing a small app and would love to deploy it to the world.

[lam-dan]

Hi All, I also used a free api listed above here: https://www.quora.com/Is-there-an-NBA-API-for-free-that-has-live-stats that provided me with the list of teams and team roster information for the website I was building as part of my bootcamp project.

However, when I try to get the player statistics using the JSON url: http://data.nba.net/data/10s/prod/v1/2018/players/201939_profile.json for example, it appears it is being blocked as I am not able to pull any results.

I was wondering if anyone else found another way to get player stats and could share as I've spent a long time building this website only to find the crucial player data is unavailable.

[skapadia]

Using either Chrome or Postman, I am able to make requests to stats.nba.com. However I'm also experiencing request timeouts when using nba.js, curl, or making requests using a Node library like 'request', from my local machine (Comcast is the ISP).

In node_modules/nba.js/lib/api/stats/utils/fetch.js, since the User-Agent is already set to the same value that Chrome uses, I tried changing it to the value Postman sends: User-Agent: PostmanRuntime/7.4.0, and voila, it consistently works! I've tried it on the following endpoints: /stats/commonallplayers (nba.stats.allPlayers), and /stats/leaguedashteamstats (nba.stats.teamGeneralStats).

This also works with curl, and using the 'request' Node library (which I believe is what nba.js uses).

Note this is the only header I've needed to get requests to work successfully.