search

wiscostret / fplscrapR

This package enables those interested in Fantasy Premier League to perform detailed data analysis of the game, using the FPL's JSON API. The fplscrapR functions help R users collect and parse data from the Official Fantasy Premier League website.
Creative Commons Zero v1.0 Universal
75 stars 16 forks source link

Authentication issues using get_league and get_league_entries #24

Closed woyzeck1978 closed 2 years ago

woyzeck1978 commented 2 years ago

First of all, thanks for sharing this great package. I've been using it with great pleasure for a few years.

Since the 2021/2022 season I have authentication issues when using the get_leagues functions to scrape my mini leagues. I did not have this issue previously.

Specifically, R provides the following error message after entering my login and password:

Error in open.connection(con, "rb") : HTTP error 405.

I reinstalled the fplscrapR package and I've minimized the code into a reproducible example:

library(fplscrapR)
FPL <- get_league_entries(45536, "classic", pages=2)

Unfortunately I cant find any info about the specific error message.

wiscostret commented 2 years ago

Thanks for logging this. The 405 error seems to indicate that it’s an issue with the curl interfacing embedded in the getleague functions, but I wasn’t able to reproduce your issue, and I haven’t run in to this before, so I am admittedly guessing a bit here.

First, what client are you running this from - RStudio? version?

Second, I'm assuming from your headline you run into the same error both with get_league() and get_league_entries() - correct?

Third, one way to troubleshoot is for you to break down the function steps, and use the 'verbose = TRUE' option for the handle to see if we can pinpoint the error source. Try running the following. This will give you more detailed information on the operations of the handle, which may help troubleshoot.

library(fplscrapR)

fplfetchhandle <- curl::new_handle(verbose=TRUE)

curl::handle_setform(fplfetchhandle,
  login=readline("Please enter your FPL login email: "),
  password=getPass::getPass(msg="Please enter your FPL password:"),
  redirect_uri="https://fantasy.premierleague.com/a/login",
  app="plfpl-web")

fplfetchmemory <- curl::curl_fetch_memory("https://users.premierleague.com/accounts/login/", handle = fplfetchhandle)

jsonlite::fromJSON(curl::curl("https://fantasy.premierleague.com/api/leagues-classic/45536/standings/", handle = fplfetchhandle))
lvtwoodpecker commented 2 years ago

having this same issue. The get_league_entries function only prompted a Password input but no field for me to put in the email. Only after I ran get_league_entries the second time did it prompt the email field in the console!

Running the curl call as above worked though.

woyzeck1978 commented 2 years ago

Thanks for logging this. The 405 error seems to indicate that it’s an issue with the curl interfacing embedded in the getleague functions, but I wasn’t able to reproduce your issue, and I haven’t run in to this before, so I am admittedly guessing a bit here.

First, what client are you running this from - RStudio? version?

Second, I'm assuming from your headline you run into the same error both with get_league() and get_league_entries() - correct?

Third, one way to troubleshoot is for you to break down the function steps, and use the 'verbose = TRUE' option for the handle to see if we can pinpoint the error source. Try running the following. This will give you more detailed information on the operations of the handle, which may help troubleshoot.

library(fplscrapR)

fplfetchhandle <- curl::new_handle(verbose=TRUE)

curl::handle_setform(fplfetchhandle,
  login=readline("Please enter your FPL login email: "),
  password=getPass::getPass(msg="Please enter your FPL password:"),
  redirect_uri="https://fantasy.premierleague.com/a/login",
  app="plfpl-web")

fplfetchmemory <- curl::curl_fetch_memory("https://users.premierleague.com/accounts/login/", handle = fplfetchhandle)

jsonlite::fromJSON(curl::curl("https://fantasy.premierleague.com/api/leagues-classic/45536/standings/", handle = fplfetchhandle))

thanks so much for your response and apologies for the late reply.

Re 1) I run the most recent R Studio version (2021.09.0 -"Ghost Orchid" Release (077589bc, 2021-09-20) for macOS Mozilla/5.0 (Macintosh; Intel Mac OS X 12_1_0) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.10 Chrome/69.0.3497.128 Safari/537.36)

Re 2) yes, I receive the same error message with both functions.

Re 3) I've no problems logging in when running the verbose option: $url [1] "https://fantasy.premierleague.com/a/login?state=success" $status_code [1] 200

py9mrg commented 2 years ago

Hello,

Just to add that I've come across the same issue (both Windows through work proxy and personal Linux without proxy). Both are latest RStudio preview version, latest R, everything updated. In both cases the longer form above worked fine. Strange?

Edit: BUT, if I replace the league id in the example (45536) with the league of interest, then I get the same error. Weirdly, I can get rid of the error by replacing -classic wit -h2h but then I get a different league to intended so it seems something to do with league type, but that could be a red herring.

Edit 2 (sorry!!!): ok so I think I know the problem. It seems like the league ids have changed and/or something about how they're accessed. Using the "old" league id I've always used doesn't work any more. If you log in to the website, go to the league of interest, and check the number (https://https://fantasy.premierleague.com/leagues/[HERE]/standings/c) then using that number works fine again. I suspect the issue is all of us using outdated league ids.

woyzeck1978 commented 2 years ago

Third, one way to troubleshoot is for you to break down the function steps, and use the 'verbose = TRUE' option for the handle to see if we can pinpoint the error source. Try running the following. This will give you more detailed information on the operations of the handle, which may help troubleshoot.

Re: point 3.

"Third, one way to troubleshoot is for you to break down the function steps, and use the 'verbose = TRUE' option for the handle to see if we can pinpoint the error source. Try running the following. This will give you more detailed information on the operations of the handle, which may help troubleshoot."

The troubleshoot info:

psydroid49 commented 2 years ago

Hello,

edit***

my problem is when I'm trying to fetch get_league and get_league_entries. Once I'm prompted to login, I enter my e-mail adress and password but it keeps saying that the authentication failed.

I've tried the following:

I've seen that I should troubleshoot with fplfetchhandle and verbose=T. Here's the output: `* Found bundle for host users.premierleague.com: 0x16d085f0 [can pipeline]

Then, running jsonlite::fromJSON(curl::curl("https://fantasy.premierleague.com/api/leagues-classic/314/standings/", handle = fplfetchhandle))

gets me the following output: `* Trying 151.101.114.133...

I've edited my previous post for it to be more readable. Was in a rush previously. Thanks for your help!

wiscostret commented 2 years ago

Hi folks, this took way too long to get back to, and I apologize. It's a doozy - it turns out the FPL API no longer requires authentication to get the league data. And I hadn't realised. I will be updating the package shortly, but essentially you can just cut out the curl authentication bits of the getleague*() commands, e.g. as follows:

get_league <- function(leagueid = NULL, leaguetype = "classic"){
  if(is.null(leagueid)) stop("You'll need to input a league ID, mate.")
  if(length(leagueid) != 1) stop("One league at a time, please.")
    {
    league <- jsonlite::fromJSON(paste("https://fantasy.premierleague.com/api/leagues-",leaguetype,"/",leagueid,"/standings/",sep=""))
    return(league)
  }
}

get_league(45536)

I will leave this open for a bit to see if anyone still has issued, otherwise I will consider it closed with the updated package version.