search

HashirZahir / FIFA-Player-Ratings

Tool to create database of FIFA football player ratings
MIT License
9 stars 8 forks source link

player_stats.jl is empty #4

Closed OCA99 closed 6 years ago

OCA99 commented 7 years ago

I ran the script as stated in the tutorial and it created the player_stats.jl file, but it was empty. I didn't get any kind of error during the process and I didn't modify the settings file at any point.

I entered this command: scrapy crawl fifa -o player_stats.jl -s JOBDIR=attempt/spider1 I'm using macOS Sierra.

EDIT: I'm using a Python3 version of this file, converted with Python's integrated 2to3 tool.

HashirZahir commented 7 years ago

Yes you are right. It seems that futhead.com itself has changed their internal html structure of the webpage (but visually it looks the same). As such, the xpaths used in the code such as url_list = sel.xpath('//tbody/tr/td[@class="player"]/a/@href') is no longer accurate as no 'player' class exists anymore and hence the scrapy tool is having issues getting the player URLs (resulting in it terminating immediately with no URLs to scrape). Similarly, stats = site.xpath('//div[@class="row player-center-container"]/div/a') is no longer accurate and hence the statistics can't be obtained either. Are you familiar with Xpaths? If so, maybe you could assist in studying the HTML structure of these pages and adding the correct Xpath to obtain the URLs and statistics. It should be a simple commit.

Currently, I do not have access to a development environment and if this issue is not solved in the next few days, I will commit the code myself once I have access to a development machine. Thanks.

OCA99 commented 7 years ago

I am not familiar with Xpaths. I've tried to take a look but I am not able to fix it. Thank you for the quick response.

HashirZahir commented 6 years ago

It should work now. I have updated the xpaths. There might be some NULL rows for the first 5 rows but otherwise works perfectly fine