prouast / cryptocurrency-analysis

Analysis and visualisation of the cryptocurrency market
MIT License
288 stars 62 forks source link

Updated Scraper? #6

Open rgomezp opened 4 years ago

rgomezp commented 4 years ago

Howdy, The scraper linked is outdated it seems. I got this for the output:

Could not scrape token unus-sed-leo.
Traceback (most recent call last):
  File "scrape.py", line 57, in <module>
    if scrapeMarketCap(token['slug'], token['name'], 'token'):
  File "scrape.py", line 46, in scrapeMarketCap
    result = coinmarketcap.parseMarketCap(jsonDump, slug)
  File "/Users/*****/Documents/Github/Personal/coinmarketcap-scraper/coinmarketcap.py", line 93, in parseMarketCap
    rawData = json.loads(jsonDump)
  File "/Users//*****/anaconda2/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/Users//*****/anaconda2/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users//*****/anaconda2/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

------------------------------------------------------------
07/08/2020 12:00:30 AM INFO: >> Could not scrape unus-sed-leo. Skipping.
07/08/2020 12:00:30 AM INFO: > Starting scrape of token usd-coin...
07/08/2020 12:00:30 AM INFO: Sleeping for 30.2140883146 seconds before request.
^CTraceback (most recent call last):
  File "scrape.py", line 57, in <module>
    if scrapeMarketCap(token['slug'], token['name'], 'token'):
  File "scrape.py", line 45, in scrapeMarketCap
    jsonDump = coinmarketcap.requestMarketCap(slug)
  File "/Users//*****/Documents/Github/Personal/coinmarketcap-scraper/coinmarketcap.py", line 51, in requestMarketCap
    graphBaseUrl, slug))
  File "/Users//*****/Documents/Github/Personal/coinmarketcap-scraper/coinmarketcap.py", line 26, in _request
    time.sleep(timeToSleep)

I'm wondering if you have either an updated version or if you could add a table with the format that this project expects the data to be in.

This looks so awesome by the way. Excited to get this working.

prouast commented 4 years ago

The scraper breaks whenever coinmarketcap changes - I don't have time to fix this at the moment. Feel free to create a pull request at https://github.com/prouast/coinmarketcap-scraper though. Check https://github.com/prouast/coinmarketcap-scraper/blob/master/db.py for the database details. Thanks!