jonasehrlich / ek-scraper

Scraper for kleinanzeigen.de
MIT License
6 stars 2 forks source link

Ebay Kleinanzeigen website structure changed #4

Closed jonasehrlich closed 1 year ago

jonasehrlich commented 1 year ago

The website structure of Ebay-Kleinanzeigen seems to have changed as the selected elements cannot be found anymore. This needs an update.

» ek-scraper --verbose run ~/config.json
WARNING: No notifications configured in '/Users/jehrlich/config.json'
Getting soup for 'https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/c203l9497'
Find pages linked on 'https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/c203l9497'
Found 6 new pages on 'https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/c203l9497'
Getting soup for 'https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/seite:6/c203l9497'
Getting soup for 'https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/seite:4/c203l9497'
Getting soup for 'https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/seite:7/c203l9497'
Getting soup for 'https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/seite:3/c203l9497'
Getting soup for 'https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/seite:5/c203l9497'
Getting soup for 'https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/seite:2/c203l9497'
Find pages linked on 'https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/seite:7/c203l9497'
Find all ad items in 'https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/altona/c203l9497'
ERROR: Error!
Traceback (most recent call last):
  File "/Users/jehrlich/projects/github/ek-scraper/ek_scraper/cli.py", line 161, in async_main
    await ret
  File "/Users/jehrlich/projects/github/ek-scraper/ek_scraper/cli.py", line 116, in run
    results: list[Result] = await asyncio.gather(*tasks)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jehrlich/projects/github/ek-scraper/ek_scraper/scraper.py", line 207, in get_new_aditems
    async for aditem in achain(*[get_aditems_from_soup(soup, url) for url, soup in soup_map.items()]):
  File "/Users/jehrlich/projects/github/ek-scraper/ek_scraper/scraper.py", line 25, in achain
    async for item in iterable:
  File "/Users/jehrlich/projects/github/ek-scraper/ek_scraper/scraper.py", line 176, in get_aditems_from_soup
    price=aditem.select(".aditem-main--middle--price")[0].text.strip(),
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
usage: ek-scraper [-h] [--version] [--verbose] {run,create-config} ...
ek-scraper: error: list index out of range
jonasehrlich commented 1 year ago

The structure of the aditem class was changed by Ebay, fixed the parsing in 9afff4d and released version 0.0.5 to PyPi