kent-lee / deviantart-scraper

personal project for downloading artworks from DeviantArt
66 stars 14 forks source link

TypeError: 'NoneType' object is not subscriptable #11

Open blairg23 opened 3 years ago

blairg23 commented 3 years ago

When running the program normally, based on the documentation.

OS: Windows 10 Python 3.8 Using GitBash as my terminal

Running for the following users:

{
    "save_directory": "./art",
    "users": [
        "kuvshinov-ilya",
        "guweiz",
        "dominaart",
        "grooooovy",
        "ilikeyoursensitivity",
        "nababa",
        "ni-nig",
        "polkin",
        "raichiyo33",
        "rossdraws",
        "sharpwriter",
        "tsuaii",
        "viccolatte",
        "wlop",
        "trungbui42"
    ]
}

Error Code:

$ python main.py artwork

there are 15 users

download artworks for user kuvshinov-ilya

Traceback (most recent call last):
  File "main.py", line 54, in <module>
    main()
  File "main.py", line 40, in main
    download_users(api, config, args.option)
  File "main.py", line 11, in download_users
    result = api.save_users_artworks(config.users, config.save_dir)
  File "H:\Dropbox\Projects\Python\scrapers\deviantart-scraper\lib\deviantart.py", line 204, in save_users_artworks
    files = self.save_user_artworks(user, dir_path)
  File "H:\Dropbox\Projects\Python\scrapers\deviantart-scraper\lib\deviantart.py", line 180, in save_user_artworks
    files = pool.map(partial(self.save_artwork, dir_path), artworks)
  File "C:\Users\<username>\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\<username>\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
  File "C:\Users\<username>\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\<username>\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\pool.py", line 48, in mapstar
    return list(map(*args))
  File "H:\Dropbox\Projects\Python\scrapers\deviantart-scraper\lib\deviantart.py", line 154, in save_artwork
    download_url = self._download_url(artwork)
  File "H:\Dropbox\Projects\Python\scrapers\deviantart-scraper\lib\deviantart.py", line 110, in _download_url
    return re.search(r'href="(https://www.deviantart.com/download/.+?)"', html)[1]
TypeError: 'NoneType' object is not subscriptable
BachoSeven commented 3 years ago

I'm also seeing the type error on Linux(arch)

mitar commented 2 years ago

It seems download link is not available anymore if you just blindly fetch the HTML. So regex trying to find it does not work anymore. I think there is some protection in place.

mitar commented 2 years ago

I made this MR: https://github.com/Kent-Lee/deviantart-scraper/pull/12 but I think it is best to simply use https://github.com/mikf/gallery-dl, which seems to work.