search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.4k stars 931 forks source link

throws error with command line while running directly from source files #4317

Closed ghbook closed 1 year ago

ghbook commented 1 year ago

Hi

I need to do some changes in few extractor files locally, so i download latest zip source code.

When running via command line it gives an error, but works fine via python script

Here is what i have done to setup the project: (followed yt-dlp)

  1. First downloaded the latest release zip, extracted and renamed the folder to gallery-dl2 for clarity.
  2. Created a bat file gallery-dl.cmd inside the folder to run via cmd (referred yt-dlp.cmd in yt-dlp project) 
    D:\gallery-dl2\gallery-dl.cmd
@py -bb -Werror -Xdev "%~dp0gallery_dl\__main__.py" %*
  3. strangely __main__.py was disabled with sys.exit, so removed it https://github.com/mikf/gallery-dl/blob/248e8bc6996229fd8c3fb064afff14b2b722fa58/gallery_dl/__main__.py#L19-L20 it will become 
    if __name__ == "__main__":
gallery_dl.main()

Now Run via python file: Created a test file and when ran, it works fine.

import logging
from gallery_dl import config, output, job

url = 'https://redgifs.com/watch/foolishforkedabyssiniancat'

output.initialize_logging(logging.DEBUG)
output.configure_logging(logging.DEBUG)
output.setup_logging_handler("unsupportedfile", fmt="{message}")

code = job.DownloadJob(url).run()

print(code)
D:\gallery-dl2>python z_test.py
[redgifs][debug] Using RedgifsImageExtractor for 'https://redgifs.com/watch/foolishforkedabyssiniancat'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.redgifs.com:443
[urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v2/auth/temporary HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v2/gifs/foolishforkedabyssiniancat HTTP/1.1" 200 None
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): thumbs44.redgifs.com:443
[urllib3.connectionpool][debug] https://thumbs44.redgifs.com:443 "GET /FoolishForkedAbyssiniancat.mp4?expires=1689754800&signature=v2:c166125d259bfbbb9dffa031b275ee51634507957e6a738d94ed36441922fad9&for=138.199.50&hash=6163438793 HTTP/1.1" 200 4636670
* .\gallery-dl\redgifs\redgifs_foolishforkedabyssiniancat.mp4
0

    Run via cmd: Gives an error:

D:\gallery-dl2>gallery-dl -v --ignore-config https://redgifs.com/watch/foolishforkedabyssiniancat
[gallery-dl][debug] Version 1.25.8
[gallery-dl][debug] Python 3.11.4 - Windows-10-10.0.19045-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.0.3
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://redgifs.com/watch/foolishforkedabyssiniancat'
Traceback (most recent call last):
  File "D:\gallery-dl2\gallery_dl\__main__.py", line 20, in <module>
    gallery_dl.main()
  File "D:\gallery-dl2\gallery_dl\__init__.py", line 285, in main
    retval |= jobtype(url).run()
              ^^^^^^^^^^^^
  File "D:\gallery-dl2\gallery_dl\job.py", line 223, in __init__
    Job.__init__(self, url, parent)
  File "D:\gallery-dl2\gallery_dl\job.py", line 26, in __init__
    extr = extractor.find(extr)
           ^^^^^^^^^^^^^^^^^^^^
  File "D:\gallery-dl2\gallery_dl\extractor\__init__.py", line 195, in find
    for cls in _list_classes():
  File "D:\gallery-dl2\gallery_dl\extractor\__init__.py", line 234, in _list_classes
    for module in _module_iter:
  File "D:\gallery-dl2\gallery_dl\extractor\__init__.py", line 243, in _modules_internal
    yield __import__(module_name, globals_, None, (), 1)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\gallery-dl2\gallery_dl\extractor\2chan.py", line 11, in <module>
    from .common import Extractor, Message
  File "D:\gallery-dl2\gallery_dl\extractor\common.py", line 905, in <module>
    from requests.packages.urllib3.contrib import pyopenssl  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python3\Lib\site-packages\urllib3\contrib\pyopenssl.py", line 64, in <module>
    warnings.warn(
DeprecationWarning: 'urllib3.contrib.pyopenssl' module is deprecated and will be removed in urllib3 v2.1.0. Read more in this issue: https://github.com/urllib3/urllib3/issues/2680

Here is another simple command.

D:\gallery-dl2>gallery-dl --list-extractors
Traceback (most recent call last):
  File "D:\gallery-dl2\gallery_dl\__main__.py", line 20, in <module>
    gallery_dl.main()
  File "D:\gallery-dl2\gallery_dl\__init__.py", line 201, in main
    for extr in extractor.extractors():
                ^^^^^^^^^^^^^^^^^^^^^^
  File "D:\gallery-dl2\gallery_dl\extractor\__init__.py", line 220, in extractors
    return sorted(
           ^^^^^^^
  File "D:\gallery-dl2\gallery_dl\extractor\__init__.py", line 234, in _list_classes
    for module in _module_iter:
  File "D:\gallery-dl2\gallery_dl\extractor\__init__.py", line 243, in _modules_internal
    yield __import__(module_name, globals_, None, (), 1)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\gallery-dl2\gallery_dl\extractor\2chan.py", line 11, in <module>
    from .common import Extractor, Message
  File "D:\gallery-dl2\gallery_dl\extractor\common.py", line 905, in <module>
    from requests.packages.urllib3.contrib import pyopenssl  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python3\Lib\site-packages\urllib3\contrib\pyopenssl.py", line 64, in <module>
    warnings.warn(
DeprecationWarning: 'urllib3.contrib.pyopenssl' module is deprecated and will be removed in urllib3 v2.1.0. Read more in this issue: https://github.com/urllib3/urllib3/issues/2680

go to provided urllib3 issue link, it suggests to remove a package called urllib3-secure-extra to suppress warning, but when ran there is no package

D:\gallery-dl2>python -m pip uninstall urllib3-secure-extra
WARNING: Skipping urllib3-secure-extra as it is not installed.

Pretty sure its not urrlib3 issue, as I have another gallery-dl installed with pip and works fine via cmd.

C:\Users\usr\Documents>gallery-dl -v --ignore-config https://redgifs.com/watch/foolishforkedabyssiniancat
[gallery-dl][debug] Version 1.25.7
[gallery-dl][debug] Python 3.11.4 - Windows-10-10.0.19045-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.0.3
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://redgifs.com/watch/foolishforkedabyssiniancat'
[redgifs][debug] Using RedgifsImageExtractor for 'https://redgifs.com/watch/foolishforkedabyssiniancat'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.redgifs.com:443
[urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v2/auth/temporary HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v2/gifs/foolishforkedabyssiniancat HTTP/1.1" 200 None
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): thumbs44.redgifs.com:443
[urllib3.connectionpool][debug] https://thumbs44.redgifs.com:443 "GET /FoolishForkedAbyssiniancat.mp4?expires=1689758400&signature=v2:4da9132e188e05bf7824e5d0da818f0f3a022027bd4c99c74fc272bd54cd1938&for=138.199.50&hash=6163438793 HTTP/1.1" 200 4636670
* .\gallery-dl\redgifs\redgifs_foolishforkedabyssiniancat.mp4

what is the actual problem, anyone have idea?

mikf commented 1 year ago

Remove these lines from extractor/common.py so it doesn't attempt to import the pyopenssl module from urllib3: https://github.com/mikf/gallery-dl/blob/3c2c7e21dd589aee99ad22687733ffb149d337a1/gallery_dl/extractor/common.py#L901-L909

ghbook commented 1 year ago

Remove these lines from extractor/common.py so it doesn't attempt to import the pyopenssl module from urllib3:

https://github.com/mikf/gallery-dl/blob/3c2c7e21dd589aee99ad22687733ffb149d337a1/gallery_dl/extractor/common.py#L901-L909

Thanks, it worked.

BTW Could you please remove sys.exit from below and add gallery-dl.cmd file as above in project folder in next commit. https://github.com/mikf/gallery-dl/blob/248e8bc6996229fd8c3fb064afff14b2b722fa58/gallery_dl/__main__.py#L19-L20