Program not running without `_sqlite3` module

xtkoba commented 3 years ago

Checklist

[ ] I'm reporting a broken site support issue
[x] I've verified that I'm running yt-dlp version 2021.07.21
[ ] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log

$ yt-dlp --verbose --version
Traceback (most recent call last):
  File "/home/me/.local/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/me/.local/lib/python3.9/runpy.py", line 147, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/me/.local/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/me/.local/lib/python3.9/site-packages/yt_dlp/__init__.py", line 16, in <module>
    from .options import (
  File "/home/me/.local/lib/python3.9/site-packages/yt_dlp/options.py", line 22, in <module>
    from .cookies import SUPPORTED_BROWSERS
  File "/home/me/.local/lib/python3.9/site-packages/yt_dlp/cookies.py", line 5, in <module>
    import sqlite3
  File "/home/me/.local/lib/python3.9/sqlite3/__init__.py", line 23, in <module>
    from sqlite3.dbapi2 import *
  File "/home/me/.local/lib/python3.9/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'

Description

The _sqlite3 Python module seems to be required since version 2021.07.21.

Can we make the program work without that module? It is of course OK that some functions are disabled in that situation.

xfzv commented 3 years ago

I'm getting the same errors on my Gentoo system despite having dev-db/sqlite-3.35.5 installed. No issue on Arch Linux with sqlite 3.36.0-1 installed.

pacman -Q | rg sqlite

lib32-sqlite 3.36.0-1
sqlite 3.36.0-1

Does someone know which package(s) I need to install on Gentoo? and why is this now required in the first place? I cannot find any mention of it in 2021.07.21 changelog.

xtkoba commented 3 years ago

@xfzv As for Gentoo, try emerging dev-lang/python with sqlite USE flag, say:

# USE=sqlite emerge -1 dev-lang/python

xfzv commented 3 years ago

@xfzv As for Gentoo, try emerging dev-lang/python with sqlite USE flag, say:
# USE=sqlite emerge -1 dev-lang/python

Thank you! that was it. For some reasons I had dev-lang/python -sqlite in my package.use file.

Ashish0804 commented 3 years ago

It is needed for the new --cookies-from-browser option. @mbway anyway to not make it an requirement for people who are not going to use the option?

something like

if ytdlp.options.cookie_from_browser:
  *handle imports*

mbway commented 3 years ago

sorry about that. I was under the impression that the sqlite3 module was guaranteed to be present since it's part of the standard library.

It should be easy to work around this. --cookies-from-browser will require sqlite for loading all cookies except for safari which uses it's own format but I can make it print a warning when you attempt to load cookies when sqlite3 is not available like I did for keyring and pycryptodome.

I can work on a fix this evening.

ghost commented 3 years ago

It is part of the standard library, but it requires Python to be compiled with the native C extension, so it can be compiled without sqlite support.

pukkandan commented 3 years ago

While this particular issue can be easily fixed, we can't be expected to support anything but the official builds of python.

Does anyone know:

why do these distros use python without the C extension?
Can a full python be installed in it?

ghost commented 3 years ago

If you build the official Python from python.org without libsqlite-dev it will compile fine but warn that the sqlite3 module is unavailable.

pukkandan commented 3 years ago

gotcha. Do you know if this affects any other libraries?

mbway commented 3 years ago

this might not be a complete list, but should cover the main list of things that can be toggled: https://wiki.gentoo.org/wiki/Project:Python/Implementation_USE_flags The only ones that seem relevant are xml ssl and ipv6. I think it makes sense to support not having sqlite3 since it's not used for anything essential but the others are probably not worth working around.

yt-dlp / yt-dlp