bibanon / tubeup

Use yt-dlp to download video/metadata and upload to the Internet Archive.
https://pypi.python.org/pypi/tubeup/
GNU General Public License v3.0
420 stars 70 forks source link

Bug report: unit test not working on both github action and local #334

Closed orkeilius closed 1 month ago

orkeilius commented 1 month ago

Checklist

If you just delete all this text and post a issue it will be closed on sight.

Carefully read and work through this check list in order to prevent the most common mistakes and misuse of tubeup, put x into all relevant boxes (like this [x])

Dependency versions

Please provide version information from core dependencies:

yt-dlp version: 2024.09.27

internetarchive python client version: 4.1.0

Tubeup version: 2023.08.19

Expected behavior some unit test return error both on github action and on local tests/test_tubeup.py F..............F.... [ 80%]

Additional context log from github action on orkeilius\tubeup log from the last github action

local test: (the error is slightly different but seems to have the same cause)

vscode ➜ /workspaces/tubeup $ pytest --cov
============================================================================= test session starts ==============================================================================
platform linux -- Python 3.12.6, pytest-8.3.3, pluggy-1.5.0
rootdir: /workspaces/tubeup
configfile: pyproject.toml
plugins: cov-4.1.0, requests-mock-1.11.0
collected 25 items                                                                                                                                                             

tests/test_tubeup.py F...................                                                                                                                                [ 80%]
tests/test_utils.py .....                                                                                                                                                [100%]

=================================================================================== FAILURES ===================================================================================
________________________________________________________________________ TubeUpTests.test_archive_urls _________________________________________________________________________

self = <tests.test_tubeup.TubeUpTests testMethod=test_archive_urls>

    def test_archive_urls(self):
        tu = TubeUp(dir_path=os.path.join(current_path,
                                          'test_tubeup_rootdir'),
                    ia_config_path=get_testfile_path('ia_config_for_test.ini'))

        videobasename = os.path.join(
            current_path, 'test_tubeup_rootdir', 'downloads',
            'KdsN9YhkDrY')

        copy_testfiles_to_tubeup_rootdir_test()

        with requests_mock.Mocker() as m:
            # Mock the request to s3.us.archive.org, so it will responds
            # a custom json. `internetarchive` library sends GET request to
            # that url to check that we don't violate the upload limit.
            m.get('https://s3.us.archive.org',
                  content=b'{"over_limit": 0}',
                  headers={'content-type': 'application/json'})

            m.get('https://archive.org/metadata/youtube-KdsN9YhkDrY',
                  content=b'{}',
                  headers={'content-type': 'application/json'})

            # Mock the PUT requests for internetarchive urls that defined
            # in mock_upload_response_by_videobasename(), so this test
            # doesn't perform upload to the real archive.org server.
            mock_upload_response_by_videobasename(
                m, 'youtube-KdsN9YhkDrY', videobasename)

>           result = list(tu.archive_urls(
                ['https://www.youtube.com/watch?v=KdsN9YhkDrY']))

tests/test_tubeup.py:563: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tubeup/TubeUp.py:415: in archive_urls
    identifier, meta = self.upload_ia(basename, custom_meta)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tubeup.TubeUp.TubeUp object at 0x7f4fe4723e00>, videobasename = '/workspaces/tubeup/tests/test_tubeup_rootdir/downloads/KdsN9YhkDrY', custom_meta = None

    def upload_ia(self, videobasename, custom_meta=None):
        """
        Upload video to archive.org.

        :param videobasename:  A video base name.
        :param custom_meta:    A custom meta, will be used by internetarchive
                               library when uploading to archive.org.
        :return:               A tuple containing item name and metadata used
                               when uploading to archive.org and whether the item
                               already exists.
        """
        json_metadata_filepath = videobasename + '.info.json'
        with open(json_metadata_filepath, 'r', encoding='utf-8') as f:
            vid_meta = json.load(f)

        # Exit if video download did not complete, don't upload .part files to IA
        for ext in ['*.part', '*.f303.*', '*.f302.*', '*.ytdl', '*.f251.*', '*.248.*', '*.f247.*', '*.temp']:
            if glob.glob(videobasename + ext):
                msg = 'Video download incomplete, please re-run or delete video stubs in downloads folder, exiting...'
>               raise Exception(msg)
E               Exception: Video download incomplete, please re-run or delete video stubs in downloads folder, exiting...

tubeup/TubeUp.py:331: Exception
----------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------
Epic Ramadan - Video Background HD1080p
------------------------------------------------------------------------------ Captured log call -------------------------------------------------------------------------------
ERROR    tubeup.TubeUp:YoutubeDL.py:951 ERROR: [requests] Unexpected error: NoMockAddress: No mock address: GET https://www.youtube.com/watch?v=KdsN9YhkDrY&bpctr=9999999999&has_verified=1; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
ERROR    tubeup.TubeUp:YoutubeDL.py:951 ERROR: [requests] Unexpected error: NoMockAddress: No mock address: POST https://www.youtube.com/youtubei/v1/player?prettyPrint=false; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
ERROR    tubeup.TubeUp:YoutubeDL.py:951 ERROR: [requests] Unexpected error: NoMockAddress: No mock address: POST https://www.youtube.com/youtubei/v1/player?prettyPrint=false; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
ERROR    tubeup.TubeUp:YoutubeDL.py:951 ERROR: [requests] Unexpected error: NoMockAddress: No mock address: GET https://www.youtube.com/s/player/b0557ce3/player_ias.vflset/en_US/base.js; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
ERROR    tubeup.TubeUp:YoutubeDL.py:951 ERROR: [requests] Unexpected error: NoMockAddress: No mock address: GET https://manifest.googlevideo.com/api/manifest/hls_variant/expire/1727705450/ei/Cl36ZrSZBNuDvdIPyoaQ8Ac/ip/86.214.98.135/id/29db0df588640eb6/source/youtube/requiressl/yes/xpc/EgVo2aDSNQ%3D%3D/playback_host/rr4---sn-hgn7yn7z.googlevideo.com/mh/la/mm/31%2C29/mn/sn-hgn7yn7z%2Csn-hgn7rn7r/ms/au%2Crdu/mv/m/mvi/4/pl/19/tx/51257028/txs/51257028%2C51257029%2C51257030%2C51257031%2C51257032/hfr/1/demuxed/1/tts_caps/1/maudio/1/initcwndbps/1658750/vprv/1/go/1/rqh/5/mt/1727683495/fvip/3/nvgoi/1/short_key/1/ncsapi/1/keepalive/yes/fexp/51299152/dover/13/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Cxpc%2Ctx%2Ctxs%2Chfr%2Cdemuxed%2Ctts_caps%2Cmaudio%2Cvprv%2Cgo%2Crqh%2Citag%2Cplaylist_type/sig/AJfQdSswRQIgdxL175JQzPndBEynpydFY7hritbSwuAz6jm6iv04dF0CIQCAB0kYJmdOjmglY-1vOhKxMVWv--llmH_2M5RCfgrnGw%3D%3D/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps/lsig/ABPmVW0wRQIhALaohcjH61hrQCTa_PFD9cSjsLvNPpQGY1g4xqIUMPDYAiA2GV9wEgJwzxoBQQrPmeUYgKwzzjVhcLY_efap_oB91g%3D%3D/file/index.m3u8; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
ERROR    tubeup.TubeUp:YoutubeDL.py:951 ERROR: [requests] Unexpected error: NoMockAddress: No mock address: GET https://www.youtube.com/watch?v=KdsN9YhkDrY&bpctr=9999999999&has_verified=1; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
ERROR    tubeup.TubeUp:YoutubeDL.py:951 ERROR: [requests] Unexpected error: NoMockAddress: No mock address: POST https://www.youtube.com/youtubei/v1/player?prettyPrint=false; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
ERROR    tubeup.TubeUp:YoutubeDL.py:951 ERROR: [requests] Unexpected error: NoMockAddress: No mock address: POST https://www.youtube.com/youtubei/v1/player?prettyPrint=false; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
ERROR    tubeup.TubeUp:YoutubeDL.py:951 ERROR: [requests] Unexpected error: NoMockAddress: No mock address: GET https://manifest.googlevideo.com/api/manifest/hls_variant/expire/1727705458/ei/El36ZsHKOpH3xN8Ps5Xb0A8/ip/86.214.98.135/id/29db0df588640eb6/source/youtube/requiressl/yes/xpc/EgVo2aDSNQ%3D%3D/playback_host/rr4---sn-hgn7yn7z.googlevideo.com/mh/la/mm/31%2C29/mn/sn-hgn7yn7z%2Csn-hgn7rn7r/ms/au%2Crdu/mv/m/mvi/4/pl/19/hfr/1/demuxed/1/tts_caps/1/maudio/1/initcwndbps/2095000/vprv/1/go/1/rqh/5/mt/1727683495/fvip/3/nvgoi/1/short_key/1/ncsapi/1/keepalive/yes/fexp/51299152/dover/13/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Cxpc%2Chfr%2Cdemuxed%2Ctts_caps%2Cmaudio%2Cvprv%2Cgo%2Crqh%2Citag%2Cplaylist_type/sig/AJfQdSswRQIgLQSmzQioo8doO8cNshZIHZjiejgDGBvAkHjVvdMU-9YCIQCP7A6vuqAMqwnXtKrqkyk4hHFqUx10sWpQEK27NSzFJg%3D%3D/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps/lsig/ABPmVW0wRgIhANhUYcDYJ8jOAp3ZfpG8Eoe446NnAU60sKV3ddNwqh5zAiEAsY_QWZlXUUcLD-ldARsKtBdhTRnfY8kaY845mmKKE1Q%3D/file/index.m3u8; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

---------- coverage: platform linux, python 3.12.6-final-0 -----------
Name                 Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------
tubeup/TubeUp.py       222     47    112     18    76%   52, 114-117, 122-123, 125, 130, 134-154, 156->164, 162, 166-170, 184-185, 295, 343, 351, 361, 369-375, 414->exit, 416, 479-480, 500-501, 506-507, 511-512, 551
tubeup/__init__.py       1      0      0      0   100%
tubeup/__main__.py      41     41      6      0     0%   54-119
tubeup/utils.py         11      0      2      0   100%
----------------------------------------------------------------
TOTAL                  275     88    120     18    68%

=========================================================================== short test summary info ============================================================================
FAILED tests/test_tubeup.py::TubeUpTests::test_archive_urls - Exception: Video download incomplete, please re-run or delete video stubs in downloads folder, exiting...
======================================================================== 1 failed, 24 passed in 30.74s =========================================================================
vxbinaca commented 1 month ago

yt-dlp problem not ours. Build tests will sometimes fail due to youtube anti-scraping tech they implemented. Even if it isn't anti-scrape, reproduce the bug and take it up with yt-dlp. We have no downloading code.