search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.7k stars 953 forks source link

[twitter][error] Using `filename` from twitter.py extractor to retrieve media file name give error and stop downloading if the the media file wasn't hosted on twitter #2482

Closed Pandaa-P closed 1 year ago

Pandaa-P commented 2 years ago

For archiving purposes I tried to download the whole media files to specific twitter members. I changed the configuration file according to the naming formula ^[1] I want and everything was fine, well until gallery-dl hit a tweet where the media file wasn't hosted on twitter [it was on YouTube]. gallery-dl wouldn't skip it and keep saying:

twitter\TwitterUserHere\[twitter] TwitterUserHere—1512139315035480071—1512139315035480071—2022.02.02.webm 
twitter: Unable to download data:  FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.download/[twitter] TwitterUserHere—1512139315035480071—1512139315035480071—2022.02.02.webm'

For instance, if I start gallery-dl with the default configuration the media file would be downloaded just fine. It just when the filename was used gallery-dl can't skip the part that it's not hosted on twitter therefor there is no name to the media file from twitter's metadata.

As a solution, maybe if the filename wasn't found gallery-dl would fill the required string with nothing as in blank name

twitter] TwitterUserHere—1512139315035480071—2022.02.02.webm

or repeat the tweet_id just like what it show up in the error message

twitter] TwitterUserHere—1512139315035480071—1512139315035480071—2022.02.02.webm

|---------------------------------------------------------------------

^[1]

"filename": "[{category}] {author[name]}—{date:%Y.%m.%d}—{retweet_id|tweet_id}—{filename}.{extension}",
--------------------------------------------------------------------------------^
AlttiRi commented 2 years ago

I never have any issue with this filename format. This filename format applies only for Twitter media files. That are hosted by Twitter.

until gallery-dl hit a tweet where the media file wasn't hosted on twitter [it was on YouTube].

For example, a tweet with youtube media: https://twitter.com/YouTube/status/1496635988214439936

gallery-dl just skips it.

I don't think that using of {filename} can have any effect.

Problems with filenames are usually because of some property is missed, it's not this case.

FileNotFoundError: [Errno 2] No such file or directory

It looks that it's an error of ffmpeg.

Most likely it's a problem with a possprocessor which you added.

No config file (at least a part for twitter), no example URL, and a very edited log (a part of it) — it's difficult to help.

mikf commented 2 years ago

As a solution, maybe if the filename wasn't found gallery-dl would fill the required string with nothing as in blank name

You can do this yourself by using {filename:?//}, but a missing field value is not what is causing this error.

or repeat the tweet_id just like what it show up in the error message

{filename|tweet_id}

The biggest question I have is how "[{category}] {author[name]}—{date:%Y.%m.%d}—{retweet_id|tweet_id}—{filename}.{extension}" somehow produces [twitter] TwitterUserHere—1512139315035480071—1512139315035480071—2022.02.02.webm as filename.

That looks like it was generated from an entirely different format string. Are you perhaps using downloader.ytdl.outtmpl for ytdl filenames?

Problems with filenames are usually because of some property is missed, it's not this case.

And even if, undefined fields get filled in with None and do not cause a hard crash

$ gallery-dl -f "{foobar}.{extension}" -s https://twitter.com/i/status/604341487988576256
None.jpg

FileNotFoundError: [Errno 2] No such file or directory

It looks that it's an error of ffmpeg.

I don't think so- That looks very much like an error directly in Python code.

AlttiRi commented 2 years ago

The biggest question I have is how ... somehow produces ...

As I said, for the purposes of conspiracy he edited a lot (and wrong) the log: 2022.02.02 is not a date of 1512139315035480071 tweet, also it's not possible to download webm from Twitter (all videos are mp4 there).

So, this log does not help, but more disinform.

Was webm extension in the real log, or no?

The part of issues for FileNotFoundError: [Errno 2] No such file or directory search show that they used youtube-dl/yt-dlp.

Pandaa-P commented 2 years ago

This filename format applies only for Twitter media files. That are hosted by Twitter.

I know, But I test it again minutes ago and removing {filename} will download the file with no problem or error. adding this string rise the issue again,

with {filename}

gallery-dl -u MyName -p MyPass https://twitter.com/YouTube/status/1496635988214439936 -d test -c gallery-dl.conf
100% 596.05kB 215.61kB/s twitter: Unable to download data:  FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.download/[twitter] YouTube—1496635988214439936—1496635988214439936—2022.02.24.webm'

It looks that it's an error of ffmpeg. Most likely it's a problem with a possprocessor which you added.

This is the only change I add to the default behavior of gallery-dl, it's based on this https://github.com/mikf/gallery-dl/discussions/2476#discussioncomment-2504391 add to the config template from here https://github.com/mikf/gallery-dl/blob/master/docs/gallery-dl.conf

        "twitter":
        {
            "cards": true,
            "conversations": true,
            "filename": "[{category}] {author[name]}—{retweet_id|tweet_id}—{filename}—{date:%Y.%m.%d}.{extension}",
            "logout": true,
            "pinned": true,
            "quoted": true,
            "replies": "self",
            "retweets": true,
            "text-tweets": true,
            "twitpic": true,
            "videos": true,
            "users": "media",
            "skip": true,
            "fallback": true,
            "blacklist": ""

        },

The biggest question I have is how "[{category}] {author[name]}—{date:%Y.%m.%d}—{retweet_id|tweet_id}—{filename}.{extension}" somehow produces [twitter] TwitterUserHere—1512139315035480071—1512139315035480071—2022.02.02.webm as filename

I took it from @AlttiRi's https://github.com/AlttiRi/twitter-click-and-save#readme project, maybe he can answer who he came up with this format.

As I said, for the purposes of conspiracy he edited a lot (and wrong) the log: 2022.02.02 is not a date of 1512139315035480071 tweet, also it's not possible to download webm from Twitter (all videos are mp4 there).

what conspiracy? the only edit was "1512139315035480071" since the original was heavy NSFW material.

also it's not possible to download webm from Twitter (all videos are mp4 there). It was a link to youtube video so the download was webm. and yes it was a real non-edited info from me.

Here the command without my user&pass but with --verbose and without any edit.

gallery-dl https://twitter.com/YouTube/status/1496635988214439936 -d test -c gallery-dl.conf --verbose
gallery-dl: Version 1.20.2-dev - Executable
gallery-dl: Python 3.9.9 - Windows-10-10.0.17763-SP0
gallery-dl: requests 2.25.1 - urllib3 1.25.11
gallery-dl: Starting DownloadJob for 'https://twitter.com/YouTube/status/1496635988214439936'
twitter: Using TwitterTweetExtractor for 'https://twitter.com/YouTube/status/1496635988214439936'
urllib3.connectionpool: Starting new HTTPS connection (1): twitter.com:443
urllib3.connectionpool: https://twitter.com:443 "GET /i/api/2/timeline/conversation/1496635988214439936.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&ext=mediaStats%2ChighlightedLabel HTTP/1.1" 200 6813
twitter: Using download archive 'C:\Users\Pandaa-P/gallery-dl/archive.sqlite3'
# test\twitter\YouTube\[twitter] YouTube—1496635988214439936—1496635988214439936—2022.02.24
twitter: Skipping 1496636072494698501 (reply)
twitter: Skipping 1496636073040125956 (reply)
twitter: Skipping 1496636076819197952 (reply)
twitter: Skipping 1496636291101962242 (reply)
twitter: Skipping 1496636351294414848 (reply)
twitter: Skipping 1496636447260278784 (reply)
twitter: Skipping 1496636693046255616 (reply)
twitter: Skipping 1496636849376444418 (reply)
twitter: Skipping 1496637142562390029 (reply)
twitter: Skipping 1496637367611969538 (reply)
urllib3.connectionpool: https://twitter.com:443 "GET /i/api/2/timeline/conversation/1496635988214439936.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&cursor=LBmmmoCygafjj8UpiIC9uYalj8UphMCy4bPwj8UphIC%2BzZ7Sj8UpgICz7ZHJj8UphMC5kd%2Bxj8UpisCosYKlj8UpgIDQzeq6j8UpgMC9zaKlj8UpgIC5zZ%2B1j8UpJQISAAA%3D&ext=mediaStats%2ChighlightedLabel HTTP/1.1" 200 8547
twitter: Skipping 1496638188202479624 (reply)
twitter: Skipping 1496639926808494084 (reply)
twitter: Skipping 1496640119356502019 (reply)
twitter: Skipping 1496641965722636288 (reply)
twitter: Skipping 1496642135751380992 (reply)
twitter: Skipping 1496645544399499264 (reply)
twitter: Skipping 1496647857511800833 (reply)
twitter: Skipping 1496649672177422343 (reply)
twitter: Skipping 1496655434366824451 (reply)
twitter: Skipping 1498747726762168322 (reply)
urllib3.connectionpool: https://twitter.com:443 "GET /i/api/2/timeline/conversation/1496635988214439936.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&cursor=LBn2FJqAsoGn44%2FFKYDAvpX5hZLFKYiAvbmGpY%2FFKYiAsqGvhZHFKYTAsuGz8I%2FFKZDAvtGVoJDFKYSAvs2e0o%2FFKYCAs%2B2RyY%2FFKYDAuK2G%2FJHFKYTAuZHfsY%2FFKYrAqLGCpY%2FFKYCA0M3quo%2FFKY7AuPXPvJXFKYDAvc2ipY%2FFKYDA08mtzJPFKYbAve3JkJHFKYbAvcmDjJjFKYCAuc2ftY%2FFKYKAuaWP64%2FFKYKAuc3%2F0pTFKSUCEgAA&ext=mediaStats%2ChighlightedLabel HTTP/1.1" 200 7904
twitter: Skipping 1496657772259450882 (reply)
twitter: Skipping 1496660697329115136 (reply)
twitter: Skipping 1496660935863382016 (reply)
twitter: Skipping 1496861107914477572 (reply)
twitter: Skipping 1496665513531174917 (reply)
twitter: Skipping 1496666687415984135 (reply)
twitter: Skipping 1496669477987094532 (reply)
twitter: Skipping 1496684986572820481 (reply)
twitter: Skipping 1496690807809712130 (reply)
twitter: Skipping 1496701005437628422 (reply)
urllib3.connectionpool: https://twitter.com:443 "GET /i/api/2/timeline/conversation/1496635988214439936.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&cursor=LBn2HJqAsoGn44%2FFKYSAuaGEl6jFKYrA05nb1pzFKYjAuNm8vZ7FKYDAvpX5hZLFKYiAvbmGpY%2FFKYiAsqGvhZHFKYLAqtWYxKXFKYTAsuGz8I%2FFKZDAvtGVoJDFKYSAvs2e0o%2FFKYCAs%2B2RyY%2FFKYDAuK2G%2FJHFKYTAuZHfsY%2FFKYSA0Z2OlJnFKYrAqLGCpY%2FFKYCA0M3quo%2FFKY7AuPXPvJXFKYDAufWgzJrFKYzAo8HO6KzFKY6AuamFm53FKYDAvc2ipY%2FFKYDA08mtzJPFKYbAve3JkJHFKYbAvcmDjJjFKYCAuc2ftY%2FFKYKAuaWP64%2FFKYKAuc3%2F0pTFKSUCEgAA&ext=mediaStats%2ChighlightedLabel HTTP/1.1" 200 8952
twitter: Skipping 1496746721543704577 (reply)
twitter: Skipping 1496753321419231234 (reply)
twitter: Skipping 1496889995373600770 (reply)
twitter: Skipping 1496895488880222210 (reply)
twitter: Skipping 1496901047473741863 (reply)
twitter: Skipping 1496902862789820418 (reply)
twitter: Skipping 1496956230446821377 (reply)
twitter: Skipping 1496980049395867650 (reply)
twitter: Skipping 1496990311389622273 (reply)
urllib3.connectionpool: https://twitter.com:443 "GET /i/api/2/timeline/conversation/1496635988214439936.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&cursor=LBn2JJqAsoGn44%2FFKYSAuaGEl6jFKYSAuYmFoYXGKYrA05nb1pzFKYjAuNm8vZ7FKYDAvpX5hZLFKYiAvbmGpY%2FFKYiAsqGvhZHFKYLAqtWYxKXFKYTAsuGz8I%2FFKZDAvtGVoJDFKYSAvs2e0o%2FFKYLAo%2Bm3sLDGKYCAs%2B2RyY%2FFKYDAuK2G%2FJHFKYTAuZHfsY%2FFKYSA0Z2OlJnFKYrAqLGCpY%2FFKYCA0M3quo%2FFKYTAovnmzcTFKY7AuPXPvJXFKYLAuYHV8KDGKYDAufWgzJrFKYzAo8HO6KzFKY6AuamFm53FKYSAo6Gj4YLGKYSA0v2gzojGKYDAvc2ipY%2FFKc6AqenL5IfGKYDA08mtzJPFKYLAvoXSzcHFKYbAve3JkJHFKYbAvcmDjJjFKYCAuc2ftY%2FFKYKAuaWP64%2FFKYKAuc3%2F0pTFKSUCEgAA&ext=mediaStats%2ChighlightedLabel HTTP/1.1" 200 10803
twitter: Skipping 1497077275958857760 (reply)
# test\twitter\denny_deluxe\[twitter] broderick—1497044382482149387—FMaR7iYXsBgov-B—2022.02.25.jpg
# test\twitter\denny_deluxe\[twitter] broderick—1497044382482149387—FMaR7iYXsAAjy3i—2022.02.25.jpg
# test\twitter\denny_deluxe\[twitter] broderick—1497044382482149387—FMaR7iYXsBsRWjh—2022.02.25.jpg
# test\twitter\denny_deluxe\[twitter] broderick—1497044382482149387—FMaR7jUWUAAb7Fx—2022.02.25.jpg
twitter: Skipping 1497114473970245633 (reply)
twitter: Skipping 1497117209277210633 (reply)
twitter: Skipping 1497237049073713158 (reply)
twitter: Skipping 1497314904050700291 (reply)
twitter: Skipping 1497465471230038018 (reply)
twitter: Skipping 1497541829306290186 (reply)
twitter: Skipping 1497560033726545920 (reply)
twitter: Skipping 1497640064997806080 (reply)
twitter: Skipping 1497677305262399490 (reply)
urllib3.connectionpool: https://twitter.com:443 "GET /i/api/2/timeline/conversation/1496635988214439936.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&cursor=LBn2LIbAuM2b%2FsPHKZqAsoGn44%2FFKYSAuaGEl6jFKYSAuYmFoYXGKYrA05nb1pzFKYTAosGwuojIKYjAuNm8vZ7FKYDAvpX5hZLFKYiAvbmGpY%2FFKYiAsqGvhZHFKYLAqtWYxKXFKYTAsuGz8I%2FFKZDAvtGVoJDFKYSAvs2e0o%2FFKYLAo%2Bm3sLDGKYCAs%2B2RyY%2FFKYDAuK2G%2FJHFKYTAuZHfsY%2FFKYSA0Z2OlJnFKYrAqLGCpY%2FFKZTAuMGAl6vIKYCA0M3quo%2FFKYTAovnmzcTFKY7AuPXPvJXFKYLAuYHV8KDGKYDAsrmJ7dfIKcDAsq249tfGKYDAufWgzJrFKYzAo8HO6KzFKY6AuamFm53FKYSAo6Gj4YLGKYSA0v2gzojGKYSAo4Xf5OjIKYDAvc2ipY%2FFKYyAvum6yqDHKc6AqenL5IfGKYDA08mtzJPFKYLAvoXSzcHFKYbAve3JkJHFKYKAvZHT6%2BjGKYbAvcmDjJjFKYCAuc2ftY%2FFKYKAuaWP64%2FFKYKAuc3%2F0pTFKSUCEgAA&ext=mediaStats%2ChighlightedLabel HTTP/1.1" 200 9309
twitter: Skipping 1497683484915245062 (reply)
twitter: Skipping 1497729035530559489 (reply)
twitter: Skipping 1497955376850944009 (reply)
twitter: Skipping 1497993758985490433 (reply)
twitter: Skipping 1498032900113113093 (reply)
twitter: Skipping 1498103971558445056 (reply)
# test\twitter\MomRovin\[twitter] slpng_giants—1498102907048062978—FMpUpwDUUAAzwc_—2022.02.28.jpg
twitter: Skipping 1498256627476094980 (reply)
twitter: Skipping 1498316330289803265 (reply)
twitter: Skipping 1498348575490822144 (reply)
twitter: Skipping 1498732413819789318 (reply)
urllib3.connectionpool: https://twitter.com:443 "GET /i/api/2/timeline/conversation/1496635988214439936.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&cursor=LBn2NobAuM2b%2FsPHKZqAsoGn44%2FFKYSAuaGEl6jFKYSAuYmFoYXGKYLAqIHj2PjJKYrA05nb1pzFKYTAosGwuojIKYjAuNm8vZ7FKYDAvpX5hZLFKYiAvbmGpY%2FFKYqAuZGLv4rKKYiAsqGvhZHFKYLAqtWYxKXFKYTAsuGz8I%2FFKZDAvtGVoJDFKYSAvs2e0o%2FFKYDAvtX%2B56rKKYLAo%2Bm3sLDGKYCAs%2B2RyY%2FFKYDAuK2G%2FJHFKYTAuZHfsY%2FFKYSA0Z2OlJnFKYiAs%2FHdnfDKKYDAvv3nhZrLKYyAvYW5zOvIKYrAqLGCpY%2FFKZTAuMGAl6vIKYCA0M3quo%2FFKYLAucHysIvLKYTAovnmzcTFKY7AuPXPvJXFKYLAuYHV8KDGKYDAsrmJ7dfIKcDAsq249tfGKYDAufWgzJrFKYzAo8HO6KzFKY6AuamFm53FKYSAo6Gj4YLGKYSA0v2gzojGKYyAvbGSzMjMKYSAo4Xf5OjIKYDAvc2ipY%2FFKYyAvum6yqDHKc6AqenL5IfGKYDA08mtzJPFKYLAvoXSzcHFKYbAve3JkJHFKYKAvZHT6%2BjGKYbAvcmDjJjFKYCAuc2ftY%2FFKYKAs8nrp4DJKYKAuaWP64%2FFKYKAuc3%2F0pTFKZKAsunRnufJKSUCEgAA&ext=mediaStats%2ChighlightedLabel HTTP/1.1" 200 8114
twitter: Skipping 1497809173135773699 (reply)
twitter: Skipping 1498732606090788867 (reply)
twitter: Skipping 1498760712835026946 (reply)
twitter: Skipping 1499222849806028800 (reply)
twitter: Skipping 1500120027890126853 (reply)
twitter: Skipping 1503427650492190725 (reply)
twitter: Skipping 1496660114312290307 (reply)
twitter: Skipping 1497735458654150656 (reply)
twitter: Skipping 1497748056116236292 (reply)
AlttiRi commented 2 years ago

what conspiracy?

Just a joke.

Yeah, the reason is "cards": true.

With it gallery-dl automatically uses youtube-dl. And applies the wrong {filename}.

It's strange, but my main OS, gallery-dl does not try to download youtube videos in tweets. (While "cards": true, is the default value.)

UPD. Well, I have update gallery-dl in my Virtual Machine and now

config ```json { "extractor": { "base-directory": "./", "twitter": { "cards": true, "filename": "[{category}] {author[name]}—{date:%Y.%m.%d}—{retweet_id|tweet_id}—{filename}.{extension}", "text-tweets": true } } } ```

Does not download youtube videos from tweets too. Even with all settings from the comment above. What I do wrong?

mikf commented 2 years ago

Pandaa-P is using a gallery-dl version before f2e8aedd. After that cards has to explicitly be set to "ytdl" to invoke youtube-dl on card tweets.

But even when rolling back to v1.20.1 (the version before switching to twitter graphql api and closest to what Pandaa-P has), I cannot reproduce a crash.

Maybe it crashed because the original tweet is nsfw and youtube-dl cannot access it, does not download anything, and there is no expected file, hence FileNotFoundError.

AlttiRi commented 2 years ago

With "cards": "ytdl" it does the work.

I also successfully download it (https://twitter.com/YouTube/status/1496635988214439936), the result file is webm.

However, in his log above even downloading of this tweet fails.

Also using for {filename} the tweet's ID in this case looks as a bug. (So, it was not a wrong editing of the file name.) None or an empty string is more correct.

BTW, is it possibly to not apply Twitter's naming rule on Youtube videos?

UPD.

Why it (gallery-dl, or youtube-dl?) put the file in tmp/.download/?

UPD2.

https://github.com/mikf/gallery-dl/blob/f4a7c6eca27174643f3d0e8027a440793b0baf56/docs/gallery-dl-example.conf#L307

AlttiRi commented 2 years ago

Try to remove

"part-directory": "/tmp/.download/",

or replace it with

"part-directory": "./tmp/.download/",
AlttiRi commented 2 years ago

Yeah. It works.

I can reproduce the bug with "downloader"."part-directory":

"part-directory": "/tmp/.download/",

It's the problem's reason.

/tmp/.download/ is absolute path — F:/tmp/.download/ (after resolving on disk F).

But gallery-dl the first time (when downloads the file) handles it as the relative one ("./tmp/.download/"/"tmp/.download/"). That's wrong.

Also I can not confirm that removing {filename} helps. It as expected does not play any role.

Pandaa-P commented 2 years ago

Thank you so much guys I will test the solutions above and bring back the results Thank you so much again

By the way can we have a config entry like

"external": false,

to force gallery-dl to not download external files that not hosted on twitter?

AlttiRi commented 2 years ago

Update to the latest version and set "cards" option. "cards": false — download only uploaded media (videos and images), "cards": true — also download link's thumbnails, (default) (For example: [1], [2]) "cards": "ytdl" — also download Youtube videos

https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#extractortwittercards

Pandaa-P commented 2 years ago

With "cards": "ytdl" it does the work.

Yep, it works.

Does not download youtube videos from tweets too. Even with all settings from the comment above. What I do wrong?

I honestly don't know why my gallery-dl try to pull videos that hosted on site other than twitter, maybe it's related to this config that supposed to pull everything from the profile.

Also I can not confirm that removing {filename} helps. It as expected does not play any role.

Well that idea of {filename} was the issue came to my mind when I pasted this config to my local config. I tinkered with the new lines and also saw that the tweet id was duplicated for the external videos and concluded that maybe gallery-dl wasn't able to put a file name after the tweet id since it wasn't part of twitter metadata. removing it solve my problem and that gave me the bias that my theory was correct. But boy was I wrong.

I will update to the last version since mine was a bit outdated.

By the way where I can read about all these small details like "cards": false, "cards": true, and"cards": "ytdl"`? Is there full documentation somewhere or do I need to hunt git commits for them?

Thank you so much guys, really appreciate your help. Also thank you so much for this naming format of twitter AlttiRi

Hrxn commented 2 years ago

By the way can we have a config entry like

"external": false,

to force gallery-dl to not download external files that not hosted on twitter?

With the "blacklist" option maybe?

mikf commented 2 years ago

By the way where I can read about all these small details like "cards": false, "cards": true, and"cards": "ytdl"`? Is there full documentation somewhere or do I need to hunt git commits for them?

All config options can be found in https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst Potential changes to those are documented in the changelog and in release notes

AlttiRi commented 2 years ago

As for filename for youtube videos it makes sense to use "v" search param (the ID of the video), and the last part of pathname splitted by "/" for any other url.