Twitter shortener t.co redirect

CeruleanSky commented 4 years ago

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

[x] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
[x] Searched the bugtracker for similar pull requests
[x] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[x] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[x] Bug fix
[x] Improvement
[ ] New extractor
[ ] New feature

Description of your pull request and other information

When getting urls from twitter they are usually of the type http://t.co and have the possibility of an additional gateway/redirect that needs to be stripped for youtube-dl to download in an automated/streamlined fashion.

blackjack4494 commented 4 years ago

Can you provide a sample link?

CeruleanSky commented 4 years ago

Here you go: https://twitter.com/tetexalinfini/status/1290024792880320513 Which is the source of the problem link https://t.co/SVVAyeyZfG?amp=1 which has the gateway "https://twitter.com/safety/unsafe_link_warning?unsafe_link=" , it really interferes when you are trying to download a bunch of videos off of twitter as you can run into it enough to see it as a wrench thrown in the gears. The amount of links and video sites that have that warning seems to increase every day, maybe the filter is too sensitive. In this case, Bitchute is a supported plugin, but that extra redirect causes the issue.

I'm sure gateway redirects will become more common as time goes on, perhaps they should eventually have their own section, but I don't know the ones from other sites.

blackjack4494 commented 4 years ago

I added a very simple extractor to twitter.

https://github.com/blackjack4494/youtube-dlc/blob/f33b7b5eb454ab182a3c8740875274469f5c3e0b/youtube_dlc/extractor/twitter.py#L613-L619

All you have to do is implementing the behaviour there.

blackjack4494 commented 4 years ago

Closing this in favor for #196

blackjack4494 / youtube-dlc