swyxio / swyxdotio

This is the repo for swyx's blog - Blog content is created in github issues, then posted on swyx.io as blog pages! Comment/watch to follow along my blog within GitHub
https://swyx.io
MIT License
336 stars 45 forks source link

How to Download Twitter Spaces That Aren't Yours #412

Closed swyxio closed 2 years ago

swyxio commented 2 years ago

title: How to Download Twitter Spaces That Aren't Yours published: true description: Grabbing the audio tags: Tips category: tutorial slug: download-twitter-spaces canonical_url: https://www.swyx.io/download-twitter-spaces cover_image: https://user-images.githubusercontent.com/6764957/156055061-5c4f6bfc-a7e9-4e57-ba83-749476c60667.png

July 2023 update: Twitter seems to have broken this API now. I think a keen developer who knows GraphQL could possibly reconstruct the query since they probably just moved some fields around. please email or DM me if you are working on this.

the current two best ways are:


I recently caught the tail end of a TimescaleDB Twitter space that I wanted to snip for my mixtape in future. Spaces are only saved for a few days so if you want them beyond that you will have to download it. However, Twitter only lets the host of the space download. I tried using youtube-dl for it but it is currently unsupported.

Fortunately 89z on GitHub is very actively maintaining his own project that interfaces with some arbitrary social media: https://github.com/89z/mech and answered my questions enough to get me using it!

The project bundles releases only for Windows, so here's a quick sequence of instructions for Mac.

Prerequisites

In case you are a nontechnical person arriving from Google - these instructions are meant for developers but written out step by step as faithfully as I can. The only prerequisite you need is:

but I won't spend any time teaching you how to use the terminal.

First get the Space ID

The primary piece of knowledge you need is the Twitter Space ID. Given a Twitter space URL https://twitter.com/i/spaces/1YqGoprjyQEJv, the space ID is 1YqGoprjyQEJv. No API key is needed as far as I can tell, which is just peachy.

IMPORTANT - this process does not work for Twitter Super Follows Spaces as far as I can tell. If you are a developer, you MAY be able to fork the project to modify it. Let me know if you do.

The quickest way (June 2022)

I've now had enough support load that i realize I should just upload the binary myself so you can just use it.

If you trust that i'm not doing anything funky, go to https://github.com/sw-yx/mech/releases/tag/swyx-fork-v1 and download that twitter binary to somewhere local.

then, open your terminal to the same folder where that binary exists, and:

chmod +x twitter # for first time only
./twitter -c YOUR_SPACE_ID_HERE   

You may need to also set permissions in System Preferences.

Instructions to compile it yourself (May 2022)

We're going to use @89z's mech project, but as of the time that he used to support Twitter Spaces. so bear with me:

# make sure you have `git` installed!
git clone https://github.com/89z/mech
cd mech
git reset 5b977a09590b883584662a656a14d393337411c1 --hard # old twitter functionality was removed https://github.com/89z/mech/issues/71
cd cmd/twitter
# make sure you have `go` installed!
go build

This builds a binary called twitter which you can run:

# mac
./twitter -c YOUR_SPACE_ID_HERE   
# windows
twitter -c YOUR_SPACE_ID_HERE   

This starts the download:

$ ./twitter -c 1YqGoprjyQEJv    
POST https://api.twitter.com/1.1/guest/activate.json
GET https://twitter.com/i/api/graphql/Uv5R_-Chxbn1FEkyUkSW2w/AudioSpaceById?variables=%7B%22id%22%3A%221RDGlgoYkDkKL%22%2C%22isMetatagsQuery%22%3Afalse%2C%22withBirdwatchPivots%22%3Afalse%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withReplays%22%3Afalse%2C%22withScheduledSpaces%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Afalse%7D
GET https://twitter.com/i/api/1.1/live_video_stream/status/28_1496228425874374659
GET https://prod-fastly-us-east-1.video.pscp.tv/Transcoding/v1/hls/Ze8MHZCOuvGIbhBQYBAevAIotnQEwAixsGxoUgsOOJaJsf0Qm6Kxhwt3JH77MIxknSr_2nobkgkUDyYP9YwzbQ/non_transcode/us-east-1/periscope-replay-direct-prod-us-east-1-public/audio-space/playlist_16801097405082696033.m3u8?type=replay
0/1299 /Transcoding/v1/hls/Ze8MHZCOuvGIbhBQYBAevAIotnQEwAixsGxoUgsOOJaJsf0Qm6Kxhwt3JH77MIxknSr_2nobkgkUDyYP9YwzbQ/non_transcode/us-east-1/periscope-replay-direct-prod-us-east-1-public/audio-space/chunk_1645642768437487255_0_a.aac
1/1299 /Transcoding/v1/hls/Ze8MHZCOuvGIbhBQYBAevAIotnQEwAixsGxoUgsOOJaJsf0Qm6Kxhwt3JH77MIxknSr_2nobkgkUDyYP9YwzbQ/non_transcode/us-east-1/periscope-replay-direct-prod-us-east-1-public/audio-space/chunk_1645642768440813639_1_a.aac
2/1299 /Transcoding/v1/hls/Ze8MHZCOuvGIbhBQYBAevAIotnQEwAixsGxoUgsOOJaJsf0Qm6Kxhwt3JH77MIxknSr_2nobkgkUDyYP9YwzbQ/non_transc
# and so on...

The final file is an aac file which is still annoying but at least convertible and playable in Audacity.

image

Old Instructions (Feb 2022)

# make sure you have `git` installed!
git clone https://github.com/89z/mech
cd mech
# update: v1.49 onwards has the twitter spaces support again! https://github.com/89z/mech/issues/18#issuecomment-1058789325
git reset --hard v1.46.6 # only if you need to pin to a specific version that supports Twitter spaces, otherwise delete this line
cd cmd/twitter
go build
amiedelisa commented 2 years ago

When I enter go build, it says command not found.

bill830309 commented 2 years ago

how can i use it by Windows?

swyxio commented 2 years ago

the default is Windows, read their readme: https://github.com/89z/mech#how-to-install

unfortunately as this is open source software, support is very limited.

zire commented 2 years ago

./twitter -s spaces-ID returns an error. The correct command should be ./twitter -c spaces-ID.

harishkotra commented 2 years ago

As per May 2022 instructions, cd twitter after cd mech, there is no 'twitter' folder inside mech in the new commit. It doesn't work.

swyxio commented 2 years ago

@harishkotra oh dear looks like he deleted it a few days ago.. so sorry https://github.com/89z/mech/commit/c34237b5c047fc7142ad25c71b619f6cf3782e61

reverting instructions to the old one which should still work

harishkotra commented 2 years ago

@sw-yx Thank you.

swyxio commented 2 years ago

I have updated my instructions to pin to a specific commit since the developer has said he is intentionally removing twitter support permanently. If anyone wants to fork the project and maintain it I'd be happy to update this post to point to you. I'd like something for Super Follows Spaces.

# make sure you have `git` installed!
git clone https://github.com/89z/mech
cd mech
git reset 5b977a09590b883584662a656a14d393337411c1 --hard # old twitter functionality was removed https://github.com/89z/mech/issues/71
cd cmd/twitter
# make sure you have `go` installed!
go build

This builds a binary called twitter which you can run:

# mac
./twitter -c 1YqGoprjyQEJv   # if this doesnt work try ./twitter -s 1YqGoprjyQEJv
# windows
twitter -c 1YqGoprjyQEJv     # if this doesnt work try twitter -s 1YqGoprjyQEJv
harishkotra commented 2 years ago

@sw-yx Which one should I fork? I'd like to fork it first, not sure if I will be able to maintain.

Visdoom commented 2 years ago

I am having troubles getting this all installed within a conda environment on Mac OS with ARM processor. Installing git works like a charm but when I try to install go via conda install go it can only fetch the 1.17 release. Unfortunately when running go build it specifically asks for the 1.18 version. I tried installing go via brew install go@1.18 but this puts the package into the wrong directory (and it seems that brew does not allow for custom package locations). I would like to install go at the location /opt/homebrew/Caskroom/miniforge/base/envs/env/go as this is where conda expects to find the binary. Can anyone help?

yashptel commented 2 years ago

I am having troubles getting this all installed within a conda environment on Mac OS with ARM processor. Installing git works like a charm but when I try to install go via conda install go it can only fetch the 1.17 release. Unfortunately when running go build it specifically asks for the 1.18 version. I tried installing go via brew install go@1.18 but this puts the package into the wrong directory (and it seems that brew does not allow for custom package locations). I would like to install go at the location /opt/homebrew/Caskroom/miniforge/base/envs/env/go as this is where conda expects to find the binary. Can anyone help?

You can directly download and install from Go's official website: https://go.dev/dl/

elevationallah commented 2 years ago

Has anyone gotten this to work? I keep getting an "access denied" when trying to run the command. image

swyxio commented 2 years ago

@elevationallah it works on my mac - image

i dont use windows unfortunately so i dont know whats up with your error.

swyxio commented 2 years ago

@harishkotra fork https://github.com/89z/mech/ and rewind back to 5b977a09590b883584662a656a14d393337411c1

swyxio commented 2 years ago

yes for the record i support your decision to deprecate support based on your principles. thanks for the original code!

woj24 commented 2 years ago

I used May 2022 instructions, but after I use the Space ID, it gives me the following: "panic: invalid character '<' looking for beginning of value"

After few more trials, it started to give me the following: "panic dial tcp: lookup api.twitter.com: no such host"

I tried accessing api.twitter.com on my browser, and gave me the following: "This api.twitter.com page can't be found No web page was found for the web address: https://api.twitter.com HTTP ERROR 404"

peleg-audiolabs commented 2 years ago

To download Twitter Spaces I recommend this tool: https://www.audiolabs.io/download-space

swyxio commented 2 years ago

an alternative way from @simonw that uses youtube-dl! https://til.simonwillison.net/twitter/export-edit-twitter-spaces for those who cant figure out my/89z’s method

Xyroid commented 2 years ago

@sw-yx can the tool download the Twitter Spaces which was not marked for recording? I tried on Windows with https://github.com/89z/mech/releases/tag/v1.45.3 but got following error.

I tried for this Spaces - https://twitter.com/i/spaces/1ZkKzXPLMewJv

non_recorded_space

swyxio commented 2 years ago

well, no, it wasn't recorded 🙃

lorensr commented 1 year ago

This worked for me, thanks @sw-yx! After downloading, needed to:

chmod +x twitter
./twitter

System Preferences -> General -> Allow twitter at bottom

swyxio commented 1 year ago

ah thanks @lorensr will update. I dont understand "System Preferences -> General -> Allow twitter at bottom" - allow twitter to do what?

murgascarrillo commented 1 year ago

Hello! I am getting the following output after trying '-c spaces_id': "-c: orden no encontrada"

ctrlcctrlv commented 1 year ago

This no longer works. I get a small AAC file. It seems to not know it has to keep reloading the dynamic playlist.

zire commented 1 year ago

This is how I download Twitter Spaces now, with another method. https://mora.app/planet/l6mn5-5qaaa-aaaan-qddfa-cai/7TVQYR7WTZ8X0DB9CM44HVHBP1

Chiplis commented 1 year ago

Moonbird author here, if you're on Windows or Mac you don't need to install the Rust toolchain, a .exe compiled version is available here: https://github.com/Chiplis/moonbird/releases/