Twitter Paywalling API Usage

[SamTherapy]

You've probably already seen this but here it is again: https://twitter.com/TwitterDev/status/1621026986784337922

Twitter plans on killing off free access to its APIs in a week from when this is being written (9 February)

[yogan]

Yeah, guess that's finally it. No more fancy technical solutions to keep a dying thing alive. It was fun while it lasted.

[NicolasConstant]

Yes, I already seen it but thanks for the heads up. 🙂

We'll see if it really happen. And at what price. It will be also interesting to see what they'll do with the "undocumented" API also used by Nitter.

[jk]

Are workarounds like scraping feasible? I'm running my own birdsitelive instance and only follow 4 accounts there. So the load isn't that high. But most probably would violate the ToS, right?

[Write]

Are workarounds like scraping feasible? I'm running my own birdsitelive instance and only follow 4 accounts there. So the load isn't that high. But most probably would violate the ToS, right?

No idea how to make it compatible, however you can easily scrap with https://github.com/JustAnotherArchivist/snscrape

[pvieito]

Nitter seems to be using Twitter private API (ie. the one used by Twitter apps) to work.

[eric-pierce]

@jk you can follow accounts in an RSS reader via Nitter's RSS feed feature. I haven't tried it but Nitter's RSS could potentially be used in conjunction with https://github.com/dariusk/rss-to-activitypub

[eric-pierce]

I just came across https://bird.makeup/, which appears to be a fork of BSL that doesn't require the API. I was able to get it up and running on my installed instance (it was a bit finnacy, and I had to manually populate the database with a single user + one account to follow before the web-ui would work) but otherwise it appears to be working fine.

@jk @rursache @Write

There is a docker-compose example on the git repo with the source code

[NicolasConstant]

Yes, I'm aware of this fork and if this new policy goes live, I might implement a way to support both ways: via the API (better content, more ethical) or via the "undocumented API" like Nitter or this fork.

[eric-pierce]

@NicolasConstant thanks for the comment! If you do go that route I'll revert back to the official BSL

[felixmueller]

I just came across https://bird.makeup/, which appears to be a fork of BSL that doesn't require the API. I was able to get it up and running on my installed instance (it was a bit finnacy, and I had to manually populate the database with a single user + one account to follow before the web-ui would work) but otherwise it appears to be working fine.

@jk @rursache @Write

There is a docker-compose example on the git repo with the source code

Thank you very much @eric-pierce for pointing this out! I managed to get my instance running by following your guide to pre-populate the database with one twitter user and one follower. So far, it seems to basically work. But I have the issue that new tweets are only displayed rarely. The most recent tweet id is inserted into the database, but the tweet is not showing up in the timeline. When I unfollow and re-follow the affected user, the new tweets are displayed, but only once, and no new tweets are shown again. Do you experience the same? I already contacted https://github.com/cloutier via Mastodon, but no reply so far. Do you use any additional settings, e.g. "ParallelTwitterRequests", to limit the requests that are basically running in a loop constantly? Thank you for any help!

@NicolasConstant Is there any timeline for the official BSL to support Twitter without the API? bird.makeup has a couple of new and neat features, for example displaying native retweets as boosts. Would you consider implementing these to BSL, too? Thanks a lot for your contributions!

[eric-pierce]

@felixmueller I haven't noticed any issues, but I could be missing tweets and not know it. If you're interested in my docker-compose setup you can check it out here: https://github.com/eric-pierce/Mastadon-and-MC/blob/main/docker-compose.yml#L554 If you're looking for bird.makeup support it looks like the repo is managed here https://todo.sr.ht/~cloutier/bird.makeup rather than on github

I'll second @felixmueller and would love to see some of the changes from the bird.makeup fork incorporated into BSL. I know @NicolasConstant has been waiting to see if the announced API changes actually go into effect, so we'll see if they do!

[NicolasConstant]

bird.makeup has a couple of new and neat features, for example displaying native retweets as boosts. Would you consider implementing these to BSL, too?

Unfortunately, native retweets and boosts are limited by how mastodon and most activitypub application handle them: you can't perform them when publishing on follower-only mode (that's why the fork changed this to publish everything publicly). follower-only publication is one of the most important feature of BSL since it prevent potential personal data being mirrored publicly everywhere without the original owner being aware of it. So that's why I didn't implemented them (I've done some tests beforehand on this subject, this limitation isn't really a technical one: it's an ethical choice).

But I'm not against merging other features from other forks obviously, if it doesn't go against some foundations of the project (that is having the best ethical mirroring service possible).

The ethical aspect is also why I always relied on proper API access to mirror content. But if twitter removes this kind of access, it's better to move the project to a more loose position about it, since a broken project won't be useful in any way, and people might rely on worse solutions (setting up public mirrors relying on RSS nitter feeds for example, common practice back in the day that was my first motivation to create BSL).

[eric-pierce]

@NicolasConstant so far it looks like your bet has paid off - did Twitter forget about the API changes? I appreciate your POV around the ethical side of public vs private boosts/retweets.

[felixmueller]

Thank you very much @eric-pierce for pointing this out!

I managed to get my instance running by following your guide to pre-populate the database with one twitter user and one follower.

So far, it seems to basically work. But I have the issue that new tweets are only displayed rarely. The most recent tweet id is inserted into the database, but the tweet is not showing up in the timeline. When I unfollow and re-follow the affected user, the new tweets are displayed, but only once, and no new tweets are shown again.

Do you experience the same?

I already contacted https://github.com/cloutier via Mastodon, but no reply so far.

Do you use any additional settings, e.g. "ParallelTwitterRequests", to limit the requests that are basically running in a loop constantly? Thank you for any help!

Quick update: the latest build of bird.makeup seems to have resolved my described issue with missing tweets!

[sentience]

Basic API access will be US$100/month for 10,000 read requests. How easy would it be to have Birdsitelive display a saturation limit for that request volume?

[davinkevin]

FYI, I won't pay for it, so I'll potentially try an alternative solution, if there is any.

PS: I don't know how long the current setup will be available for already existing users 🤷… my instance is still working, so 🤞

[rursache]

An updated and working bird.makup docker image is available here. Uses nitter instead of twitter so it's free