search

sekai-soft / guide-nitter-self-hosting

A guide for self-hosting a Nitter instance
MIT License
145 stars 8 forks source link

RSS not working well #20

Open sushione opened 2 months ago

sushione commented 2 months ago

Hello, having issues with RSS : When fetching from rss client, I am getting almost only « client error 429 », barely can use rss feeds. I am using my own private Nitter instance (from sekai-soft fork, "I only want a Nitter instance" method w/ brand new burner account).

I only have like 30 accounts followed, not even fetching that much (twice a day), use to work well on first fetch but then stop working as normal. I can wait hours between fetches and still have error, even after 12 hours.

Any idea on how to resolve it ? Thanks ! (Web access seems to be working well, no error)

PS : even with everything configured as expected in doc, it seems that it does not require psw key for RSS to work...

docker logs :

web    | 2024-08-19 13:04:49 - [accounts] Rate limited, retrying userScreenName request...
web    | 2024-08-19 13:04:49 - [accounts] Rate limited, retrying userScreenName request...
web    | error: ProtocolError, msg: Connection was closed before full request has been made, accountId: 182038099XXXXX, url: https://api.twitter.com/graphql/q94uRCEn65LZThakYcPT6g/TweetDetail?variables=%7B%22focalTweetId%22%3A%221497915933364563968%22%2C%22includeHasBirdwatchNotes%22%3Afalse%2C%22includePromotedContent%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Afalse%2C%22withV2Timeline%22%3Atrue%7D&features=%7B%22android_graphql_skip_api_media_color_palette%22%3Afalse%2C%22blue_business_profile_image_shape_enabled%22%3Afalse%2C%22creator_subscriptions_subscription_count_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Afalse%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Afalse%2C%22hidden_profile_likes_enabled%22%3Afalse%2C%22highlights_tweets_tab_ui_enabled%22%3Afalse%2C%22interactive_text_enabled%22%3Afalse%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Afalse%2C%22longform_notetweets_richtext_consumption_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Afalse%2C%22responsive_web_edit_tweet_api_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Afalse%2C%22responsive_web_media_download_video_enabled%22%3Afalse%2C%22responsive_web_text_conversations_enabled%22%3Afalse%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Afalse%2C%22responsive_web_twitter_blue_verified_badge_is_enabled%22%3Atrue%2C%22rweb_lists_timeline_redesign_enabled%22%3Atrue%2C%22spaces_2022_h2_clipping%22%3Atrue%2C%22spaces_2022_h2_spaces_communities%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Afalse%2C%22subscriptions_verification_info_enabled%22%3Atrue%2C%22subscriptions_verification_info_reason_enabled%22%3Atrue%2C%22subscriptions_verification_info_verified_since_enabled%22%3Atrue%2C%22super_follow_badge_privacy_enabled%22%3Afalse%2C%22super_follow_exclusive_tweet_notifications_enabled%22%3Afalse%2C%22super_follow_tweet_api_enabled%22%3Afalse%2C%22super_follow_user_api_enabled%22%3Afalse%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Afalse%2C%22tweetypie_unmention_optimization_enabled%22%3Afalse%2C%22unified_cards_ad_metadata_container_dynamic_card_content_query_enabled%22%3Afalse%2C%22verified_phone_label_enabled%22%3Afalse%2C%22vibe_api_enabled%22%3Afalse%2C%22view_counts_everywhere_api_enabled%22%3Afalse%7D
web    | 2024-08-19 13:18:10 - [accounts] Rate limited, retrying tweetDetail request...
KTachibanaM commented 2 months ago

It is odd that web access works well but RSS does not. Can you double check your RSS fetching frequency to make sure it's not fetching too fast?

The rss endpoint would return status code 200 but an empty response if a password is not provided. Can you make sure the INSTANCE_RSS_PASSWORD environment variable is set in the .env file and the value is not empty string?

sushione commented 2 months ago

Can you double check your RSS fetching frequency to make sure it's not fetching too fast?

Where can I check this ? I mean is it a nitter instance config ? Because if it’s on RSS client side I fetch manualy so it should not be that fast. I did exactly what was in guide but still no need for any psw for web access nor rss feed !

root@miniDeb:~/guide-nitter-self-hosting# ls -la
total 64
drwxr-xr-x 7 root root 4096 Aug 22 10:05 .
drwx------ 6 root root 4096 Aug  9 21:11 ..
-rw-r--r-- 1 root root  427 Aug  9 21:51 .env
drwxr-xr-x 8 root root 4096 Aug  4 15:58 .git
drwxr-xr-x 3 root root 4096 Aug  4 15:58 .github
-rw-r--r-- 1 root root   81 Aug  4 15:58 .gitignore
-rw-r--r-- 1 root root 1070 Aug  4 15:58 LICENSE
-rw-r--r-- 1 root root 4028 Aug  4 15:58 README.md
-rw-r--r-- 1 root root 2164 Aug  4 15:58 docker-compose.nitter-xposter.yml
-rw-r--r-- 1 root root  954 Aug  4 15:58 docker-compose.nitter.yml
-rw-r--r-- 1 root root 3573 Aug  4 15:58 docker-compose.yml
drwxr-xr-x 2 root root 4096 Aug  4 15:58 docs
drwxr-xr-x 2 root root 4096 Aug  4 15:58 miniflux-chilled-refresher
drwxr-xr-x 2 root root 4096 Aug 18 14:37 nitter-auth
-rw-r--r-- 1 root root 1932 Aug 22 10:01 nitter.conf
-rw-r--r-- 1 root root 1859 Aug  7 01:18 nitter.example.conf.OLD

nitter.conf rssMinutes has been tested with 1 and 100 but still doesn't work. TokenCount Also been tested with 1, 50 and 100, same problem.

rssMinutes = 10  # how long to cache rss queries
enableRSS = true  # set this to false to disable RSS feeds
tokenCount = 10

.env

TWITTER_USERNAME=XXXXXXX
TWITTER_PASSWORD= XXXXXXX

INSTANCE_RSS_PASSWORD= XXXXXXX
INSTANCE_WEB_USERNAME=nitter
INSTANCE_WEB_PASSWORD= XXXXXXX

INSTANCE_TITLE=SushiNitter
INSTANCE_THEME=Twitter Dark
INSTANCE_INFINITE_SCROLL=1
INSTANCE_HOSTNAME=192.168.XX.XX:8080

I confirm that it doesn't ask for password nor for web instance or RSS feed. Is there any other config I should do ? Everything should be on these two files, right ?

When I fetch manually my RSS feed from client, I keep having on logs : 

web    | 2024-08-26 08:08:56 - [accounts] Rate limited, retrying userTweets request...
web    | 2024-08-26 08:08:56 - [accounts] Rate limited, retrying userScreenName request...

What is even stranger is that if I had a new user's RSS feed it will fetch correctly last 20 tweets. It seems to be rate limited by user, not instance.

I confirm that web access is working as expected, very well (but don't ask for any password neither). Thanks !

KTachibanaM commented 4 weeks ago

Can you try removing the space in front of the TWITTER_PASSWORD, INSTANCE_RSS_PASSWORD and INSTANCE_WEB_PASSWORD values (unless your passwords do contain spaces, in which case there needs to be a new way to passing those passwords). Having spaces in front of those values makes them empty values.

KTachibanaM commented 4 weeks ago

Looks like I did make a mistake in the docker-compose.yml file that made passwords not working. Can you pull this repo and try again?

sushione commented 3 weeks ago

Tried to recreate from zero. git clone https://github.com/sekai-soft/guide-nitter-self-hosting && cd guide-nitter-self-hosting Just modified .env adding my data (untouched docker-compose.yml) get me a 502 bad gateway error after asking for my user name and password. If I try to reload then it can't connect anymore (connection lost).

Docker logs (my login&psw are Ok as I can connect from web browser) : 

Dumping auth env...
Dumping custom path env...
Dumping redis connection env...
Dumping instance customization env...
Dumping instance guardian env...
Writing Procfile...
system | Tmux socket name: overmind-src-8BZfQcgsZcg7KzZVdfabS
system | Tmux session ID: src
system | Listening at ./.overmind.sock
nginx  | Started with pid 23...
web    | Started with pid 22...
web    | Running auth...
nginx  | Running gen_nginx_conf...
nginx  | Launching nginx...
web    | DEBUG:root:TW_ANDROID_BASIC_TOKEN=Basic M25WdVNvQlpueDZVNHZ6VXhmNXc6QmNzNTlFRmJic2RGNlNsOU5nNzFzbWdTdFdFR3dYWEtTall2UFZ0N3XXXXX==
web    | DEBUG:root:start auth
web    | DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.twitter.com:443
web    | DEBUG:urllib3.connectionpool:https://api.twitter.com:443 "POST /oauth2/token HTTP/11" 200 152
web    | DEBUG:root:bearer_token=bearer AAAAAAAAAAAAAAAAAAAAAFXzAwAAAAAAMHCxpeSDG1gLNLghVe8d74hl6k4%3DRUMF4xAQLsbeBhTSRrCiQpJtxoGWeyHrDb5te2jpGskWDXXXXX
web    | DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.twitter.com:443
web    | DEBUG:urllib3.connectionpool:https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/11" 200 62
web    | DEBUG:root:guest_token=18436676949496XXXXX
web    | DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.twitter.com:443
web    | DEBUG:urllib3.connectionpool:https://api.twitter.com:443 "POST /1.1/onboarding/task.json?flow_name=login&api_version=1&known_device_token=&sim_country_code=us HTTP/11" 200 599
web    | DEBUG:urllib3.connectionpool:https://api.twitter.com:443 "POST /1.1/onboarding/task.json HTTP/11" 200 567
web    | DEBUG:urllib3.connectionpool:https://api.twitter.com:443 "POST /1.1/onboarding/task.json HTTP/11" 200 457
web    | DEBUG:urllib3.connectionpool:https://api.twitter.com:443 "POST /1.1/onboarding/task.json HTTP/11" 403 98
web    | Failed authentication. You might have entered the wrong username/password. Please rerun with environment variable DEBUG=1 for debugging.
web    | Exited with code 1
nginx  | Interrupting...
nginx  | Exited with code 0
KTachibanaM commented 2 weeks ago

Looks like the credentials are still wrong for some reason.. Can you try using the newly added multiple account approach?

sushione commented 2 weeks ago

Cool, much better now, it is working. Thanks !

2 questions :

KTachibanaM commented 2 weeks ago
  1. If you are following this guide then you have to use environment variables for customization as per this table. The goal was to simplify deployment so if you want a particular config to be customizable, I can make that change. If you are feeling adventurous, you could also just mount your nitter.conf to /src/nitter.conf in the container, but you have to comment out this line and build your own image locally, e.g. point your docker compose stack to use a local nitter repo with the aforementioned change.
  2. This is exposing the 8081 port in your container to 0.0.0.0:8080 to your host, so this is normal. If you want it only exposed on localhost, change 0.0.0.0 to 127.0.0.1
sushione commented 2 weeks ago

Great, we are almost there !

One last question : Is there a way to run instance without needing any authentification ? my RSS client can fetch full text tweet from web but it cannot authenticate.

If I try to disable NGINX authentification with DISABLE_NGINX=1 then nitter will not launch or be accessible. Leaving blank these fields, or #commenting, will also prevent nitter to work : INSTANCE_RSS_PASSWORD= INSTANCE_WEB_USERNAME= INSTANCE_WEB_PASSWORD= None of these settings work.

KTachibanaM commented 2 weeks ago

That's an oversight from me lol. You need to change this line to "0.0.0.0:8080:8080" so that the original nitter port 8080 (instead of nginx port 8081) is mapped to host 8080

sushione commented 1 week ago

Thank you very much ! with you advices I have now a working Nitter instance w/ authentication.

But still have issue fetching RSS feeds.. Can fetch feed one or two times and then get a Request failed: client error (409). Do you know if there is a setting to enable more fetches or any flood protection on nginx ?

KTachibanaM commented 1 week ago

If you've disabled NGINX there shouldn't be any NGINX flood protection, so it's likely a nitter issue. Do you have error logs from nitter container that looks suspicious?