search

bluesky-social / atproto

Social networking technology created by Bluesky
Other
7.29k stars 515 forks source link

ATProto/api : Calling AppView's searchPosts with a cursor returns a 403 error. #2838

Open usounds opened 1 month ago

usounds commented 1 month ago

Describe the bug This issue occurs with AppView's API as follows: https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts?q=aaaaaa&limit=100&cursor=0 https://api.bsky.app/xrpc/app.bsky.feed.searchPosts?q=aaaaaa&limit=100&cursor=0

API return the following message. 403 Forbidden Request forbidden by administrative rules.

This issue does not occur when the API is called without a cursor. https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts?q=aaaaaa&limit=100 https://api.bsky.app/xrpc/app.bsky.feed.searchPosts?q=aaaaaa&limit=100

The PDS API works properly regardless of whether a cursor is used or not. https://lionsmane.us-east.host.bsky.network/xrpc/app.bsky.feed.searchPosts?q=aaaaaa%5C&limit=20&cursor=20&sort=latest https://lionsmane.us-east.host.bsky.network/xrpc/app.bsky.feed.searchPosts?q=aaaaaa%5C&limit=20&sort=latest

To Reproduce

Steps to reproduce the behavior:

1.Calling the AppView API's searchPosts with the cursor parameter. https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts?q=aaaaaa&limit=100&cursor=0

Expected behavior

The API returns results without any errors.

db3000 commented 1 week ago

I spent a bit of time trying to replicate this locally using this codebase to no avail.

The oddly specific formatting of the error page (compare the 403 error to say a 400 response from the API) makes me wonder if its being returned by something before the API. In particular, that format is exactly what haproxy outputs for a 403 error if you hit an ACL.

Is it possible this error is being returned by an instance of haproxy deployed in front of the API that is set to deny requests with the cursor query parameter set?

bnewbold commented 1 week ago

we have been having problems with folks trying to use search as a non-search API. Eg, to drive feed generators, or to keep up with all posts, instead of using the firehose or other more appropriate APIs for that use case. We have had to degrade some features, like cursors.

usounds commented 1 week ago

@bnewbold Thank you for your comment. Apologies; this may apply to me. Based on my understanding of your comment, it seems that using the app.bsky.feed.searchPosts API for feed generators is not recommended when accessing the authenticated bsky.social host. Would that be correct?

sabatale commented 1 week ago

Would someone be able to describe how to use cursors for searchPosts with the public API?

Or how to do it some other way without authentication?

afontenot commented 1 week ago

I want to mention here that this impacts not just the API but the UI if you're scrolling a search page while logged out. You get yanked back up to the top of the page and an error is shown.