Spike: Twitter advanced search documentation updates

lacyg4 commented 1 month ago

Problem

Users of the Twitter endpoint want more granular search capabilities. Our API supports advanced search this but its not documented. Advanced search queries can be constructed here: https://x.com/search-advanced?lang=en

We currently do not document examples of advanced search in the swagger documentation or developer documentation so people understand the full functionality of the twitter search API.

Here are some examples of advanced search queries:

curl -X 'POST' \
  'http://localhost:8080/api/v1/data/twitter/tweets/recent' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "query": "Masa Moon (from:getmasafi) until:2022-01-01 since:2006-01-01",
  "count": 10
}'

curl -X 'POST' \
  'http://localhost:8080/api/v1/data/twitter/tweets/recent' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "query": "(Masa OR CoinList)(from:getmasafi) until:2022-01-01 since:2006-01-01",
  "count": 10
}'

These developer documentation can be found here: https://help.x.com/en/using-x/x-advanced-search

Customer feedback

https://masafinance.slack.com/archives/C078JCDS81E/p1719870998918199 https://masafinance.slack.com/archives/C073K1YP8CR/p1717494858189539

Acceptance criteria:

Documentation Structure:
- [ ] Create a single, dedicated "Advanced Search" section in the API documentation
- [ ] Implement a dropdown or tabbed interface for users to select different search types TBD
Search Type Documentation:
- [ ] For each search type, provide:
  - [ ] A brief description of the search type
  - [ ] The correct syntax or format to use
  - [ ] At least one clear example of usage
  - [ ] Any relevant limitations or special considerations
Testing:
- [ ] Conduct user testing to ensure the documentation is easy to understand and use
- [ ] Verify all examples are correct and functional

Advance search to support:

[ ] Exact phrase search
[ ] Hashtag search
[ ] Mention search (@ symbol)
[ ] From user search
[ ] To user search
[ ] Language-specific search
[ ] Date range search
[ ] Retweets filtering
[x] Minimum likes filter
[x] Minimum retweets filter
[x] Keyword exclusion (NOT operator)
[x] OR operator for multiple terms
[x] Geo-location based search
[x] URL inclusion search
[x] Question tweets filter
[x] Safe search mode

Possible solutions

Document each case in swagger as an example (expensive)
Document each search query and have a couple of examples in dev docs

teslashibe commented 1 month ago

We may want to think about renaming the /data/twitter/tweets/recent endpoint to /data/twitter/tweets

giovaroma commented 1 month ago

@teslashibe updated the advance search option from Twitter and the acceptance criteria, please review the list of advance search options and let me know if this looks correct or we should remove some.

teslashibe commented 1 month ago

@giovaroma this looks fantastic - let move forward with this.

I have added this to a release in the roadmap here: https://github.com/masa-finance/roadmap/issues/2 (cc @mudler @H34D @Luka-Loncar)

I changed this to "In review" on product board added this to no-status in the protocol board and it can be groomed on Monday

giovaroma commented 1 month ago

Wonderful, moving tickets down the pipeline @teslashibe