尝试从 twitter 拉取 tweets 的数据

[iannono]

请尝试通过 twitter 的 api 实现用户 tweets 的拉取。 如果需要， 可以尝试下面的 key：

[0xRichardH]

https://developer.twitter.com/en/docs/twitter-api

https://api.twitter.com/2/openapi.json

https://www.postman.com/twitter/workspace/twitter-s-public-workspace/collection/9956214-784efcda-ed4c-4491-a4c0-a26470a67400?ctx=info&tab=pre-request-scripts

[0xRichardH]

https://developer.twitter.com/en/docs/authentication/oauth-2-0/bearer-tokens

curl -u "$API_KEY:$API_SECRET_KEY" \
  --data 'grant_type=client_credentials' \
  'https://api.twitter.com/oauth2/token'

[0xRichardH]

https://developer.twitter.com/en/docs/authentication/guides/v2-authentication-mapping

[0xRichardH]

[0xRichardH]

• option 1: 升级 twitter api 的 link https://developer.twitter.com/en/portal/petition/essential/basic-info?plan=basic

---

• option 2: 可以使用 https://github.com/vladkens/twscrape scrape twitter feed

https://github.com/brtr/twitter-gpt/issues/5#issuecomment-2076261972

---

• option 3 https://socialdata.gitbook.io/docs/twitter-lists/retrieve-list-details
  • https://socialdata.tools/

[0xRichardH]

用爬虫定期拉 3到5个人的 twitter feed scrape the tweet list

Using twscrape

import asyncio
import aiofiles
import datetime
from twscrape import API, gather
from twscrape.logger import set_log_level

async def main():
    api = API()  # or API("path-to.db") - default is `accounts.db`

    # ADD ACCOUNTS (for CLI usage see BELOW)
    await api.pool.add_account(
        "xxxxx@xxx.com", "supersecurepassword", "xxxxx@xxx.com", "mail_pass1"
    )
    await api.pool.login_all()

    # list info
    list_id = 1464100857402769409  # https://twitter.com/i/lists/1464100857402769409
    # await gather(api.list_timeline(list_id))

    current_date = datetime.datetime.now().strftime("%Y-%m-%d")
    filename = f"tweets_{list_id}_{current_date}.txt"
    async with aiofiles.open(filename, "a") as file:
        async for tweet in api.list_timeline(list_id, limit=100):
            await file.write(str(tweet.json()) + "\n")
            print(tweet.json())
            # print(tweet.id, tweet.user.username, tweet.rawContent)
            print("=====================================")

if __name__ == "__main__":
    asyncio.run(main())

tweets_1464100857402769409_2024-05-03.txt

---

Use https://socialdata.gitbook.io/docs/twitter-lists/retrieve-list-details

curl "https://api.socialdata.tools/twitter/lists/show?id=1464100857402769409" \
-H 'Authorization: Bearer API_KEY' \
-H 'Accept: application/json' \

[0xRichardH]

Tech Spec

Background

• 1. 想了解最近 tweets , 但是 tweets 的内容太多看不过来。（summeriaze)
• 2. 想通过提问的方式，知道关联 tweets 内容。如，bitcoin 最近怎么样？ (knowledge base)

Design

• Openai API
  • https://platform.openai.com/docs/guides/text-generation/chat-completions-api
  • https://platform.openai.com/docs/guides/embeddings
• Vector Database
  • qdrant https://github.com/qdrant/qdrant?tab=readme-ov-file

[0xRichardH]

定时下载: https://twitter.com/i/lists/1762857078656532875

• 每天跑一次

[fmb-chin]

ssh deploy@57.180.63.58

新主机

i-0adad346a9973642c, 57.180.63.58, 172.31.23.201

[0xRichardH]

DONE:

• GPT: https://chatgpt.com/g/g-UbwMG7Dxn-tweet-gpt
• Knowledge base: https://tweets-knowledge-base.tonob.net/