missuo / missuo.github.io

A repository for writing blogs
https://missuo.me
1 stars 0 forks source link

Create query API for 500 million Weibo data | Vincent's Notes #28

Open missuo opened 1 year ago

missuo commented 1 year ago

https://missuo.me/posts/weibo-data-searcher/

Preparation Clickhouse Database Linux/Windows/macOS Server Data Import First you need to deploy Clickhouse and finish creating a brand new database. Create two tables. Table For Query Weibo Uid. CREATE TABLE wb_uid( uid String, mobile String )ENGINE = MergeTree() ORDER BY (uid) PRIMARY KEY (uid); Table For Query Mobile Phone Number CREATE TABLE wb_mobile( uid String, mobile String )ENGINE = MergeTree() ORDER BY (mobile) PRIMARY KEY (mobile); Query Data WeiboSearcher is an API program written in Golang.