CAFECA-IO / BAIFA-web-crawling

BAIFA conducts web crawling from iSunCloud in frequence
0 stars 0 forks source link

解析資料表 black_lists、chains、addresses #27

Closed gibbs-shih closed 7 months ago

gibbs-shih commented 8 months ago

document: BFAER00002

解析資料表 black_lists、chains、addresses

gibbs-shih commented 8 months ago

截圖 2024-01-09 下午4 49 31(2)

gibbs-shih commented 8 months ago
gibbs-shih commented 8 months ago
  1. 開立工作票更新前端畫面及設計稿的evidence的state(#460 #461)、transaction的type(#462 #463
  2. 回頭更新3個raw data資料(用JSON.stringify()後再存入raw data資料表), remote server 跟 local都要
    • block 截圖 2024-01-10 下午4 58 17(2)
gibbs-shih commented 8 months ago

建立新資料, 推schema至遠端資料庫 截圖 2024-01-10 下午5 42 07(2)拷貝

gibbs-shih commented 8 months ago

ERD圖更新:BFAER00002

codes資料表對應參照文件: DB Schema: codes

截圖 2024-01-11 下午2 26 23(2)

修改prisma.schema(local及remote), 並推上新schema至遠端資料庫 npx prisma db push --schema=./prisma/schema.prisma

gibbs-shih commented 7 months ago

調整 crawler start為每五秒執行一次, 並確保以同步方式執行 截圖 2024-01-15 上午11 17 38(2)

remote server 拉取parsing分支並進行爬蟲, 路徑位置: baifa@isunfa-crawler-001:/workspace/test/BAIFA-web-crawling$

gibbs-shih commented 7 months ago
gibbs-shih commented 7 months ago

截圖 2024-01-16 下午3 20 21(2)

gibbs-shih commented 7 months ago
gibbs-shih commented 7 months ago
gibbs-shih commented 7 months ago

issues #26-#30, each issue takes about 11-12 hrs on average