部署爬蟲到測試環境

[gibbs-shih]

推一版 crawler + parser 程式碼至遠端 server, 並建立pm2執行

[gibbs-shih]

• 於遠端建立 crawler_parser 專案資料夾, 從分支 feature/parsing 拉下來以 pm2 執行
• git clone -b 分支名稱 儲存庫網址
• 將遠端 main.ts 中 bootstrap() 裡面改成 Promise.all([crawler.start(), parser.start()])
• 建立 pm2 執行 pm2 start npm --name BAIFA-CRAWLER-PARSER -- start

[gibbs-shih]

• bug 發現addresses、contracts、currencies、token_balances、token_transfers 沒資料, transactions 筆數有少

  • 以 block number 80352 檢查, 發現可能是關聯設置的問題

• 先取消 public_tags 的 target 關連
  

[gibbs-shih]

• [x] addresses.address、review_datas.target 關聯修正 應為一對多
• [x] 新增 transactions.hash、review_datas.target 關聯 應為一對多
• [x] 修改關聯圖

  
  model addresses {
  id                  Int      @id @default(autoincrement())
  chain_id            Int? 
  created_timestamp   DateTime?
  address             String? @unique
  score               Int?
  latest_active_time  DateTime?
  risk_records        risk_records[]
  black_lists         black_lists?
  chains              chains? @relation(fields: [chain_id], references: [id])
  review_datas        review_datas[]    
  }

model review_datas { id Int @id @default(autoincrement()) target String? @unique target_type String? created_timestamp DateTime? content String? stars Int? author_address String? addresses addresses? @relation(fields:[target], references: [address], map: "review_datas_addresses_fkey") transactions transactions? @relation(fields:[target], references: [hash], map: "review_datas_transactions_fkey") }

model transactions { id Int @id @default(autoincrement()) chain_id Int? created_timestamp DateTime? hash String? @unique type String? status String? block_hash String? from_address String? to_address String? evidence_id String? value Int? fee BigInt? related_addresses String[] chains chains? @relation(fields: [chain_id], references: [id]) blocks blocks? @relation(fields: [block_hash], references: [hash]) review_datas review_datas[]
}

- [x] red_flags 新增欄位 currency_id, 及建立關聯至currencies id, 多對一
![CleanShot 2024-01-29 at 17 32 17](https://github.com/CAFECA-IO/BAIFA-web-crawling/assets/73210852/27c02a7b-7fa4-4b13-9728-5f4036b208ea)

model red_flags { id Int @id @default(autoincrement()) chain_id Int? created_timestamp DateTime? red_flag_type String? total_amount Int? symbol String? related_addresses String[] related_transactions String[] currency_id String? currencies currencies? @relation(fields: [currency_id], references: [id])
chains chains? @relation(fields: [chain_id], references: [id])
}

model currencies { id String @id risk_level String? price Int? volume_in_24h Int? symbol String? total_amount Int? holder_count Int? total_transfers Int? chain_id Int? name String? token_transfers token_transfers[] red_flags red_flags[] token_balances token_balances? chains chains? @relation(fields: [chain_id], references: [id])
}

- 修改bug：
![CleanShot 2024-01-29 at 16 33 13](https://github.com/CAFECA-IO/BAIFA-web-crawling/assets/73210852/061ef402-3187-4eba-855e-1ae997370a97)

- 相依性問題, 需調整function順序
![CleanShot 2024-01-29 at 17 52 15](https://github.com/CAFECA-IO/BAIFA-web-crawling/assets/73210852/14cecd0a-e40a-484c-bf90-3db5911b8bc8)

[gibbs-shih]

• 調整 toTransactions 裏面的 function 執行順序

  • parsedTransaction > (ERC 20 transfer / normal transfer 才會有後面的) toCurrencies > toTokenTransfers > toTokenBalances

• 推一版至遠端伺服器, 並建立pm2執行

• 更新 review_datas：target_type 對應欄位參照

[gibbs-shih]

• 發現 transactions、token_balances、token_transfers table value 欄位格式有誤, 造成有些交易解析失敗, 故修正及更新schema

• 更新 schema 中所有解析資料表 value、fee 欄位資料格式為 string （下圖紅線）

  • blocks、transactions、red_flags、currencies、token_transfers、token_balances
  • 更新 關聯圖
  • 更新 db schema 文件
  • 更新 schema.prisma
  • 更新程式碼

• 更新 schema 中所有解析資料表 timestamp 欄位資料格式為 number （下圖綠線）

  • contracts、evidences、transactions、black_lists、review_datas、red_flags、blocks、addresses、risk_records
  • 更新 關聯圖
  • 更新 db schema 文件
  • 更新 schema.prisma
  • 更新程式碼

model contracts {
  id                  Int      @id @default(autoincrement())
  chain_id            Int?
  contract_address    String? 
  source_code         String?
  creator_address     String?
  created_timestamp   Int?
  chains              chains? @relation(fields: [chain_id], references: [id])     
}

model evidences {
  id                  Int      @id @default(autoincrement())
  evidence_id         String?      
  chain_id            Int? 
  created_timestamp   Int?
  contract_address    String?
  state               String?
  content             String?
  creator_address     String?
  token_id            String?
  chains              chains? @relation(fields: [chain_id], references: [id])
}

model transactions {
  id                  Int      @id @default(autoincrement())
  chain_id            Int? 
  created_timestamp   Int?
  hash                String? @unique
  type                String?
  status              String?
  block_hash          String? 
  from_address        String?
  to_address          String?
  evidence_id         String?
  value               String?
  fee                 String?
  related_addresses   String[]
  chains              chains? @relation(fields: [chain_id], references: [id])
  blocks              blocks? @relation(fields: [block_hash], references: [hash])
  review_datas        review_datas[]       
}

model black_lists {
  id                  Int      @id @default(autoincrement())
  chain_id            Int? 
  created_timestamp   Int?
  address_id          Int? @unique
  public_tag          String?
  chains              chains? @relation(fields: [chain_id], references: [id])
  addresses           addresses? @relation(fields: [address_id], references: [id])              
}

model review_datas {
  id                  Int      @id @default(autoincrement())
  target              String? @unique
  target_type         String?
  created_timestamp   Int?
  content             String?
  stars               Int?
  author_address      String?
  addresses           addresses? @relation(fields:[target], references: [address], map: "review_datas_addresses_fkey")
  transactions        transactions? @relation(fields:[target], references: [hash], map: "review_datas_transactions_fkey")
}

model red_flags {
  id                  Int      @id @default(autoincrement())
  chain_id            Int? 
  created_timestamp   Int?
  red_flag_type       String?
  total_amount        String?
  symbol              String?
  related_addresses   String[]
  related_transactions String[]
  currency_id         String?
  currencies          currencies? @relation(fields: [currency_id], references: [id])                
  chains               chains? @relation(fields: [chain_id], references: [id])                
}

model blocks {
  id                  Int      @id @default(autoincrement())
  chain_id            Int? 
  created_timestamp   Int?
  miner               String?
  reward              String?
  size                Int?
  symbol              String?
  transaction_count   Int?
  parent_hash         String?
  burnt_fees          String?   
  hash                String? @unique
  number              Int? @unique
  transactions        transactions[]
  chains              chains? @relation(fields: [chain_id], references: [id])     
}

model token_balances {
  id                  Int      @id @default(autoincrement())
  address             String?
  value               String?
  chain_id            Int?
  currency_id         String? @unique
  chains              chains? @relation(fields: [chain_id], references: [id])
  currencies          currencies? @relation(fields: [currency_id], references: [id])  
}

model token_transfers {
  id                  Int      @id @default(autoincrement())
  from_address        String?
  to_address          String?
  value               String?
  chain_id            Int?
  currency_id         String? 
  transaction_hash    String?
  index               Int?
  chains              chains? @relation(fields: [chain_id], references: [id])
  currencies          currencies? @relation(fields: [currency_id], references: [id])      
}

model currencies {
  id                  String      @id 
  risk_level          String?
  price               Int?
  volume_in_24h       String?
  symbol              String?
  total_amount        String?
  holder_count        Int?
  total_transfers     Int?
  chain_id            Int?
  name                String?
  token_transfers     token_transfers[]
  red_flags           red_flags[]
  token_balances      token_balances?
  chains              chains? @relation(fields: [chain_id], references: [id])           
}

model addresses {
  id                  Int      @id @default(autoincrement())
  chain_id            Int? 
  created_timestamp   Int?
  address             String? @unique
  score               Int?
  latest_active_time  Int?
  risk_records        risk_records[]
  black_lists         black_lists?
  chains              chains? @relation(fields: [chain_id], references: [id])
  review_datas        review_datas[]    
}

model risk_records {
  id                  Int      @id @default(autoincrement())
  name                String?
  address_id          Int? 
  created_timestamp   Int?
  addresses           addresses? @relation(fields: [address_id], references: [id])              
}

• 推新測試版至遠端並建立pm2

[gibbs-shih]

• 解bug, 有四筆交易未解析完成, block number (81518, 678)

  

  • 原因可能是 token_balances currency_id 為 unique 造成 prisma create 失敗, 故修正資料表關聯

    • currencies、token_balances
    • 更新 關聯圖
    • 更新 schema.prisma
      
    • block number 678 測試成功

  • block number 678 + 81518 測試成功

    

[gibbs-shih]

• 建新 pm2 執行

[gibbs-shih]

• 至 block 291949, 目前blocks、transactions 無遺漏解析

[gibbs-shih]

• 根據有的 token_transfers, token_balances 值計算正確

[gibbs-shih]

• 新推一版 pm2 重新執行 test20240202
  • 更新程式碼, 排除 token transfers 及 token balances value 為 0 的交易
  • 更新程式碼, value 為 0 的交易不計算在 currencies 的 total_transfers 中
  • token_transfers 新增 created_timestamp

[gibbs-shih]

• 初步解析完成
• 屬於 erc20、normal 的交易共14筆

[gibbs-shih]

takes about 11-12 hrs