search

bnb-chain / greenfield-cmd

support cmd tool for Greenfield
GNU Lesser General Public License v3.0
27 stars 12 forks source link

feat: update get object and client for speed & batch upload #108

Closed annielz closed 7 months ago

annielz commented 7 months ago

Description

Get Object mainly makes 2 update to spped up get object:

  1. remove unnecessary head object
  2. add sp endpoint as an option for get object, which can skip refresh SPs and get SP host steps.

Batch Upload update: Supports batch file upload

Rationale

Get Object: To make the user have better download experience via gnfd-cmd.

For 50K file download, the speed on normal computer change from ~4s to ~2s (1.6s for files in AP region and 2.4s for files in US region), speed up around 100%. If use EC2 machine, download 50K file in same region can speed up to ~0.8s.

For large files e.g. 500MB download, the download speed also decrease ~2s. Since we optimize the steps before download but not download it self.

Batch Upload: Recursively uploading files requires management/viewing progress, as well as retry capabilities

Example

Get Object: ./gnfd-cmd --config config.toml --passwordfile password.txt object get --spEndpoint https://gnfd-testnet-sp3.nodereal.io gnfd://annie-ap-sp3/flower1.jpeg

Batch Upload: ./gnfd-cmd --home ./ bucket create gnfd://constwz5 ./gnfd-cmd --home ./ object put --recursive ~/Desktop/t-folder gnfd://constwz5 .gnfd-cmd --home ./ task status --taskId 193977d7-93b1-48d3-ad83-cf2faf9feeb2 ./gnfd-cmd --home ./ task retry --taskId 735b85cd-251c-4f73-a81e-7c60da2bf3fd ./gnfd-cmd --home ./ task delete --taskId 034b5786-bb62-4620-a145-c2de9a1e70f6

Changes

Notable changes: