search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] <Fogmeta Labs> - <Chainsnap> #1999

Open hengdingy opened 1 year ago

hengdingy commented 1 year ago

Data Owner Name

Fogmeta Labs

What is your role related to the dataset

Data Preparer

Data Owner Country/Region

China

Data Owner Industry

Web3 / Crypto

Website

https://fogmetalabs.com/chainsnap

Social Media

Twitter: https://twitter.com/FogMeta
GitHub: https://github.com/FogMeta

Total amount of DataCap being requested

15PiB

Weekly allocation of DataCap requested

1PiB

On-chain address for first allocation

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Share a brief history of your project and organization

Chainsnap is a project developed by Fogmeta Labs for maintaining blockchain snapshots. This project enables the storing and retrieval of snapshots for blockchain projects including Aptos, Avalanche, Binance Smart Chain, Ethereum, Filecoin,,Near, Pocket, and Polygon. Please refer to: https://github.com/FogMeta/chainsnap.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

These are public blockchain snapshots that are maintained by the network communities.
Please refer to the links presented below:
1. Aptos: https://bwarelabs.com/snapshots
2. Avalanche: http://186.233.187.26/
3. BSC: https://github.com/bnb-chain/bsc-snapshots
4. Ethereum: https://github.com/allada/eth-archive-snapshot
5. Filecoin: https://lotus.filecoin.io/lotus/install/start-lotus/
6. NEAR: https://near-nodes.io/intro/node-data-snapshots
7. Pocket: https://github.com/pokt-foundation/pocket-snapshotter
8. Polygon: https://snapshots.polygon.technology/

Where was the data currently stored in this dataset sourced from

Other

If you answered "Other" in the previous question, enter the details here

These snapshots are maintained by 1)the network teams, 2) community volunteers, 3) ourselves and are from several sources, such as AWS Cloud and our own storage infra.

How do you plan to prepare the dataset

IPFS, lotus, graphsplit, others/custom tool

If you answered "other/custom tool" in the previous question, enter the details here

We're going to use the `go-mc-sdk` to prepare the datasets. 
https://github.com/FogMeta/go-mc-sdk

Please share a sample of the data

1. Aptos
- https://snapshots.bwarelabs.com/aptos/mainnet/aptos20230517.tar.lz4 (133G)
- https://snapshots.bwarelabs.com/aptos/testnet/aptos20230517.tar.lz4 (179G)

2. Avalanche
- https://youravaxsnapshot.senseinode.com/avalanche_mainnet_20230507.tar (828G)

3. BSC
- https://pub-c0627345c16f47ab858c9469133073a8.r2.dev/geth-20230516.tar.lz4(1.9T)

4. Ethereum
- s3://public-blockchain-snapshots/eth

5. Filecoin
- https://snapshots.mainnet.filops.net/minimal/latest.zst(45.7G)

6. NEAR
- s3://near-protocol-public/backups/testnet/rpc/latest (469.2G)
- s3://near-protocol-public/backups/mainnet/rpc/latest (473.7G)
- s3://near-protocol-public/backups/testnet/archive/latest (5.2T)
- s3://near-protocol-public/backups/mainnet/archive/latest (9.0T)

7. Pocket
- https://snapshot.nodes.pokt.network/latest.tar (575G)

8. Polygon:
- https://snapshot-download.polygon.technology/bor-mainnet-incremental-compiled-files.txt (1467.43 GB)
- https://snapshot-download.polygon.technology/bor-mumbai-incremental-compiled-files.txt (250.05 GB)
- https://snapshot-download.polygon.technology/heimdall-mainnet-incremental-compiled-files.txt (218.07 GB)
- https://snapshot-download.polygon.technology/heimdall-mumbai-incremental-compiled-files.txt (34.40 GB)
- https://snapshot-download.polygon.technology/erigon-mainnet-archive-2023-04-21.tar.gz (2210.24 GB)
- https://snapshot-download.polygon.technology/erigon-mumbai-archive-2023-04-21.tar.gz (228.66 GB)

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Monthly

For how long do you plan to keep this dataset stored on Filecoin

2 to 3 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, Africa, North America, South America, Europe, Australia (continent), Antarctica

How will you be distributing your data to storage providers

HTTP or FTP server, IPFS, Shipping hard drives, Others

How do you plan to choose storage providers

Slack, Partners, Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

FilSwan platform (https://filswan.com/) is another good choice for us to choose storage providers who meet our requirements.

If you already have a list of storage providers to work with, fill out their names and provider IDs below

The storage providers we'd like to work with are presented below. Some of them are from the FilSwan platform.
f03624
f010088
f02301
f08399
f02401
f0187709
f01163272
f01402814
f01072221
f0240185
f0143858
f01390330
f01225882
f0717969
f03223
f01395673
f01786736
f0836160
f032824
f01443744
f01871352
f01907556
f01946551
f02012951
f01970630

How do you plan to make deals to your storage providers

Boost client, Lotus client, Others/custom tool

If you answered "Others/custom tool" in the previous question, enter the details here

https://github.com/FogMeta/go-mc-sdk

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

Sunnyiscoming commented 1 year ago

How much data does each of the following datasets have, and how many backups do you plan to store?

  1. Aptos: https://bwarelabs.com/snapshots
  2. Avalanche: http://186.233.187.26/
  3. BSC: https://github.com/bnb-chain/bsc-snapshots
  4. Ethereum: https://github.com/allada/eth-archive-snapshot
  5. Filecoin: https://lotus.filecoin.io/lotus/install/start-lotus/
  6. NEAR: https://near-nodes.io/intro/node-data-snapshots
  7. Pocket: https://github.com/pokt-foundation/pocket-snapshotter
  8. Polygon: https://snapshots.polygon.technology/

Do these storage providers offer retrieval services?

hengdingy commented 1 year ago

@Sunnyiscoming the snapshot size is as follows, Chainsnap will store a series of public chain snapshots (Including but not limited to the following chains), and every snapshot will be sent to 15 storage providers.

Our backup plan and deal metadata can be seen here. These storage providers are from filswan platform, which is required to offer retrieval services.

Chainsnap product introduction can be found https://fogmetalabs.com/chainsnap

  1. Aptos

  2. Avalanche

  3. BSC

  4. Ethereum

    • s3://public-blockchain-snapshots/eth

  5. Filecoin

  6. NEAR

    • s3://near-protocol-public/backups/testnet/rpc/latest (469.2G)
    • s3://near-protocol-public/backups/mainnet/rpc/latest (473.7G)
    • s3://near-protocol-public/backups/testnet/archive/latest (5.2T)
    • s3://near-protocol-public/backups/mainnet/archive/latest (9.0T)

  7. Pocket

  8. Polygon:

  9. SUI:

    • s3://mysten-mainnet-snapshots/ 294.0 GiB
    • s3://mysten-testnet-snapshots/ 1.1T
Sunnyiscoming commented 1 year ago

The data samples you provided is no more than 30 TB. And you want to apply for 15 PB datacap. How many copies will you store? Can you provide all the types of public chain and how much data does each of them have?

hengdingy commented 1 year ago

@Sunnyiscoming Chainsnap plans to store the public chains snapshot, including but not limited to the following chains:

Sunnyiscoming commented 1 year ago

Large datasets are open to existing datasets. May I ask what data sources do you have for existing datasets? What is the corresponding data amount? According to the information you provided, you only have 30 TB data at present, right? If so, it is recommended that you apply directly to notaries for Datacap.

Sunnyiscoming commented 1 year ago

Any update here?

hengdingy commented 1 year ago

@Sunnyiscoming Thanks for your review, Chainsnap will download the snapshots from the above link, which is maintained by the public chain team or ecosystem project team. We are downloading every chain snapshot by the two 10Gbps bandwidths(the 30TiB dataset will be downloaded every week). And we have maintained some chain snapshots, such as Filecoin, Polygon, BSC, Pocket and so on, we will export and store these snapshot every weeks. you can see the website: https://fogmetalabs.com/demo/All

More and more snapshot datasets will be downloaded, and hope you can support the Chainsnap project.

cryptowhizzard commented 1 year ago

This one looks ok. Waiting for the datacap trigger

Sunnyiscoming commented 1 year ago

Weekly allocation of DataCap requested 1PiB

the 30TiB dataset will be downloaded every week

There is a big difference between the two Numbers. Can you explain it?

hengdingy commented 1 year ago

@Sunnyiscoming the 30TiB dataset will be downloaded every week, we will store the dataset, Chainsnap will backup these snapshots every week and will store 10 copies for every snapshot. And every piece will be 16GiB(requires 32GiB datacap), so 30TiB*1024/16GiB*32GiB*10/1024= 600TiB/week. But in the first round, there is just 256TiB datacap will be allocated.

Chainsnap needs 2.4PiB datacap per month, 15PiB datacap is just can be used about 6 months.

Sunnyiscoming commented 1 year ago

Datacap Request Trigger

Total DataCap requested

15PiB

Expected weekly DataCap usage rate

1PiB

Client address

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

DataCap allocation requested

512TiB

Id

ba088081-eca2-49b8-8fe1-f4c500f4db54

nj-steve commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedcqflbjuvoonlqud2k5kskaqqw4hd6u5al5pijvqfbm7g7m4n6qw

Address

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

Datacap Allocated

512.00TiB

Signer Address

f1xx6555qijma7igpnjspyvdunc4vfxkawnpqy5ii

Id

ba088081-eca2-49b8-8fe1-f4c500f4db54

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedcqflbjuvoonlqud2k5kskaqqw4hd6u5al5pijvqfbm7g7m4n6qw

nj-steve commented 1 year ago

sounds good.I trust you on the first round

zcfil commented 1 year ago

I have reviewed historical information and am willing to provide support in the first round, and will follow up on the data situation in the later stage.

zcfil commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceboq3iokqrj2jg2khxevlpkonsf56jz5iur4hlgc2k5jvxrcmo72m

Address

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

Datacap Allocated

512.00TiB

Signer Address

f1cjzbiy5xd4ehera4wmbz63pd5ku4oo7g52cldga

Id

ba088081-eca2-49b8-8fe1-f4c500f4db54

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceboq3iokqrj2jg2khxevlpkonsf56jz5iur4hlgc2k5jvxrcmo72m

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

Normalnoise commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

hengdingy commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

Normalnoise commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

hengdingy commented 1 year ago

chainsnap has been renamed to chainnode

cryptowhizzard commented 1 year ago

Confirmed that this client is storing snapshot data.

LDN 1999 Client f02216069 SP f0466405 DealID 46933172 CID baga6ea4seaqk7myyjkzu32fokolo3xjalc4ubvkvlylpwkhndw36xehdvbnj4ea Filename : geth-20230708.tar.lz4.00000000

Normalnoise commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

Normalnoise commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

Normalnoise commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

Normalnoise commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

DataCap allocation requested

512TiB

Id

7e308ad3-ab05-46d9-9fd0-a537684b23b6

Normalnoise commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

kernelogic commented 1 year ago

Report LGTM. No dispute in the tracker either.

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacechw3qxcwbqygtvgmzzbp5cs6nyjvi2uup3j7jvqc2ueylqw3pqu2

Address

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

Datacap Allocated

512.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

7e308ad3-ab05-46d9-9fd0-a537684b23b6

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacechw3qxcwbqygtvgmzzbp5cs6nyjvi2uup3j7jvqc2ueylqw3pqu2

stcloudlisa commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecwme26pvbs4s3qmelg237rmzhodlgnpeqfvpft4p4vdsr3pw7evs

Address

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

Datacap Allocated

512.00TiB

Signer Address

f1jvvltduw35u6inn5tr4nfualyd42bh3vjtylgci

Id

7e308ad3-ab05-46d9-9fd0-a537684b23b6

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecwme26pvbs4s3qmelg237rmzhodlgnpeqfvpft4p4vdsr3pw7evs

hengdingy commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

Normalnoise commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

hengdingy commented 1 year ago

keep storing the snapshots

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.