search

joshua-ne / FIL_DC_Allocator_1022

This repo serves as a bookkeeping tool for DC allocation from Allocator_1022
0 stars 0 forks source link

[DataCap Application] <ZEN OPENDATA> - <ZOD-USGOD-1> #29

Open Dengminer opened 3 months ago

Dengminer commented 3 months ago

Version

1

DataCap Applicant

ZEN OPENDATA

Project ID

ZOD-USGOD-1

Data Owner Name

U.S. General Services Administration

Data Owner Country/Region

United States

Data Owner Industry

Government

Website

https://www.gsa.gov/

Social Media Handle

X: https://x.com/USGSA

Social Media Type

Twitter

What is your role related to the dataset

Data Preparer

Total amount of DataCap being requested

4.5 PiB

Expected size of single dataset (one copy)

900 TiB

Number of replicas to store

5

Weekly allocation of DataCap requested

600 TiB

On-chain address for first allocation

f1qtxjoxzobjmcxn3aaxn7tyj7e3p24r3d2bw3ygy

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Identifier

No response

Share a brief history of your project and organization

Zen OpenData is dedicated to building an open data ecosystem, focusing on the collection, sharing, and management of data. Our organization aims to foster collaboration, transparency, and innovation by providing access to high-quality datasets. Through our initiatives, Zen OpenData empowers individuals, researchers, and organizations to leverage data for informed decision-making and problem-solving, driving progress across various sectors.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

U.S. Government's Open Data is built by the United States Government and is designed to unleash the power of government open data to inform decisions by the public and policymakers, drive innovation and economic activity, achieve agency missions, and strengthen the foundation of an open and transparent government.

Where was the data currently stored in this dataset sourced from

My Own Storage Infra

If you answered "Other" in the previous question, enter the details here

No response

If you are a data preparer. What is your location (Country/Region)

Singapore

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

We have customized tools to retrieve and convert data to car files.

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

No response

Please share a sample of the data

http://www.hud.gov/offices/cpd/systems/census/sf1/Tables.zip

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

2 to 3 years

In which geographies do you plan on making storage deals

Asia other than Greater China

How will you be distributing your data to storage providers

Cloud storage (i.e. S3), HTTP or FTP server, IPFS, Shipping hard drives

How did you find your storage providers

Slack

If you answered "Others" in the previous question, what is the tool or platform you used

No response

Please list the provider IDs and location of the storage providers you will be working with.

f01422327, Japan
f02252023, Japan
f02252024, Japan
f01111110, Vietnam
f01909705, Vietnam
f03224283, Japan
f03224551, Japan
f03230392, Vietnam
f03230423, Malaysia
f03232064, Malaysia
f03232134, Malaysia

How do you plan to make deals to your storage providers

Boost client

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

joshua-ne commented 4 weeks ago

checker:manualTrigger

joshua-ne commented 4 weeks ago

trigger:run_retrieval_test method=lassie sp_list=f01422327,f02252023,f02252024,f01111110,f01909705,f03224283,f03224551 client=f1qtxjoxzobjmcxn3aaxn7tyj7e3p24r3d2bw3ygy limit=10 start_datetime="2024-08-01 00:00:00" end_datetime="2024-10-29 00:00:00"

myfil512 commented 4 weeks ago
miner_id retrieval_rate retrieval_success_counts retrieval_fail_counts
f01422327 80% 8 2
f02252023 80% 8 2
f02252024 80% 8 2
f01111110 100% 10 0
f01909705 0% 0 10
f03224283 100% 10 0
f03224551 100% 10 0
joshua-ne commented 4 weeks ago

The official bot is not currently working. But from my end, the retrieval looks good. I will go on to support one more time.

joshua-ne commented 4 weeks ago

But meanwhile, please keep up your retrieval rate and contact the operator of f01909705 to fix their retrieval problem. Keep me in the loop.

datacap-bot[bot] commented 4 weeks ago

Application is in Refill

datacap-bot[bot] commented 4 weeks ago

Last pending allocation reverted for an application f1qtxjoxzobjmcxn3aaxn7tyj7e3p24r3d2bw3ygy.

datacap-bot[bot] commented 4 weeks ago

Application is in Refill

datacap-bot[bot] commented 4 weeks ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacectnvqfd7bgsdx65utnqxu5typormx6fdgqcszg5lrvwy47b7xjrw

Address

f1qtxjoxzobjmcxn3aaxn7tyj7e3p24r3d2bw3ygy

Datacap Allocated

1PiB

Signer Address

f1sfffys4o2w64rdpd3alpmvpvj4ik6x2iyjsjmry

Id

0cc92f50-28b2-46b7-b06a-9413a3b1ea6d

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacectnvqfd7bgsdx65utnqxu5typormx6fdgqcszg5lrvwy47b7xjrw

datacap-bot[bot] commented 4 weeks ago

Application is Granted

Dengminer commented 4 weeks ago

Hi, since we have started to onboard more data now, and some of the SPs we previously worked with are no longer sealing, we now apply to modify the SP list we are working with as below:

new SPs: f03230392, Vietnam f03230423, Malaysia f03232064, Malaysia f03232134, Malaysia

SPs no longer work with: f03068013, Hongkong f03035686, Guangdong f01926635, Sichuang f01999119, Sichuang f03136267, Hongkong

I have updated the application form accordingly. Thank you for your support and please let me know if you have any questions.

And we will do our best to ensure the public retrievability.

datacap-bot[bot] commented 4 weeks ago

Issue has been modified. Changes below:

(OLD vs NEW)

Please list the provider IDs and location of the storage providers you will be working with: f01422327, Japan f02252023, Japan f02252024, Japan f01111110, Vietnam f01909705, Vietnam f03224283, Japan f03224551, Japan f03230392, Vietnam f03230423, Malaysia f03232064, Malaysia f03232134, Malaysia vs f03068013, Hongkong f03035686, Guangdong f01926635, Sichuang f01999119, Sichuang f03136267, Hongkong f01422327, Japan f02252023, Japan f02252024, Japan f01111110, Vietnam f01909705, Vietnam f03224283, Japan f03224551, Japan State: ChangesRequested vs Granted

datacap-bot[bot] commented 4 weeks ago

Issue information change request has been approved.

joshua-ne commented 4 weeks ago

Hi @Dengminer I have approved your request to add/remove some of the SP. As I notice, some of the new SPs are quite new. I will keep an extra eye on those and make sure they support retrieval.

datacap-bot[bot] commented 3 weeks ago

Client used 75% of the allocated DataCap. Consider allocating next tranche.

joshua-ne commented 3 weeks ago

trigger:run_retrieval_test method=lassie sp_list=f01422327,f02252023,f02252024,f01111110,f01909705,f03230392,f03230423,f03232064,f03232134 client=f1qtxjoxzobjmcxn3aaxn7tyj7e3p24r3d2bw3ygy limit=10 start_datetime="2024-08-01 00:00:00" end_datetime="2024-10-29 00:00:00"

myfil512 commented 3 weeks ago
miner_id retrieval_rate retrieval_success_counts retrieval_fail_counts
f01422327 80% 8 2
f02252023 80% 8 2
f02252024 80% 8 2
f01111110 100% 10 0
f01909705 0% 0 10
f03230392 NA 0 0
f03230423 NA 0 0
f03232064 NA 0 0
f03232134 NA 0 0
joshua-ne commented 3 weeks ago

trigger:run_retrieval_test method=lassie sp_list=f01422327,f02252023,f02252024,f01111110,f01909705,f03230392,f03230423,f03232064,f03232134 client=f1qtxjoxzobjmcxn3aaxn7tyj7e3p24r3d2bw3ygy limit=10 start_datetime="2024-08-01 00:00:00" end_datetime="2024-11-01 00:00:00"

myfil512 commented 3 weeks ago
miner_id retrieval_rate retrieval_success_counts retrieval_fail_counts
f01422327 80% 8 2
f02252023 80% 8 2
f02252024 80% 8 2
f01111110 100% 10 0
f01909705 0% 0 10
f03230392 80% 8 2
f03230423 30% 3 7
f03232064 100% 10 0
f03232134 100% 10 0
joshua-ne commented 3 weeks ago

@Dengminer The retrieval rate of the newly added SPs looks good to me, though f03230423 needs improvement. And f01909705 still needs repair. Again, keep me in the loop on how things going. I will go ahead and sign the last batch of Datacap as requested in the application.

joshua-ne commented 3 weeks ago

It is interesting that the first 3 SPs always show a retrieval rate of 80%, which is a little weird to me. Do they randomly delete 20% of their unsealed files? Will dig into this with some additional tests. And it would be nice if you could contact the SPs and set up a meeting to discuss this, if it is not done by the SPs intentionally. @Dengminer

datacap-bot[bot] commented 3 weeks ago

Application is in Refill

datacap-bot[bot] commented 3 weeks ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebujhspzmc6sdvyfg5jd23d7vpiqn6anmox6nzwfvv2xgbtkqbrkk

Address

f1qtxjoxzobjmcxn3aaxn7tyj7e3p24r3d2bw3ygy

Datacap Allocated

2PiB

Signer Address

f1sfffys4o2w64rdpd3alpmvpvj4ik6x2iyjsjmry

Id

0fa7442b-4e59-489a-a7a5-6ac0f9cdc4a8

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebujhspzmc6sdvyfg5jd23d7vpiqn6anmox6nzwfvv2xgbtkqbrkk

datacap-bot[bot] commented 3 weeks ago

Application is Granted

datacap-bot[bot] commented 1 week ago

Application is Completed

joshua-ne commented 1 week ago

Hi can you please submit your dataset card by the end of next week. Thank you. This will be mandatory for the following allocation of datacap. It would be appreciated even though you have finished this application.

https://github.com/joshua-ne/FIL_DC_Allocator_1022_Dataset_Card/issues/new/choose