[DataCap Application] End-Use Load Profiles for the U.S. Building Stock

quxxMel commented 1 month ago

Version

1

DataCap Applicant

quxxMel

Project ID

1

Data Owner Name

National Renewable Energy Laboratory

Data Owner Country/Region

United States

Data Owner Industry

Environment

Website

https://www.nrel.gov/buildings/end-use-load-profiles.html

Social Media Handle

Marcus.Bianchi@nrel.gov

Social Media Type

Other

What is your role related to the dataset

Dataset Preparer

Total amount of DataCap being requested

2PiB

Expected size of single dataset (one copy)

216TiB

Number of replicas to store

10

Weekly allocation of DataCap requested

256TiB

On-chain address for first allocation

f12avrdgtgb2lqdruxsno3qdksz7ow5b25gc5if7q

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

[ ] Use Custom Multisig

Identifier

No response

Share a brief history of your project and organization

From breakthroughs in fundamental science to new clean technologies to integrated energy systems that power our lives, NREL researchers are transforming the way the nation and the world use energy.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

The U.S. Department of Energy (DOE) funded a three-year project, End-Use Load Profiles for the U.S. Building Stock, that culminated in this publicly available dataset of calibrated and validated 15-minute resolution load profiles for all major residential and commercial building types and end uses, across all climate regions in the United States. These EULPs were created by calibrating the ResStock and ComStock physics-based building stock models using many different measured datasets, as described here. This dataset includes load profiles for both the baseline building stock and the building stock with various energy efficiency, electrification, and demand flexibility upgrades applied.

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

No response

If you are a data preparer. What is your location (Country/Region)

Hong Kong

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

Tool: servers, lotus
Lotus is the primary tool we use to transform data. We have 3 servers to support us dealing with dataset. We also support transfer data to sps online/offline.

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

No

Please share a sample of the data

aws s3 ls --no-sign-request s3://oedi-data-lake/nrel-pds-building-stock/

Confirm that this is a public dataset that can be retrieved by anyone on the Network

[X] I confirm

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Sporadic

For how long do you plan to keep this dataset stored on Filecoin

More than 3 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, North America, South America

How will you be distributing your data to storage providers

HTTP or FTP server, Shipping hard drives, Lotus built-in data transfer

How did you find your storage providers

Slack, Filmine, Big Data Exchange

If you answered "Others" in the previous question, what is the tool or platform you used

No response

Please list the provider IDs and location of the storage providers you will be working with.

f03233966 Zhejiang
f03228358 Zhejiang
f03216485 Sichuan
f03100009 Guangdong
f01084941 Hongkong

How do you plan to make deals to your storage providers

Lotus client

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

datacap-bot[bot] commented 1 month ago

Application is waiting for allocator review

quxxMel commented 1 month ago

@Destore2023 Hi allocator, can you sign my application?

Destore2023 commented 3 weeks ago

Please fill in this form to complete your information. https://www.wenjuan.com/s/qAVFfuN/

quxxMel commented 3 weeks ago

@Destore2023 Hello, I have filled the form.

Destore2023 commented 3 weeks ago

@quxxMel Got your form!

Seems to be the first time this dataset has been stored in Filecoin network. Welcome to store data in filecoin.

When you want to use datacap to store data, you'd better to select sps in different regions. Also, please make your sps support retrieval! We will check it from time to time.

quxxMel commented 3 weeks ago

@Destore2023 Thank you. We'll do as you say.

Destore2023 commented 3 weeks ago

@quxxMel ok. Will give you the first round support.

datacap-bot[bot] commented 3 weeks ago

Datacap Request Trigger

Total DataCap requested

2PiB

Expected weekly DataCap usage rate

256TiB

DataCap Amount - First Tranche

256TiB

Client address

f12avrdgtgb2lqdruxsno3qdksz7ow5b25gc5if7q

datacap-bot[bot] commented 3 weeks ago

DataCap Allocation requested

Multisig Notary address

Client address

f12avrdgtgb2lqdruxsno3qdksz7ow5b25gc5if7q

DataCap allocation requested

256TiB

Id

1f99cabe-822d-4509-a4ef-db0f70ade363

datacap-bot[bot] commented 3 weeks ago

Application is ready to sign

datacap-bot[bot] commented 3 weeks ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebx2os4dzwmwvpdqsxkogclby7rmd7nlbfgybj5qb6okjdjp2khgc

Address

f12avrdgtgb2lqdruxsno3qdksz7ow5b25gc5if7q

Datacap Allocated

256TiB

Signer Address

f1rppfznglfb7uyn3k6sfzeh47yq54ptvia5ixwsq

Id

1f99cabe-822d-4509-a4ef-db0f70ade363

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebx2os4dzwmwvpdqsxkogclby7rmd7nlbfgybj5qb6okjdjp2khgc

datacap-bot[bot] commented 3 weeks ago

Application is Granted

datacap-bot[bot] commented 2 weeks ago

Client used 75% of the allocated DataCap. Consider allocating next tranche.

Destore2023 commented 2 weeks ago

checker:manualTrigger

datacap-bot[bot] commented 2 weeks ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

⚠️ 1 storage providers sealed too much duplicate data - f03233966: 22.78%

⚠️ 66.67% of Storage Providers have retrieval success rate equal to zero.

⚠️ 100.00% of Storage Providers have retrieval success rate less than 75%.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report.

Destore2023 commented 2 weeks ago

@quxxMel Please check your sps with 0 retrieval success rate. Also you should focus on data distributing.

quxxMel commented 2 weeks ago

@Destore2023 Hello allocator, sp gave feedback that they met this problem. announce message to https://cid.contact/ingest/announce: 429 Too Many Requests: Too Many Requests They are checking about it.

Destore2023 commented 2 weeks ago

@quxxMel ok. Look forward to your improvements.

datacap-bot[bot] commented 2 weeks ago

Application is in Refill

datacap-bot[bot] commented 2 weeks ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecyuf65oogoxmdqudnr7ngtxftdx2kbdozr5q5h2jupz3j3f2yebq

Address

f12avrdgtgb2lqdruxsno3qdksz7ow5b25gc5if7q

Datacap Allocated

1PiB

Signer Address

f1rppfznglfb7uyn3k6sfzeh47yq54ptvia5ixwsq

Id

d564de1e-8c9f-4b02-b437-283d3fd99a0c

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecyuf65oogoxmdqudnr7ngtxftdx2kbdozr5q5h2jupz3j3f2yebq

datacap-bot[bot] commented 2 weeks ago

Application is Granted

datacap-bot[bot] commented 1 week ago

Client used 75% of the allocated DataCap. Consider allocating next tranche.

Destore2023 commented 1 week ago

@quxxMel This is the response from spark team. https://github.com/filecoin-station/spark/issues/100 The current condition is that the spark team made a limit and your request was too fast, resulting in a 429 warning. You can try to pay attention to control the speed.

@Destore2023 Hello allocator, sp gave feedback that they met this problem. announce message to https://cid.contact/ingest/announce: 429 Too Many Requests: Too Many Requests They are checking about it.

quxxMel commented 1 week ago

@Destore2023 Thank you very much! We invite more sps to join our storage. Please check it. f01660795 Guangdong f03192503 HongKong f03214920 HongKong f03066836 Hunan f03231666 Sichuan

Destore2023 commented 1 week ago

checker:manualTrigger

datacap-bot[bot] commented 1 week ago

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

⚠️ 8 storage providers sealed too much duplicate data - f03100009: 23.01%, f03216485: 30.72%, f03233966: 22.82%, f03228358: 42.38%, f01660795: 22.75%, f03066836: 20.94%, f03214920: 24.05%, f03192503: 26.32%

⚠️ 11.11% of Storage Providers have retrieval success rate equal to zero.

⚠️ 55.56% of Storage Providers have retrieval success rate less than 75%.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report.

Destore2023 commented 1 week ago

@quxxMel Improvement in comparison with the last retrieval result. Glad to see your improvement on retrieval.

Last time This time

But can you explain why sps sealed too much duplicate data?

quxxMel commented 1 week ago

@Destore2023 Hello allocator, can you help us check the report? The amount of total deals sealed in report is more than datacap we received.

quxxMel commented 1 week ago

For example As shown in the picture, the total amount of datacap got by f03228358 is 217703302299648 bytes (198TiB).

But it shows that this sp has got 339.63TiB.

Please help us about checking it. Thank you!

Destore2023 commented 1 week ago

@quxxMel Hello, thanks for your words. There's some problem on the cid report about duplicate data. Someone has submitted this issue, and I will focus on the situation. Please don't worry about it.

Destore2023 / MetaPathways-Bookkeeping