search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] Transiting Exoplanet Survey Satellite (TESS) #2054

Closed jevticgallonsd1466 closed 7 months ago

jevticgallonsd1466 commented 1 year ago

Data Owner Name

Space Telescope Science Institute.

What is your role related to the dataset

Storage provider filling out application on behalf of the data owner

Data Owner Country/Region

United States

Data Owner Industry

Not-for-Profit

Website

http://astroquery.readthedocs.io/en/latest/mast/mast.html

Social Media

N/A

Total amount of DataCap being requested

7PiB

Expected size of single dataset (one copy)

750TiB

Number of replicas to store

10

Weekly allocation of DataCap requested

400TiB

On-chain address for first allocation

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Identifier

No response

Share a brief history of your project and organization

We are topblocks team - https://www.topblocks.io

We are leader sp on https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1248.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

The Transiting Exoplanet Survey Satellite (TESS) is a multi-year survey that will discover exoplanets in orbit around bright stars across the entire sky using high-precision photometry. The survey will also enable a wide variety of stellar astrophysics, solar system science, and extragalactic variability studies. More information about TESS is available at MAST and the TESS Science Support Center.

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

No response

How do you plan to prepare the dataset

singularity

If you answered "other/custom tool" in the previous question, enter the details here

No response

Please share a sample of the data

aws s3 ls --no-sign-request s3://stpubdata/tess/

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Sporadic

For how long do you plan to keep this dataset stored on Filecoin

Less than 1 year

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, North America, Europe

How will you be distributing your data to storage providers

HTTP or FTP server

How do you plan to choose storage providers

Slack, Partners

If you answered "Others" in the previous question, what is the tool or platform you plan to use

No response

If you already have a list of storage providers to work with, fill out their names and provider IDs below

|minerId|region|org|
|--|--|--|
|f02145020|CN|hongbiao|
|f01969779|US|Nick|
|f020522|DE|phantom|
|f02201190|US|James|
|f02032191|CN|treal|
|f01923786|CN|zcLabs|
|f01938671|CN|zcLabs|

My nodes are f02301,f03223,f0240185,f0143858 in US

How do you plan to make deals to your storage providers

Singularity

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!
large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

jevticgallonsd1466 commented 1 year ago

@Sunnyiscoming We have applied for an ldn before by michellilamiebqaj, but cannot access, I don't nkown why. I found that there was a problem with that account, so I reapplied with this account

think you

image

image

Sunnyiscoming commented 1 year ago

40% is a very large percentage, each sp should not have more than 30% of Datacap, so I can't move forward with this application。

jevticgallonsd1466 commented 1 year ago

40% is a very large percentage, each sp should not have more than 30% of Datacap, so I can't move forward with this application。

@Sunnyiscoming Thank you for your correction, this is our mistake. I promise we will not store more than 30% of Datacap to our nodes.

Sunnyiscoming commented 1 year ago

Datacap Request Trigger

Total DataCap requested

7PiB

Expected weekly DataCap usage rate

400TiB

Client address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

DataCap allocation requested

200TiB

Id

79e1c373-7fe4-4a13-a113-c2d6d3a8d38c

Aaron01230 commented 1 year ago

The client contact we and we had made some survey, according to the above records , we think this application meet the requirements of FIL Plus and willing to support this round and keep focusing.

Aaron01230 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceapu3pccvujexw7h3xtzsdaze7ouvjitxy7axenswyfd4etrpng5q

Address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

Datacap Allocated

200.00TiB

Signer Address

f1xrnysd4gimg64d4l6qi7ulzwwq22c6vfg6lpw3i

Id

79e1c373-7fe4-4a13-a113-c2d6d3a8d38c

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceapu3pccvujexw7h3xtzsdaze7ouvjitxy7axenswyfd4etrpng5q

DaYouGroup commented 1 year ago

Based on historical information, willing to support the first round, the following data will be followed up.

DaYouGroup commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebryh75osi2qixg2zubjxqpfvcnjvdogn5l7qrannf3ffb3v726us

Address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

Datacap Allocated

200.00TiB

Signer Address

f1nwjsd2mc6hu4qrwnmd6ukrfkuu4h5fhs7u3exii

Id

79e1c373-7fe4-4a13-a113-c2d6d3a8d38c

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebryh75osi2qixg2zubjxqpfvcnjvdogn5l7qrannf3ffb3v726us

herrehesse commented 1 year ago

Screenshot 2023-06-21 at 08 38 03

@Aaron01230 @DaYouGroup Can you explain your signature on a 7PiB request if the set contains only 5.49T of data?

Tagging: @raghavrmadya @galen-mcandrew @dkkapur @simonkim0515

Requesting a pauze on this application, will open a dispute.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

DataCap allocation requested

400TiB

Id

5143a561-a4e9-4759-b8ab-0daddcd234fe

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

Rule to calculate the allocation request amount

100% of weekly dc amount requested

DataCap allocation requested

400TiB

Total DataCap granted for client so far

200TiB

Datacap to be granted to reach the total amount requested by the client (7PiB)

6.80PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
3580 5 200TiB 23.18 43.32TiB
DaYouGroup commented 1 year ago

The original data is 430T, which is public and can be found on the website, https://registry.opendata.aws/tess/

aws s3 ls s3://stpubdata/tess --no-sign-request --summarize --human-readable --recursive

image

jevticgallonsd1466 commented 1 year ago

@herrehesse Hello. There must be something wrong with your statistical method. This dataset has 430.1TiB. image

The produced car files are each 19GiB, which can generate 430.1*1024/19=23180 cars, which generates 23180/32=724.375TiB raw data. image

jevticgallonsd1466 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

⚠️ All storage providers are located in the same region.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

kernelogic commented 1 year ago

I believe @herrehesse forgot to use --recursive parameter (or was it a result from chatGPT? )

alchemypunk commented 1 year ago

I believe @herrehesse forgot to use --recursive parameter (or was it a result from chatGPT? )

I think it is necessary to provide basic technical training for notaries.

jevticgallonsd1466 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

⚠️ All storage providers are located in the same region.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

jevticgallonsd1466 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

⚠️ All storage providers are located in the same region.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

jevticgallonsd1466 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

kernelogic commented 1 year ago

Report LGTM, plus the DD I performed above regarding source dataset size.

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebsyeic7typ2gw3v72z3yq3prt7h6tn2rw3hkriw3voflmrl7jbse

Address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

Datacap Allocated

400.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

5143a561-a4e9-4759-b8ab-0daddcd234fe

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebsyeic7typ2gw3v72z3yq3prt7h6tn2rw3hkriw3voflmrl7jbse

Bitrise0111 commented 1 year ago

the check bot shows healthy

Bitrise0111 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceazqipsg7x7bvcxahyt3fxa6j5pfg2lumvxoodp7xsustevsyvbiy

Address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

Datacap Allocated

400.00TiB

Signer Address

f1nknj7ayq4o43czrtdoauggtwl43fbqatmqis3yy

Id

5143a561-a4e9-4759-b8ab-0daddcd234fe

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceazqipsg7x7bvcxahyt3fxa6j5pfg2lumvxoodp7xsustevsyvbiy

herrehesse commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

jevticgallonsd1466 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

jevticgallonsd1466 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

woshidama323 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

DataCap allocation requested

800TiB

Id

5ab7f029-f844-470f-a108-200dfd478285

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

Rule to calculate the allocation request amount

200% of weekly dc amount requested

DataCap allocation requested

800TiB

Total DataCap granted for client so far

363797.9YiB

Datacap to be granted to reach the total amount requested by the client (7PiB)

363797.9YiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
16487 12 400TiB 12 102.75TiB
zcfil commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

zcfil commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebfsp47vtuyfgazas3bfwxss4eu773zuvrmkd2uwnsd5yfuykstzq

Address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

Datacap Allocated

800.00TiB

Signer Address

f1cjzbiy5xd4ehera4wmbz63pd5ku4oo7g52cldga

Id

5ab7f029-f844-470f-a108-200dfd478285

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebfsp47vtuyfgazas3bfwxss4eu773zuvrmkd2uwnsd5yfuykstzq

zcfil commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceauqb4b6kcmjbwwezlgh2zvl43r5annt2jtlu7ztlcllgkzpjrfua

Address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

Datacap Allocated

800.00TiB

Signer Address

f1cjzbiy5xd4ehera4wmbz63pd5ku4oo7g52cldga

Id

5ab7f029-f844-470f-a108-200dfd478285

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceauqb4b6kcmjbwwezlgh2zvl43r5annt2jtlu7ztlcllgkzpjrfua

ipollo00 commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

ipollo00 commented 1 year ago

This application has a clear explanation during the T&T call, which is trustworthy for all. The checker report is healthy. Willing to support.

ipollo00 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebweqlbd35kcwyia3x5ojgmre44r6uxghwlc22k47lnzi5nxusayw

Address

f1hhv5js4gxnw5z4vs7jvgr2cwygtp6gcecog4mui

Datacap Allocated

800.00TiB

Signer Address

f1n5wlrrhoxpkgwij25xrtt7w7g2k3fhbthmdn6ri

Id

5ab7f029-f844-470f-a108-200dfd478285

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebweqlbd35kcwyia3x5ojgmre44r6uxghwlc22k47lnzi5nxusayw

cryptowhizzard commented 1 year ago

This application has a clear explanation during the T&T call, which is trustworthy for all. The checker report is healthy. Willing to support.

No it has not been explained during T&T Working group call. Secondly https://github.com/filecoin-project/notary-governance/issues/913 is still open.

Good to know that we don't need to hold of on signing anymore when there are disputes. This behaviour is destructive for the whole ecosystem.