search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] Slingshot Terra Fusion Data Sampler dataset for Haoqian Mx team #308

Closed CodeIsLaw0108 closed 1 year ago

CodeIsLaw0108 commented 2 years ago

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Share a brief history of your project and organization.

An application for participating slingshot 2.8
We are newcomer for slingshot and we want to try our best to finish all the tasks

What is the primary source of funding for this project?

No cost ,
All the machines or any other fee for this application will be payed by ourselves 
Wish we will win finishing prize for covering some of them.

What other projects/ecosystem stakeholders is this project associated with?

Slingshot, MinerX, enterprise-sp-wp.

Use-case details

Describe the data being stored onto Filecoin

The Terra Basic Fusion dataset is a fused dataset of the original Level 1 radiances from the five Terra instruments.

Where was the data in this dataset sourced from?

https://registry.opendata.aws/terrafusion/

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this. 

https://terrafusiondatasampler.s3.amazonaws.com/P108/TERRA_BF_L1B_O10204_20011118010522_F000_V001.h5

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

yes

What is the expected retrieval frequency for this data?

Once a day

For how long do you plan to keep this dataset stored on Filecoin?

1 year+

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

All regions.

How will you be distributing your data to storage providers? Is there an offline data transfer process?

I will upload my prepared CAR files to a web server and coordinate with providers to download and propose offline deals.
A prepared web file system where storing car files will be launched online for anyone who want to download for sealing 
Of course offline deal  is prefer.

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

Any SPs who could prove that they have ability to make data safe and can finish the sealing tasks.

How will you be distributing deals across storage providers?

No more than 30% each project for a SP entry

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

Yes have some, Will appreciate if community could help to find more suitable sp for sealing.
large-datacap-requests[bot] commented 2 years ago

Thanks for your request! :exclamation: We have found some problems in the information provided. We could not find any Web site or social media info in the information provided

Please, take a look at the request and edit the body of the issue providing all the required information.
large-datacap-requests[bot] commented 2 years ago

Thanks for your request! :exclamation: We have found some problems in the information provided. We could not find any Web site or social media info in the information provided

Please, take a look at the request and edit the body of the issue providing all the required information.
large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 2 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

CodeIsLaw0108 commented 2 years ago

@galen-mcandrew could you help to sign this issue if all the info is correct?

galen-mcandrew commented 2 years ago

@dkkapur can you verify this slingshot request?

dkkapur commented 2 years ago

Confirming that this is on the eligible datasets for Slingshot - https://github.com/filecoin-project/slingshot/blob/master/datasets.md.

However - applicants that have received DataCap in the past for Slingshot V2 have met a very high bar in proving the quality of the work they are doing as well as the robustness of the deal distribution mechanism to ensure they work with several SPs and do not self-deal. On behalf of the Slingshot community, I'd love to hear more about @CodeIsLaw0108's plan to identify SPs and fairly distribute deals across them.

CodeIsLaw0108 commented 2 years ago

Hi Any update ? @dkkapur For "plan to identify SPs and fairly distribute deals across them"

  1. Need to check the sps' IPs , Get useful infos from slack channels about a specific miner ID which i want to work with for identifying if there are together.
  2. Talk to SPs and make sure will agree with slingshot rules, If broken, We will stop sending deals if not followed
  3. Make sure all the SPs will be assigned with 3:1:1 for a certain project by monitoring the dealing procedure

I think there are no reliable methods for doing this. They are no reliable info on-chain for clarifying this unless any miners need to publish their public IP honestly.

galen-mcandrew commented 2 years ago

Has there been any progress in identifying SP's to work with? I know there are multiple other Slingshot participants that may be able to help provide some more details or starting points for you.

dkkapur commented 2 years ago

@kernelogic if you have any suggestions here, might be good to get your advice!

kernelogic commented 2 years ago

I agree there's no reliable methods for identifying without a pre-registration platform. I guess the second best alternative would be SPs (or the entities) registered during the slingshot restore program earlier this year.

Given the short remaining time on the current phase 2.8 and the size of this LDN, if OP could give a planned list of SP and their respective owning entity to prove no self-dealing and following decentralization rules, that would be very convincing.

dkkapur commented 2 years ago

Given the short remaining time on the current phase 2.8 and the size of this LDN, if OP could give a planned list of SP and their respective owning entity to prove no self-dealing and following decentralization rules, that would be very convincing.

+1. Looking forward to hearing from @CodeIsLaw0108

CodeIsLaw0108 commented 2 years ago

Hi, Very happy while we know slingshot2.8 will continue up to 7.6 ,Thanks Could you help to give us a list of SPs , and we could contact them directly ? I think there is no necessary for us to proving such issue if our list is from community ?

dkkapur commented 2 years ago

@CodeIsLaw0108 fair enough. The Slingshot site has information available about every deal uploaded and the SPs involved, e.g., for one of @kernelogic's projects, you can see the recent deal metadata here: https://slingshot.filecoin.io/project/62340a7a7d907fffc878620e/deals.

Additionally, from Slingshot Evergreen, we publish a list of active SP IDs and their relative deal making here: https://api.evergreen.filecoin.io/public/qap.txt.

What do you think?

CodeIsLaw0108 commented 2 years ago

Yeah Thanks @dkkapur

  1. f01850141 (from cabrina)
  2. (from https://slingshot.filecoin.io/project/62340a7a7d907fffc878620e/deals. Will ask kernelogic for help to contact the sp)

As for https://api.evergreen.filecoin.io/public/qap.txt. Could you please help to give slack name also ? It's hard for me to find slack name from miner ID ?

CodeIsLaw0108 commented 2 years ago

So Can i send deal first for f01850141 As time fly, and wenkend again But

The current reward period ends in 19d 11h 30m 31s
Chris00618 commented 2 years ago

Hey Cabrina, does this node f01850141 belong to you? @xingjitansuo https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/312 https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/232 As you know, self dealing in LDN is not allowed in our community. In addition,we found another case connected with you ,could you give some clarification about it. https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/354

NiwanDao commented 2 years ago

@Chris00618 I totally understand and follow the community rule on prohibition of seal-dealing. I do not own SP f01850141. I just recommend this node to @CodeIsLaw0108 .

CodeIsLaw0108 commented 2 years ago

@Chris00618 Thanks for your reply after waiting so long time . Have checked with @xingjitansuo , Sorry my fault ,Just one of their SP before slingshot 2.8 But can I still send deals for this miner ? Could you help to comments this and help us to speed up this procedure ? image

We all want to participate slingshot 2.8 and of course want to get reward from this project But , some guys have finished almost all the task about slingshot 2.8 while we are still struggling applying the project, And project is almost end Many SPs have to pay for Datacap as they need deals for sealing but only few client have gotten their datacap

Wish community could handle issues as quickly as possible , And you can closed this issue if there is any issue that will not meet the requirement, then there is no necessary for us to ask for checking this issue again and again.

dkkapur commented 2 years ago

@CodeIsLaw0108 I recommend you join #slingshot-evergreen channel to find some more SPs to work with. Approving this one for now so you can start to make deals in the final 2 weeks of v2.8.

dkkapur commented 2 years ago

Datacap Request Trigger

Total DataCap requested

1 PiB

Expected weekly DataCap usage rate

100 TiB

Client address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Multisig Notary address

f01858410

Client address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

DataCap allocation requested

50TiB

kernelogic commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedkgfhztnbjtrujxeq7aksyxjpezrk3dg2zlebiurwcx7b7kkynly

Address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Datacap Allocated

50.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedkgfhztnbjtrujxeq7aksyxjpezrk3dg2zlebiurwcx7b7kkynly

NiwanDao commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedfuwpyxffdezk2ddmgsf24mftex67gwsr5ke42sw7vwliblszmjw

Address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Datacap Allocated

50.00TiB

Signer Address

f1a2lia2cwwekeubwo4nppt4v4vebxs2frozarz3q

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedfuwpyxffdezk2ddmgsf24mftex67gwsr5ke42sw7vwliblszmjw

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f01858410

Client address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

DataCap allocation requested

100TiB

large-datacap-requests[bot] commented 2 years ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Last two approvers

xingjitansuo & kernelogic

Rule to calculate the allocation request amount

100% of weekly dc amount requested

DataCap allocation requested

100TiB

Total DataCap granted for client so far

50TiB

Datacap to be granted to reach the total amount requested by the client (1 PiB)

974TiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
1122 4 50TiB 33.07 9.90TiB
1ane-1 commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedv4j7evy4wbdwo62ld3dbfkazavhsyzzeoxy27p7o3xg22monsik

Address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Datacap Allocated

100.00TiB

Signer Address

f1mdk7s2vntzm6hu35yuo6vjubtrpfnb2awhgvrri

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedv4j7evy4wbdwo62ld3dbfkazavhsyzzeoxy27p7o3xg22monsik

psh0691 commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacea3gryhu4zkbeksb3yjfcz7yv4czx5wo27xc5nz6ceuqon5a7dzlu

Address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Datacap Allocated

100.00TiB

Signer Address

f1qdko4jg25vo35qmyvcrw4ak4fmuu3f5rif2kc7i

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea3gryhu4zkbeksb3yjfcz7yv4czx5wo27xc5nz6ceuqon5a7dzlu

psh0691 commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacec7iowtwost6ury7evnieetnrkf2k3vmmj4pskbivjhd7qdz3m7j4

Address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Datacap Allocated

100.00TiB

Signer Address

f1qdko4jg25vo35qmyvcrw4ak4fmuu3f5rif2kc7i

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec7iowtwost6ury7evnieetnrkf2k3vmmj4pskbivjhd7qdz3m7j4

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f01858410

Client address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

DataCap allocation requested

200TiB

large-datacap-requests[bot] commented 2 years ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Last two approvers

psh0691 & psh0691

Rule to calculate the allocation request amount

200% of weekly dc amount requested

DataCap allocation requested

200TiB

Total DataCap granted for client so far

50TiB

Datacap to be granted to reach the total amount requested by the client (1 PiB)

974TiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
1567 5 100TiB 26.42 2.06TiB
Filefly-ph commented 2 years ago

So do you have any idea of who is f01850141? This guy is a major SP in Filecoin and you're cooperating with him/her.

jamerduhgamer commented 2 years ago

Is this a bug where psh0691 was able to both propose and approve the datacap request?

psh0691 commented 2 years ago

@jamerduhgamer @Filefly-ph I am not cooperating with this applicant, and I cannot make proposals and approvals at once. In this case, when there was a bug where the proposal request was repeated, I signed the approval, and the proposal was repeated. I remember reporting to Slack's notary channel. https://filecoinproject.slack.com/archives/C01HRNU4VBK/p1661305706567739

jamerduhgamer commented 2 years ago

Ah that makes sense now. You tried to approve the datacap request and the wrong message was sent. Then you tried to approve it again which went through with the correct message. Thank you for answering my question @psh0691

NiwanDao commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceb4iahqcggxwwzmuoqrkcopozakg23jeada535lggedkm2dq333gu

Address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Datacap Allocated

100.00TiB

Signer Address

f1a2lia2cwwekeubwo4nppt4v4vebxs2frozarz3q

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceb4iahqcggxwwzmuoqrkcopozakg23jeada535lggedkm2dq333gu

BDE-io commented 2 years ago

@CodeIsLaw0108 Hi! Great to see you have gotten approval for DataCap. If you are looking for storage providers to store these data or have any questions, please visit #bigdata-exchange on Filecoin Slack or reply here.

We have strong demand from a diverse group of SPs, who are actively looking to onboard more data.

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Request number 4

Multisig Notary address

f01858410

Client address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

DataCap allocation requested

400TiB

large-datacap-requests[bot] commented 2 years ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Last two approvers

xingjitansuo & psh0691

Rule to calculate the allocation request amount

400% of weekly dc amount requested

DataCap allocation requested

400TiB

Total DataCap granted for client so far

150TiB

Datacap to be granted to reach the total amount requested by the client (1 PiB)

874TiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
1567 5 200TiB 26.42 46.31TiB
newwebgroup commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecqz62lvdlyururvosbmjmvex6oajkiwqz6u4ud7osq75qhnhj7h6

Address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Datacap Allocated

400.00TiB

Signer Address

f1e77zuityhvvw6u2t6tb5qlnsegy2s67qs4lbbbq

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecqz62lvdlyururvosbmjmvex6oajkiwqz6u4ud7osq75qhnhj7h6

kernelogic commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebbcmchsz2ofwurkw7xxy6tknrgsncxcbdwvskd6woa5jfhmgbr4m

Address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Datacap Allocated

400.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebbcmchsz2ofwurkw7xxy6tknrgsncxcbdwvskd6woa5jfhmgbr4m

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f01858410

Client address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

DataCap allocation requested

274TiB

Id

5ba40089-e49d-4f9e-821e-44c0bc47cffb

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Last two approvers

kernelogic & newwebgroup

Rule to calculate the allocation request amount

800% of weekly dc amount requested

DataCap allocation requested

274TiB

Total DataCap granted for client so far

550TiB

Datacap to be granted to reach the total amount requested by the client (1 PiB)

474TiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
10859 8 400TiB 19.72 89.06TiB
psh0691 commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacediwkrbapbdiqqupsvau6rwslwtmvfnb53gmt7jcf2oqfxjcve5za

Address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Datacap Allocated

274.00TiB

Signer Address

f1qdko4jg25vo35qmyvcrw4ak4fmuu3f5rif2kc7i

Id

5ba40089-e49d-4f9e-821e-44c0bc47cffb

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacediwkrbapbdiqqupsvau6rwslwtmvfnb53gmt7jcf2oqfxjcve5za

kernelogic commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacec5om4mrhhx4uamsyqa25kcm35o736quxawscj3clstyjvkktsi32

Address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Datacap Allocated

274.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

5ba40089-e49d-4f9e-821e-44c0bc47cffb

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacec5om4mrhhx4uamsyqa25kcm35o736quxawscj3clstyjvkktsi32

filplus-checker commented 1 year ago

DataCap and CID Checker Report[^1]

If this is the first time a provider takes verified deal, it will be marked as new.

For most of the datacap application, below restrictions should apply.

⚠️ f01926698 has unknown IP location.

Provider Location Total Deals Sealed Percentage Unique Data Duplicate Deals
f01969779new Clifton, New Jersey, US 83.13 TiB 18.40% 83.13 TiB 0.00%
f0143858 Clifton, New Jersey, US 80.34 TiB 17.79% 80.34 TiB 0.00%
f03223 San Jose, California, US 79.06 TiB 17.50% 79.06 TiB 0.00%
f02301 San Jose, California, US 77.25 TiB 17.10% 77.25 TiB 0.00%
f0240185 Clifton, New Jersey, US 57.47 TiB 12.72% 57.47 TiB 0.00%
f0142637 Chengdu, Sichuan, CN 41.16 TiB 9.11% 41.16 TiB 0.00%
f01926698 Unknown 27.19 TiB 6.02% 27.19 TiB 0.00%
f01850141 Hong Kong, Central and Western, HK 6.13 TiB 1.36% 6.13 TiB 0.00%

Provider Distribution

Deal Data Replication

The below table shows how each many unique data are replicated across storage providers.

✔️ Data replication looks healthy.

Unique Data Size Total Deals Made Number of Providers Deal Percentage
19.06 TiB 19.06 TiB 1 4.22%
19.75 TiB 39.50 TiB 2 8.74%
3.69 TiB 11.06 TiB 3 2.45%
24.44 TiB 97.75 TiB 4 21.64%
35.78 TiB 178.91 TiB 5 39.61%
16.41 TiB 98.44 TiB 6 21.79%
1.00 TiB 7.00 TiB 7 1.55%

Replication Distribution

Deal Data Shared with other Clients

The below table shows how many unique data are shared with other clients. Usually different applications owns different data and should not resolve to the same CID.

⚠️ CID sharing has been observed.

Other Client Application Total Deals Affected Unique CIDs Verifier
f1o54sve7ede7im4caux3ug7lsyjmbue7ss3zzl6y FilSwan 85.25 TiB 384 LDN v3 multisig
f1bstbq5bi72kyovhh7zoo2f6l6uivsjz4ey5dnqq FilSwan 33.38 TiB 258 LDN v3 multisig
f14r2jybmccwiu6hze4fu55jyhclktvwacec56hea FilSwan 9.16 TiB 126 LDN v3 multisig
f1r3d25hl2y7rqlsu2mgczdethy4qqjmkfdlmibfq NEXRAD - FilSwan 6.19 TiB 198 LDN v3 multisig
f13d6zt424jkp55u7kp67azotkzadtlokwnz2ntxa FilSwan - Slingshot Restore 1.97 TiB 63 LDN v3 multisig

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

large-datacap-requests[bot] commented 1 year ago

The issue reached the total datacap requested. This should be closed

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01858410

Client address

f1eto4dfafkoeukxenoc7vzexrvf6i4yqvhszrq2y

Last two approvers

kernelogic & psh0691

Rule to calculate the allocation request amount

total dc reached

DataCap allocation requested

0

Total DataCap granted for client so far

824TiB

Datacap to be granted to reach the total amount requested by the client (1 PiB)

200TiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
24025 8 274TiB 26.79 65.78TiB
cryptowhizzard commented 1 year ago

Hello @CodeIsLaw0108

It seems that the total datacap is reached. Can you close the issue please?

Thanks!