filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
110 stars 62 forks source link

[DataCap Application] Solana Labs - Solana Ledger #2145

Closed yihau closed 6 months ago

yihau commented 1 year ago

Data Owner Name

Solana Labs

What is your role related to the dataset

Dataset Owner

Data Owner Country/Region

United States

Data Owner Industry

Web3 / Crypto

Website

https://solana.com

Social Media

https://twitter.com/solanalabs

Total amount of DataCap being requested

5PiB

Expected size of single dataset (one copy)

~ 2TiB. (may become bigger in the future)

Number of replicas to store

3

Weekly allocation of DataCap requested

50TiB

On-chain address for first allocation

f3xgawho4wlo2okfyj63m27qh34l7ovlxyqslsiskm437d2rqudqethzvemqpzoemfvf7sjix3gdughfb5tk3q

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Identifier

No response

Share a brief history of your project and organization

Solana is a blockchain built for mass adoption. It's a high performance network that is utilized for a range of use cases, including finance, NFTs, payments, and gaming. Solana operates as a single global state machine, and is open, interoperable and decentralized.

Is this project associated with other projects/ecosystem stakeholders?

Yes

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

The ledger of the Solana blockchain.

Where was the data currently stored in this dataset sourced from

Google Cloud

If you answered "Other" in the previous question, enter the details here

No response

If you are a data preparer. What is your location (City and Country)

No response

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

1) download our ledger data from the GCP cloud storage bucket
2) use https://github.com/anjor/go-fil-dataprep to generate car files
3) upload all car files to another GCP cloud storage bucket
4) using boost client to send deals

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

Yes, we would like to control the flow on our end. In our previous cooperation, we only provided raw data, which may have increased our communication costs.

Please share a sample of the data

https://console.cloud.google.com/storage/browser/chido/epoch-2

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Daily

For how long do you plan to keep this dataset stored on Filecoin

Permanently

In which geographies do you plan on making storage deals

global

How will you be distributing your data to storage providers

Cloud storage (i.e. S3), HTTP or FTP server

How do you plan to choose storage providers

We will be working with the network growth team at PL to identify storage providers that will serve retrievals.

If you answered "Others" in the previous question, what is the tool or platform you plan to use

No response

If you already have a list of storage providers to work with, fill out their names and provider IDs below

No response

How do you plan to make deals to your storage providers

Boost client, Singularity

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

Sunnyiscoming commented 1 year ago

This information will be reviewed by Fil+ Governance team to confirm validity and then the application will be triggered for notary review. Let us know if you have any questions.

anjor commented 1 year ago

@Sunnyiscoming, @yihau is the client and not the SP. So I don't think they will be able to answer the first question.

For the second question -- the SPs have not been identified yet, but as @yihau said in the issue, Solana Labs is working with the PL network growth team as well as FF to identify the SPs to work with.

I will let @yihau handle the last 2 questions.

kevzak commented 1 year ago

@anjor apologies, was not aware this was a Spade use case looking to better test retrieval requirements with SPs TBD. Please confirm and we can move this forward.

anjor commented 1 year ago

To clarify: this is a potential future tenant for spade once the tenant onboarding service for spade is ready (it's currently being developed).

In the meantime, @yihau will be working with Yan Bo, a solutions architect at the filecoin foundation, and me, an engineer at protocol labs to set up the onboarding pipeline. This involves figuring out the correct technical solution, identifying SPs, and facilitating the interaction between the client (Solana labs) and the SPs to make sure the data is retrievable to the client's satisfaction.

kevzak commented 1 year ago

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

50TiB

Client address

f3xgawho4wlo2okfyj63m27qh34l7ovlxyqslsiskm437d2rqudqethzvemqpzoemfvf7sjix3gdughfb5tk3q

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f3xgawho4wlo2okfyj63m27qh34l7ovlxyqslsiskm437d2rqudqethzvemqpzoemfvf7sjix3gdughfb5tk3q

DataCap allocation requested

25TiB

Id

d229c745-9004-4784-a086-a3b40418e3d2

kernelogic commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacede3w5fvhf7mlmv3buxnv3jochncfms2moownstaasmmpdfvqtblo

Address

f3xgawho4wlo2okfyj63m27qh34l7ovlxyqslsiskm437d2rqudqethzvemqpzoemfvf7sjix3gdughfb5tk3q

Datacap Allocated

25.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

d229c745-9004-4784-a086-a3b40418e3d2

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacede3w5fvhf7mlmv3buxnv3jochncfms2moownstaasmmpdfvqtblo

kernelogic commented 1 year ago

Looks legit from KYC perspective.

Bennyyangpu commented 1 year ago

To clarify: this is a potential future tenant for spade once the tenant onboarding service for spade is ready (it's currently being developed).

In the meantime, @yihau will be working with Yan Bo, a solutions architect at the filecoin foundation, and me, an engineer at protocol labs to set up the onboarding pipeline. This involves figuring out the correct technical solution, identifying SPs, and facilitating the interaction between the client (Solana labs) and the SPs to make sure the data is retrievable to the client's satisfaction.

willing to support!

Bennyyangpu commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaced4d53semu6qpwgfi5ubyml6ijqrk2gpzis45ti6zc24tgwfwwrig

Address

f3xgawho4wlo2okfyj63m27qh34l7ovlxyqslsiskm437d2rqudqethzvemqpzoemfvf7sjix3gdughfb5tk3q

Datacap Allocated

25.00TiB

Signer Address

f174fg3bqbln3zjnkxtyf6s54txqkr7yqkj6cig7y

Id

d229c745-9004-4784-a086-a3b40418e3d2

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaced4d53semu6qpwgfi5ubyml6ijqrk2gpzis45ti6zc24tgwfwwrig

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

anjor commented 1 year ago

Keep open

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

yihau commented 1 year ago

keep open!

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f3xgawho4wlo2okfyj63m27qh34l7ovlxyqslsiskm437d2rqudqethzvemqpzoemfvf7sjix3gdughfb5tk3q

DataCap allocation requested

50TiB

Id

40a61c16-6a96-417d-8932-0ca1db5127d3

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f3xgawho4wlo2okfyj63m27qh34l7ovlxyqslsiskm437d2rqudqethzvemqpzoemfvf7sjix3gdughfb5tk3q

Rule to calculate the allocation request amount

100% of weekly dc amount requested

DataCap allocation requested

50TiB

Total DataCap granted for client so far

25TiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

4.97PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
0 0 25TiB NaN 25TiB
spaceT9 commented 1 year ago

checker:manualTrigger

joshua-ne commented 1 year ago

checker:manualTrigger

spaceT9 commented 1 year ago

checker:manualTrigger

joshua-ne commented 12 months ago

checker:manualTrigger

cryptowhizzard commented 11 months ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebxriac4bfbd5hzdd2li7mhtgy4pf4nejkak4f6ul6nit4fjwpcia

Address

f3xgawho4wlo2okfyj63m27qh34l7ovlxyqslsiskm437d2rqudqethzvemqpzoemfvf7sjix3gdughfb5tk3q

Datacap Allocated

50.00TiB

Signer Address

f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa

Id

40a61c16-6a96-417d-8932-0ca1db5127d3

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebxriac4bfbd5hzdd2li7mhtgy4pf4nejkak4f6ul6nit4fjwpcia

joshua-ne commented 11 months ago

checker:manualTrigger

joshua-ne commented 11 months ago

I'm wondering why the bot fails to work on this application. @raghavrmadya @Kevin-FF-USA
@cryptowhizzard Have you done related DD on this, and if so, would you mind share it? Thanks

anjor commented 11 months ago

My guess would be it's because no data has been onboarded yet.

Is that right @yihau ?

yihau commented 11 months ago

yes, we haven't started to onboard any data. I will consume some of them late this week for testing (if everything goes well on my end 🀞 )

joshua-ne commented 11 months ago

It's strange to me. If no DC is consumed yet, how the second round of allocation is triggered?

anjor commented 11 months ago

That is a good question. @simonkim0515 @panges2

github-actions[bot] commented 11 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

yihau commented 11 months ago

keep open πŸ™

github-actions[bot] commented 11 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

github-actions[bot] commented 10 months ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

-- Commented by Stale Bot.

yihau commented 9 months ago

@anjor could you reopen this one πŸ™

anjor commented 9 months ago

@simonkim0515 can you help?

github-actions[bot] commented 9 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

yihau commented 9 months ago

πŸ™

Sunnyiscoming commented 9 months ago

Please provide ID, City, Country, Organization of each SP here.

github-actions[bot] commented 8 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

yihau commented 8 months ago

πŸ™

Sunnyiscoming commented 8 months ago

@yihau Hi, please provide ID, City, Country, Organization of each SP here.

github-actions[bot] commented 8 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

yihau commented 8 months ago

πŸ™

github-actions[bot] commented 8 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

yihau commented 7 months ago

πŸ™

Ng-YanBo commented 7 months ago

@Sunnyiscoming @Kevin-FF-USA @simonkim0515 Hey Folks, Yan from Filecoin Foundation here. Since they have regional requirements, Solana will have to wait for one last SP in America to free up the space for Solana as the last replica. Can I check if the LDN will have an issue when the new FIL + program is in place as well. Estimated time for this is 2-3weeks

github-actions[bot] commented 7 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

yihau commented 7 months ago

πŸ™

github-actions[bot] commented 7 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.

yihau commented 7 months ago

πŸ™

github-actions[bot] commented 6 months ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

-- Commented by Stale Bot.