search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
109 stars 62 forks source link

[DataCap Application] Speedium - Fly Brain Anatomy dataset #77

Closed cryptowhizzard closed 1 year ago

cryptowhizzard commented 3 years ago

Large Dataset Notary Application

To apply for DataCap to onboard your dataset to Filecoin, please fill out the following.

Core Information

Please respond to the questions below by replacing the text saying "Please answer here". Include as much detail as you can in your answer.

Project details

Share a brief history of your project and organization.

Speedium has participated in SR1 and we are building a bridge between Usenet and Filecoin.
We are ramping up at the moment and have a ready to go system to onboard data to filecoin, however it is not stress-tested on many miners with heavy traffic. We intend to do one or more slingshot rounds to practice the onboarding of large set' s of data at high speed to test our backends and make things work.

What is the primary source of funding for this project?

We have not received any grants for sending data to the network untill now. We aim to utilize this datacap at zero cost, and send data to all small(er)  active community providers (< 2PiB) out there on Slack, especially MinerX / enterprise-sp-wg members who are willing and capable of receiving deals.

We participate in slingshot to aim to have some  return on our efforts to get everything working properly and cover the costs of bandwidth, hardware and communicating with providers to keep everything aligned during this phase. We aim to do this in the SP group.

What other projects/ecosystem stakeholders is this project associated with?

Slingshot, MinerX, enterprise-sp-wp.

Use-case details

Describe the data being stored onto Filecoin

Please answer here.

Where was the data in this dataset sourced from?

Fly Brain Anatomy, made available by Janelia's FlyLight project, consists of fluorescence images of Drosophila melanogaster driver lines, aligned to standard templates, and stored in formats suitable for rapid searching in the cloud.

Can you share a sample of the data? A link to a file, an image, a table, etc., are good ways to do this. 

https://registry.opendata.aws/janelia-flylight/

Confirm that this is a public dataset that can be retrieved by anyone on the Network (i.e., no specific permissions or access rights are required to view the data).

Yes. It has the license of https://creativecommons.org/licenses/by/4.0/.

What is the expected retrieval frequency for this data?

Anyone interested in retrieving this datacet ( a few times we guess )

For how long do you plan to keep this dataset stored on Filecoin?

1.5 yr.

DataCap allocation plan

In which geographies (countries, regions) do you plan on making storage deals?

All regions but mainly EU and US as Speedium has the most bandwidth towards those regions.

How will you be distributing your data to storage providers? Is there an offline data transfer process?

Online!

How do you plan on choosing the storage providers with whom you will be making deals? This should include a plan to ensure the data is retrievable in the future both by you and others.

We will contact the MinerX and enterprise-sp-wg members as well as any other providers interested to receive the deals.

How will you be distributing deals across storage providers?

Evenly across all providers as long as they can handle. If a miner is a notary itself, this notary will receive no more than 10% of the total granted datacap.

Do you have the resources/funding to start making deals as soon as you receive DataCap? What support from the community would help you onboard onto Filecoin?

We have enough funding / recourses to give this a try-out.
large-datacap-requests[bot] commented 3 years ago

Thanks for your request! :exclamation: We have found some problems in the information provided. We could not find the Datacap requested in the information provided

Please, take a look at the request and edit the body of the issue providing all the required information.
large-datacap-requests[bot] commented 3 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

Reiers commented 3 years ago

Looks good - you have my support.

s0nik42 commented 3 years ago

good for me

MegTei commented 3 years ago

Happy to support

MRJAVAZHAO commented 3 years ago

Is this application the same as the other application (https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/60)?

cryptowhizzard commented 3 years ago

@MRJAVAZHAO no it isn't.

large-datacap-requests[bot] commented 3 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

galen-mcandrew commented 3 years ago

Multisig Notary requested

Total DataCap requested

2PiB

Expected weekly DataCap usage rate

200TiB

large-datacap-requests[bot] commented 3 years ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

large-datacap-requests[bot] commented 3 years ago

**Multisig created and sent to RKH f01403237

large-datacap-requests[bot] commented 3 years ago

DataCap Allocation requested

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

DataCap allocation requested

100TiB

s0nik42 commented 3 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceazpxcbtlhfxm7awsdcr3byine5ifyxigej3r7ollwxnefczwuwkw

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

100TiB

Signer Address

f1wxhnytjmklj2czezaqcfl7eb4nkgmaxysnegwii

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceazpxcbtlhfxm7awsdcr3byine5ifyxigej3r7ollwxnefczwuwkw

MegTei commented 3 years ago

image Attempted to approve but getting this error

MegTei commented 3 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebvas6rrvh44yluytymja6kdmaar73uroqwjulj33ekitviq7z5h6

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

100TiB

Signer Address

f1ystxl2ootvpirpa7ebgwl7vlhwkbx2r4zjxwe5i

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebvas6rrvh44yluytymja6kdmaar73uroqwjulj33ekitviq7z5h6

MegTei commented 3 years ago

2nd time lucky, should be good to go @cryptowhizzard

dkkapur commented 3 years ago

Ended up being a duplicate tx above as well - adding status: signed for now! We have 1 approval and 2 proposals. Second proposal should be ignored.

cryptowhizzard commented 2 years ago

Hello @s0nik42 @Reiers

Can you please state that you are ok that we use this datacap also for the slingshot recovery program? That program has priority now and we are focussing our efforts on this.

Reiers commented 2 years ago

Everything looks good. Speedium have my support!

s0nik42 commented 2 years ago

Same here

cryptowhizzard commented 2 years ago

@dkkapur @galen-mcandrew it seems that the bot did apply for automatic renewal of my datacap. Can you please do RKH request manually.

@s0nik42 it is possible that you give us some manual datacap in the meantime, like 20 TB on above address?

galen-mcandrew commented 2 years ago

The previous allocation was for 100 TiB of DataCap, and checking this client address still has 94 TiB. The subsequent allocation bot will not request more DataCap until 75% of the previous allocation has been sealed in deals.

Additionally, this multisig still has a pending transaction from the previous triple signature. So when you have used more DataCap and the SA bot kicks off the next allocation, it should only require one notary signature.

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

DataCap allocation requested

200TiB

large-datacap-requests[bot] commented 2 years ago

Stats for DataCap Allocation

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Last two approvers

MegTei & s0nik42

DataCap allocation requested

200TiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
3787 54 100TiB 21.08 14.6TiB
flyworker commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacednxl4kwt7aneduncpvairawg5iykmb5vvznjbjoqf55d4seqbslk

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

200TiB

Signer Address

f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacednxl4kwt7aneduncpvairawg5iykmb5vvznjbjoqf55d4seqbslk

galen-mcandrew commented 2 years ago

Looks like there are now two pending transactions, ID 2 for 100TiB and ID 3 for 200TiB. Adjusting labels, next signature should clear one of these transactions.

s0nik42 commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebmffl7e6eymaxa5du4vqxebbvmrcqdfoq4wubk72d4mz4thorlge

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

200TiB

Signer Address

f1wxhnytjmklj2czezaqcfl7eb4nkgmaxysnegwii

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebmffl7e6eymaxa5du4vqxebbvmrcqdfoq4wubk72d4mz4thorlge

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

DataCap allocation requested

400TiB

large-datacap-requests[bot] commented 2 years ago

Stats for DataCap Allocation

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Last two approvers

s0nik42 & flyworker

DataCap allocation requested

400TiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
4186 55 200TiB 18.82 33.8TiB
Reiers commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecgsa4rzrsmaclpebdlgi7yem5htewibhp3ggzoo7cjjbi6mlxwim

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

400TiB

Signer Address

f1oz43ckvmtxmmsfzqm6bpnemqlavz4ifyl524chq

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecgsa4rzrsmaclpebdlgi7yem5htewibhp3ggzoo7cjjbi6mlxwim

s0nik42 commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceccd27phjtfp442juy4yz6vypdeyoyavgcglf7shrgqkkmiydw6qg

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

400TiB

Signer Address

f1wxhnytjmklj2czezaqcfl7eb4nkgmaxysnegwii

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceccd27phjtfp442juy4yz6vypdeyoyavgcglf7shrgqkkmiydw6qg

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

DataCap allocation requested

800TiB

large-datacap-requests[bot] commented 2 years ago

Stats for DataCap Allocation

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Last two approvers

s0nik42 & Reiers

DataCap allocation requested

800TiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
0 0 400TiB 0 37.89150682091713TiB
MegTei commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedytzjmpliktxusddnbzmxfi6k7gg4sru3dam4bbdudfot4hlwlkk

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

800.00TiB

Signer Address

f1ystxl2ootvpirpa7ebgwl7vlhwkbx2r4zjxwe5i

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedytzjmpliktxusddnbzmxfi6k7gg4sru3dam4bbdudfot4hlwlkk

flyworker commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecwhy7i4wr3mdeqdl4efmein27skly37c3ndkizeaqwybprox7ywq

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

800.00TiB

Signer Address

f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecwhy7i4wr3mdeqdl4efmein27skly37c3ndkizeaqwybprox7ywq

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Request number 5

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

DataCap allocation requested

1.12109375PiB

large-datacap-requests[bot] commented 2 years ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Last two approvers

not found & not found

Rule to calculate the allocation request amount

800% of weekly dc amount requested

DataCap allocation requested

1.12109375PiB

Total DataCap granted for client so far

900TiB

Datacap to be granted to reach the total amount requested by the client (2PiB)

1.12109375PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
20618 57 800TiB 23.93 194.39150682091713TiB
large-datacap-requests[bot] commented 2 years ago

The allocation of the requested datacap cannot be performed, because it would result in the exceeding of the total granted data cap.

dannyob commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecbrizbpjznjn7obnxxjjit63mehb5jdtg4iydez6h4b2i4gwf4si

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

1.12PiB

Signer Address

f1k6wwevxvp466ybil7y2scqlhtnrz5atjkkyvm4a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecbrizbpjznjn7obnxxjjit63mehb5jdtg4iydez6h4b2i4gwf4si

flyworker commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecj5fgnq6j6ns66rqqxnrzxt4yry2ft4dqxzclzb3pbrfy4tkmwa2

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

1.12PiB

Signer Address

f1hlubjsdkv4wmsdadihloxgwrz3j3ernf6i3cbpy

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecj5fgnq6j6ns66rqqxnrzxt4yry2ft4dqxzclzb3pbrfy4tkmwa2

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Request number 6

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

DataCap allocation requested

1.56PiB

large-datacap-requests[bot] commented 2 years ago

The allocation of the requested datacap cannot be performed, because it would result in the exceeding of the total granted data cap.

large-datacap-requests[bot] commented 2 years ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Last two approvers

not found & not found

Rule to calculate the allocation request amount

800% of weekly dc amount requested

DataCap allocation requested

1.56PiB

Total DataCap granted for client so far

32GiB

Datacap to be granted to reach the total amount requested by the client (2PiB)

1.99PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
0 0 1.12109375PiB 0 272.23TiB
dannyob commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceaje5t3iu3iepnvtavdrqdwmhbmqvihi25bshvjwfyxnneeyr5d2c

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

1.56PiB

Signer Address

f1k6wwevxvp466ybil7y2scqlhtnrz5atjkkyvm4a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceaje5t3iu3iepnvtavdrqdwmhbmqvihi25bshvjwfyxnneeyr5d2c

MegTei commented 2 years ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecjkw6xtorljstcw4apgxl5jwgkpqmhk4sni6g2dvkxvdn4d7a3m4

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

1.56PiB

Signer Address

f1ystxl2ootvpirpa7ebgwl7vlhwkbx2r4zjxwe5i

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecjkw6xtorljstcw4apgxl5jwgkpqmhk4sni6g2dvkxvdn4d7a3m4

large-datacap-requests[bot] commented 2 years ago

DataCap Allocation requested

Request number 7

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

DataCap allocation requested

1.56PiB

large-datacap-requests[bot] commented 2 years ago

The allocation of the requested datacap cannot be performed, because it would result in the exceeding of the total granted data cap.

large-datacap-requests[bot] commented 2 years ago

Stats & Info for DataCap Allocation

Multisig Notary address

f01403237

Client address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Last two approvers

not found & not found

Rule to calculate the allocation request amount

800% of weekly dc amount requested

DataCap allocation requested

1.56PiB

Total DataCap granted for client so far

32GiB

Datacap to be granted to reach the total amount requested by the client (2PiB)

1.99PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
0 0 1.56PiB 0 227.15TiB
Reiers commented 2 years ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebvde5d6rvsgrnuvmv2qv76yec355gazlr5eky46bpxs4ytawhiem

Address

f17ia7m5mvizrdug3sqtevqw3tifiqvxqr3kdaeuq

Datacap Allocated

1.56PiB

Signer Address

f1oz43ckvmtxmmsfzqm6bpnemqlavz4ifyl524chq

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebvde5d6rvsgrnuvmv2qv76yec355gazlr5eky46bpxs4ytawhiem

dkkapur commented 2 years ago

FYI - looks like the bot is still bugging out and this allocation will not go through because it will exceed the maximum. The maximum is 348 TiB left to get to 2 PiB as requested by the client. Doing this manually now.