[DataCap Application] <MODRONS' LAW> - <MODRONS'LAW-EUD-1>

[Jennyism]

Version

1

DataCap Applicant

MODRONS' LAW

Project ID

MODRONS'LAW-EUD-1

Data Owner Name

European Union

Data Owner Country/Region

Luxembourg

Data Owner Industry

Government

Website

https://data.europa.eu/

Social Media Handle

https://x.com/EU_opendata

Social Media Type

Twitter

What is your role related to the dataset

Data Preparer

Total amount of DataCap being requested

3 PiB

Expected size of single dataset (one copy)

600 TiB

Number of replicas to store

5

Weekly allocation of DataCap requested

600 TiB

On-chain address for first allocation

f1h53uih27nfesuug76trd7cddszz6yfqzcvumcvi

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

• [X] Use Custom Multisig

Identifier

No response

Share a brief history of your project and organization

At Modrons' Law, we believe that rules and laws are the fundamental principles governing the world, and that the mechanical paradigm is the key to the advancement of civilization. Humanity is progressively deepening its understanding of these principles, categorizing them into various scientific disciplines. As civilization evolves by emulating these principles—whether socially or mathematically—our comprehension of the world's rules is both validated and enhanced. Our mission is to drive this emulation forward in the field of computer science, fostering innovation and progress.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

The European Data organization, led by the EU, offers a broad range of datasets across various fields, such as environment, economy, healthcare, and demographics. These datasets support research, policy-making, and innovation, aiming to increase transparency and stimulate growth through data-driven approaches. The data is designed for easy use across different sectors and regions within the EU.

Where was the data currently stored in this dataset sourced from

Other

If you answered "Other" in the previous question, enter the details here

Official site of the data owner.

If you are a data preparer. What is your location (Country/Region)

None

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

No response

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

No response

Please share a sample of the data

https://data.europa.eu/data/datasets/b0203a76-ab90-4564-9789-9b7c21fb9c65~~1

Confirm that this is a public dataset that can be retrieved by anyone on the Network

• [X] I confirm

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

More than 3 years

In which geographies do you plan on making storage deals

Asia other than Greater China

How will you be distributing your data to storage providers

Shipping hard drives, Lotus built-in data transfer

How did you find your storage providers

Slack, Others

If you answered "Others" in the previous question, what is the tool or platform you used

Off-line events like meet-up, summit.

Please list the provider IDs and location of the storage providers you will be working with.

f03068013-HK
f03035686-GuangDong
f01926635-SiChuang
f01999119-SiChuang
f01422327-Japan
f02252023-Japan
f02252024-Japan
f01989013-Malaysia
f01989014-Malaysia
f01989015-Malaysia
f02105010-Malaysia

How do you plan to make deals to your storage providers

Boost client

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

[datacap-bot[bot]]

Application is waiting for allocator review

[joshua-ne]

Hi @Jennyism, as we have done SP localization verification offline and sample checks, I have also checked online copies of your dataset. It seems to be a new set.

So I think we are good to go for now. But before we start with a test run with 256TiB Datacap, officially, can you confirm that all the SP's you are working with shall store the unsealed files of the sectors and support public retrieval, like SPARK?

[Jennyism]

Thank you for your quick reponse! And yes, I confirm that all the SP's you are working with shall store the unsealed files of the sectors and support public retrieval, like SPARK. Expecting to start onboarding soon.

[datacap-bot[bot]]

Datacap Request Trigger

Total DataCap requested

3 PiB

Expected weekly DataCap usage rate

600 TiB

DataCap Amount - First Tranche

256TiB

Client address

f1h53uih27nfesuug76trd7cddszz6yfqzcvumcvi

[datacap-bot[bot]]

DataCap Allocation requested

Multisig Notary address

Client address

f1h53uih27nfesuug76trd7cddszz6yfqzcvumcvi

DataCap allocation requested

256TiB

Id

56064274-9408-4422-9ade-f8742aa6d80d

[datacap-bot[bot]]

Application is ready to sign

[datacap-bot[bot]]

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacecjfr552aw7tlfi6rzy35nwjxrvnl64egg5vr3o777wu3qrucr632

Address

f1h53uih27nfesuug76trd7cddszz6yfqzcvumcvi

Datacap Allocated

256TiB

Signer Address

f1sfffys4o2w64rdpd3alpmvpvj4ik6x2iyjsjmry

Id

56064274-9408-4422-9ade-f8742aa6d80d

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecjfr552aw7tlfi6rzy35nwjxrvnl64egg5vr3o777wu3qrucr632

[datacap-bot[bot]]

Application is Granted

[joshua-ne]

I have signed 256TiB Datacap to you to start. Do make sure you support public retrieval, or we will not be able to continue allocating more to you. Happy onboarding!

[datacap-bot[bot]]

Client used 75% of the allocated DataCap. Consider allocating next tranche.

[joshua-ne]

trigger:run_retrieval_test method=lassie sp_list=f03068013,f03035686,f01926635,f01999119,f01422327,f02252023,f02252024,f01989013,f01989014,f01989015,f02105010 client=f1h53uih27nfesuug76trd7cddszz6yfqzcvumcvi limit=30

[myfil512]

miner_id	retrieval_rate	retrieval_success_counts	retrieval_fail_counts
f03068013	NA	0	0
f03035686	NA	0	0
f01926635	NA	0	0
f01999119	NA	0	0
f01422327	93%	28	2
f02252023	90%	27	3
f02252024	86%	26	4
f01989013	100%	30	0
f01989014	100%	30	0
f01989015	100%	30	0
f02105010	NA	0	0

[joshua-ne]

It seems the retrieval statistics from SPARK have not been updated yet. But from our retrieval bot, it seems pretty good retrieval rate. At least I am convinced that your collaborating SPs are indeed storing the unsealed dataset. I will support you by signing another 1PiB for now and hopefully, when SPARK updates the result, it will be fine. Otherwise, we might need to work with your SPs to working on supporting SPARK retrieval if needed.

[datacap-bot[bot]]

Application is in Refill

[datacap-bot[bot]]

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedfhhdgbhp74u52um7l53nbflra2iikzqmoztkqdofqfaiqdfw7lu

Address

f1h53uih27nfesuug76trd7cddszz6yfqzcvumcvi

Datacap Allocated

1PiB

Signer Address

f1sfffys4o2w64rdpd3alpmvpvj4ik6x2iyjsjmry

Id

2d3d445d-61ae-482d-a378-a2bf61354d3e

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedfhhdgbhp74u52um7l53nbflra2iikzqmoztkqdofqfaiqdfw7lu

[datacap-bot[bot]]

Application is Granted

[Jennyism]

Thank you for your support! I have stressed this for a LOT of times with our SPs to make sure they store unsealed sectors. Please do not hesitate if there is anything we need to do or change on SPs side.

[datacap-bot[bot]]

Client used 75% of the allocated DataCap. Consider allocating next tranche.

[nicelove666]

checker:manualTrigger

[datacap-bot[bot]]

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report.

[joshua-ne]

checker:manualTrigger

[datacap-bot[bot]]

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

⚠️ 42.86% of Storage Providers have retrieval success rate equal to zero.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report.

[joshua-ne]

From this report, it seems that 4 out of 7 SP's you are working with have good SPARK support. For the other 3, the result is conflicting between our own bot and the SPARK system. I will keep look into this, and maybe your SP's can also do some tests. @Jennyism

[Jennyism]

Thank you for your reminder! I will contact and work with the SP's to see what happens and what can be done from their end.

And good to see that your retrieval bot it being more robust and timely than the SPARK system. It's a very nice complement to the community tooling 👍

[joshua-ne]

trigger:run_retrieval_test method=lassie sp_list=f03068013,f03035686,f01926635,f01999119,f01422327,f02252023,f02252024,f01989013,f01989014,f01989015,f02105010 client=f1h53uih27nfesuug76trd7cddszz6yfqzcvumcvi limit=30 start_datetime="2024-08-01 00:00:00" end_datetime="2024-08-13 00:00:00"

[myfil512]

miner_id	retrieval_rate	retrieval_success_counts	retrieval_fail_counts
f03068013	NA	0	0
f03035686	NA	0	0
f01926635	NA	0	0
f01999119	NA	0	0
f01422327	0%	0	30
f02252023	100%	30	0
f02252024	96%	29	1
f01989013	100%	30	0
f01989014	100%	30	0
f01989015	100%	30	0
f02105010	100%	30	0

[joshua-ne]

checker:manualTrigger

[datacap-bot[bot]]

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

⚠️ 42.86% of Storage Providers have retrieval success rate equal to zero.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report.

[joshua-ne]

There seems inconsistency between SPARK retrieval and our own retrieval result.

[joshua-ne]

With SPARK, f01089013, f02252023, and f02252024 can not be retrieved, however, with own system, all of them are retrievable, instead, f01422327 is not retrievable. From my backend, it shows as blow, maybe it can help your SP's to debug and optimize.

Executing: lassie fetch --progress --providers /ip4/42.125.245.71/tcp/19283/p2p/12D3KooWKbfRMtKyEj3VFiN3HVm8kHAQkHDaAh5FmBA1mAMvhiXt -o /dev/null bafybeich7xqbu6cq4yp6qutj6zrcqomznte4dnwomr54763vi3cug25lce 2024-08-15 10:41:33 - INFO - /root/ns-fil_retrieval_bot/methods/common.py:27:target - Fetching bafybeich7xqbu6cq4yp6qutj6zrcqomznte4dnwomr54763vi3cug25lce from specified provider(s) 2024-08-15 10:41:33 - INFO - /root/ns-fil_retrieval_bot/methods/common.py:27:target - Querying indexer for bafybeich7xqbu6cq4yp6qutj6zrcqomznte4dnwomr54763vi3cug25lce... 2024-08-15 10:41:35 - INFO - /root/ns-fil_retrieval_bot/methods/common.py:27:target - Using the specified storage provider(s): 2024-08-15 10:41:35 - INFO - /root/ns-fil_retrieval_bot/methods/common.py:27:target - 12D3KooWKbfRMtKyEj3VFiN3HVm8kHAQkHDaAh5FmBA1mAMvhiXt, Protocols: [transport-graphsync-filecoinv1] 2024-08-15 10:41:35 - INFO - /root/ns-fil_retrieval_bot/methods/common.py:27:target - Retrieving from [12D3KooWKbfRMtKyEj3VFiN3HVm8kHAQkHDaAh5FmBA1mAMvhiXt] (started-retrieval)... 2024-08-15 10:41:35 - INFO - /root/ns-fil_retrieval_bot/methods/common.py:27:target - Retrieving from [12D3KooWKbfRMtKyEj3VFiN3HVm8kHAQkHDaAh5FmBA1mAMvhiXt] (connected-to-provider)... 2024-08-15 10:41:35 - INFO - /root/ns-fil_retrieval_bot/methods/common.py:27:target - Retrieving from [12D3KooWKbfRMtKyEj3VFiN3HVm8kHAQkHDaAh5FmBA1mAMvhiXt] (proposed)... 2024-08-15 10:41:43 - INFO - /root/ns-fil_retrieval_bot/methods/common.py:48:run_retrieval_command - Timeout reached. Terminating process. 2024-08-15 10:41:55 - INFO - /root/ns-fil_retrieval_bot/methods/common.py:27:target - Retrieval failure for [12D3KooWKbfRMtKyEj3VFiN3HVm8kHAQkHDaAh5FmBA1mAMvhiXt]: retrieval timed out after 20s

[Jennyism]

We are running pretty low on Datacap, and since we look pretty good on retrieval so far and actively trying to solve the flaws, looking forward to your continuous support! Thanks!

[datacap-bot[bot]]

Application is in Refill

[datacap-bot[bot]]

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacebndbcs6fw3mtajrwxh46zhlqneqceq3cprm7hn35jhp7pnbiv66o

Address

f1h53uih27nfesuug76trd7cddszz6yfqzcvumcvi

Datacap Allocated

1PiB

Signer Address

f1sfffys4o2w64rdpd3alpmvpvj4ik6x2iyjsjmry

Id

0e288138-315e-4633-926c-a9a7a0c5c463

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebndbcs6fw3mtajrwxh46zhlqneqceq3cprm7hn35jhp7pnbiv66o

[datacap-bot[bot]]

Application is Granted

[datacap-bot[bot]]

Client used 75% of the allocated DataCap. Consider allocating next tranche.

[Jennyism]

Since some of our current collaborating SP's have changes of plans, we are going to add the following SP's to work with:

f01111110: Japan f01909705: Japan

Thank you!

[gimims]

checker:manualTrigger

[datacap-bot[bot]]

DataCap and CID Checker Report Summary[^1]

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

⚠️ 42.86% of Storage Providers have retrieval success rate equal to zero.

⚠️ 42.86% of Storage Providers have retrieval success rate less than 75%.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report.

[joshua-ne]

Everything seems so far so good, I will continue allocating the rest of the DataCap to you. Hopefully, you can get the rest of the SP's support SPARK, or work with the SPARK team to see what is going on.

[datacap-bot[bot]]

Application is in Refill

[datacap-bot[bot]]

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedm3agqhnurxpee4muli64y66p4snrzhuj2rtlmqkvcpglr4y6mo6

Address

f1h53uih27nfesuug76trd7cddszz6yfqzcvumcvi

Datacap Allocated

768TiB

Signer Address

f1sfffys4o2w64rdpd3alpmvpvj4ik6x2iyjsjmry

Id

81ce63b5-3bf4-48a7-87fd-0268aa13ccb1

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedm3agqhnurxpee4muli64y66p4snrzhuj2rtlmqkvcpglr4y6mo6

[datacap-bot[bot]]

Application is Granted

[datacap-bot[bot]]

Application is Completed

[joshua-ne]

Hi can you please submit your dataset card by the end of next week. Thank you. This will be mandatory for the following allocation of datacap. It would be appreciated even though you have finished this application.

https://github.com/joshua-ne/FIL_DC_Allocator_1022_Dataset_Card/issues/new/choose