search

filecoin-project / filecoin-plus-large-datasets

Hub for client applications for DataCap at a large scale
109 stars 62 forks source link

[DataCap Application] <SXX Future Data> - LAMOST DR5 Public Dataset #1992

Open sxxfuture-official opened 1 year ago

sxxfuture-official commented 1 year ago

Data Owner Name

SXX Future Data

What is your role related to the dataset

Data Preparer

Data Owner Country/Region

China

Data Owner Industry

Not-for-Profit

Website

https://www.sxxfuture.com/

Social Media

https://twitter.com/sxxfuture
https://medium.com/@sxxfuture.official
https://discord.gg/GFsJaUFS9K
https://weibo.com/u/7801654981

Total amount of DataCap being requested

15PiB

Weekly allocation of DataCap requested

500TiB

On-chain address for first allocation

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

Share a brief history of your project and organization

Focusing on distributed big data storage, disaster data protection, encryption algorithm research, encrypted data product application and Web3.0 new-generation Internet research and development.
SXX Future Data provides government enterprises and individual users with products and service system with data value as the core to meet the ever-expanding demand for mass data storage, management and application.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

The Large Sky Area Multi-Object Fiber Spectroscopic Telescope (LAMOST) is a Chinese national scientific research facility operated by the National Astronomical Observatories, Chinese Academy of Sciences. It is a special reflecting Schmidt telescope with 4000 fibers in a field of view of 20 deg2
 in the sky. Until July 2017, LAMOST has completed its pilot survey which was launched in October 2011 and ended in June 2012, and the first five years of regular survey which was initiated on September 2012. After this six-year-survey, we totally obtain 9,026,365 spectra, which consist of stars, galaxies, quasars and other unknown objects[1−7]
. Now, the fifth data release (DR5) has published online (http://dr5.lamost.org/), and released data products include：

Spectra. - IIn general, there are 9,026,365 flux- and wavelength-calibrated, sky-subtracted spectra in DR5, including 8,183,160 stars, 152,863 galaxies, 52,453 quasars, and 637,889 unknown objects, and these spectra cover the wavelength range of 3690-9100 angstrom with a resolution of 1800[2−3]
 at the 5500 angstrom.
Spectroscopic Parameters Catalogs. - In this data release, six spectroscopic parameters catalogs are also published，and they are the LAMOST general catalog, the A, F, G and K type star catalog, the A type star catalog, the M dwarf catalog, the observed plate information catalog, and the input catalog respectively. In the LAMOST general catalog, it includes 36 columns of basic spectroscopic information, for example, right ascension, declination, signal to noise ratio, magnitude, classification and redshift, which are also provided by the A, F, G and K type star catalog, the A type star catalog, and the M dwarf catalog. These three catalogs also provide other parameters, for example, atmospheric parameters (effective temperature, surface gravity, and metallicity), spectral line indices, line widths, the metallicity sensitive parameter, and the magnetic activity flag. In addition, the observed plate information catalog mainly contains nine basic plate information for all published plates, and the input catalog includes 24 basic fields mentioned above and three new fields which are not included in above catalogs.

http://dr5.lamost.org/v3/doc/data-production-description

Guoshoujing Telescope (the Large Sky Area Multi-Object Fiber Spectroscopic Telescope LAMOST) is a National Major Scientific Project built by the Chinese Academy of Sciences. Funding for the project has been provided by the National Development and Reform Commission. LAMOST is operated and managed by the National Astronomical Observatories, Chinese Academy of Sciences.

Where was the data currently stored in this dataset sourced from

Other

If you answered "Other" in the previous question, enter the details here

http://dr5.lamost.org/

How do you plan to prepare the dataset

lotus, singularity, graphsplit

If you answered "other/custom tool" in the previous question, enter the details here

No response

Please share a sample of the data

http://dr5.lamost.org/v3/sas/catalog/
http://dr5.lamost.org/v3/sas/fits/20111024/F5902/
http://dr5.lamost.org/v3/sas/fits/20111024/F5907/
http://dr5.lamost.org/v3/sas/fits/20111024/F5909/
http://dr5.lamost.org/v3/sas/png/20111024/F5902/
http://dr5.lamost.org/v3/sas/png/20111024/F5907/
http://dr5.lamost.org/v3/sas/png/20111024/F5909/
http://dr5.lamost.org/v3/sas/sky/20111024/

Confirm that this is a public dataset that can be retrieved by anyone on the Network

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

2 to 3 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, North America, Europe

How will you be distributing your data to storage providers

HTTP or FTP server, Shipping hard drives

How do you plan to choose storage providers

Slack, Partners

If you answered "Others" in the previous question, what is the tool or platform you plan to use

No response

If you already have a list of storage providers to work with, fill out their names and provider IDs below

f02188881
f01986236
f02169409
f01986262
f02144581

How do you plan to make deals to your storage providers

Boost client, Lotus client, Singularity

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

large-datacap-requests[bot] commented 1 year ago

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!
large-datacap-requests[bot] commented 1 year ago

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

sxxfuture-official commented 1 year ago

image

sxxfuture-official commented 1 year ago

The public time node table of its data. http://www.lamost.org/lmusers/ image

simonkim0515 commented 1 year ago

It appears that there are currently two applications open for the same project: application #1992 and #1993. Going forward, kindly close one of the applications and submit a single application requesting the full DataCap amount.

sxxfuture-official commented 1 year ago

OK, I will modify it.

simonkim0515 commented 1 year ago

@sxxfuture-official It appears that you have submitted applications for 10PiB in issue #1992 and another 10PiB in issue #1993, resulting in a combined total of 20PiB. Can you clarify the reason behind the revised application requesting a reduced capacity of 15PiB. Additionally, could you kindly provide details regarding the dataset size and the number of replicas that will be necessary?

sxxfuture-official commented 1 year ago

@simonkim0515 Until July 2017, LAMOST has completed its pilot survey which was launched in October 2011 and ended in June 2012, and the first five years of regular survey which was initiated on September 2012. After this six-year-survey, we totally obtain 9,026,365 spectra. The fifth data release (DR5) has published online (http://dr5.lamost.org/), and released data products include：

The main data occupation is spectral image data, in order to accurately retrieve the CID of each image, we will not compress the data before sealing. The uncompressed data size of each spectral data is ≈ 205MByte.

Originally we planned to save 6-7 replicas to different SPs. 205MB 9,026,365 Spectra / 18GB per sector 6 replicas = 601757 sectors = 18.36 PiB DataCap

So I applied for two 10PiB LDNs for the first time, but you asked me to close one of them and only keep one LDN, but the maximum capacity of a single LDN is only 15PiB, so I can only reduce the number of replicas for some data during the sealing process.

simonkim0515 commented 1 year ago

@sxxfuture-official Thank you for providing an explanation. It is worth noting that the DataCap application limit is not limited to 15PiB. Applicants have the option to apply for a higher amount by selecting the "More Than 15PiB" option, which will prompt contact from @Kevin-FF-USA.

sxxfuture-official commented 1 year ago

@simonkim0515 Thank you very much for your reminder ! At the current stage, 15PiB is enough to save 5-6 replicas to avoid complicating the process. The current application quota is acceptable.

Sunnyiscoming commented 1 year ago

Could you send an email to filplus-app-review@fil.org with your official domain in order to confirm your identity? Email name should includes the issue id #1992.

sxxfuture-official commented 1 year ago

@Sunnyiscoming The email has been sent, please check~ image

Sunnyiscoming commented 1 year ago

Datacap Request Trigger

Total DataCap requested

15PiB

Expected weekly DataCap usage rate

500TiB

Client address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

DataCap allocation requested

250TiB

Id

c88af20a-820a-4d81-b2f8-fcde9f6f8111

nj-steve commented 1 year ago

Hello @sxxfuture-official How much data have you prepared? How many copies do you plan to send?

sxxfuture-official commented 1 year ago

@nj-steve

The main data occupation is spectral image data, in order to accurately retrieve the CID of each image, we will not compress the data before sealing. The uncompressed data size of each spectral data is ≈ 205MByte.

Now, we planned to save 5-6 replicas to different SPs. 205MB 9,026,365 Spectra / 18GB per sector 6 replicas = 601757 sectors = 18.36 PiB DataCap

nj-steve commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebc57vbxm5l763r3ixvyj2mwc5pecg4rbw6sqr5katcofqgs27jfq

Address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

Datacap Allocated

250.00TiB

Signer Address

f1xx6555qijma7igpnjspyvdunc4vfxkawnpqy5ii

Id

c88af20a-820a-4d81-b2f8-fcde9f6f8111

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebc57vbxm5l763r3ixvyj2mwc5pecg4rbw6sqr5katcofqgs27jfq

kernelogic commented 1 year ago

In support for public dataset, especially something outside of AWS S3

kernelogic commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceco4ln3setfcidyz2aobcm43cv4ie3kdqtzrg3r5xgjtdkmkz6gd2

Address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

Datacap Allocated

250.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

c88af20a-820a-4d81-b2f8-fcde9f6f8111

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceco4ln3setfcidyz2aobcm43cv4ie3kdqtzrg3r5xgjtdkmkz6gd2

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

sxxfuture-official commented 1 year ago

@simonkim0515 @galen-mcandrew Data is being sealed continuously

sxxfuture-official commented 1 year ago

image

data-programs commented 1 year ago
KYC

This user’s identity has been verified through filplus.storage

cryptowhizzard commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 2

Multisig Notary address

f02049625

Client address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

DataCap allocation requested

500TiB

Id

8f69ef04-a2ea-422c-a8d2-06fc8fdac3c7

large-datacap-requests[bot] commented 1 year ago

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

Rule to calculate the allocation request amount

100% of weekly dc amount requested

DataCap allocation requested

500TiB

Total DataCap granted for client so far

250TiB

Datacap to be granted to reach the total amount requested by the client (15PiB)

14.75PiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
5943 3 250TiB 33.65 59.81TiB
cryptowhizzard commented 1 year ago

Hi @sxxfuture-official

Seems you sealed 62% of this datacap with your own miners. Can you explain us why this is?

Scherm­afbeelding 2023-07-05 om 16 05 32

Secondly none of your data is retrievable.

As notary you need to have an example function for the community and clean sheet. What is your plan here?

sxxfuture-official commented 1 year ago

@cryptowhizzard Because it is the first round of quota, our Miner packaging will be faster, and other SPs will follow the pace of packaging. Regarding the retrieval problem, we will definitely solve this problem. More SPs is on the way.

Since we have been using lotus publish storage deals before, the technical team has been studying how to support http retrieval, and I believe it will be improved in the near future.

sxxfuture-official commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

Bitengine-reeta commented 1 year ago

@sxxfuture-official According to the results of the Retrieval Report, only one (f01964215) of the current 4 Miners can be part retrieved. What is the situation of the other 3 miners? Is there a plan?

sxxfuture-official commented 1 year ago

@Bitengine-reeta Up to now, our SPs have solved the retrieval of most of the data. The results of my lotus test are as follows. The update of RetrieveBot's results may need to wait. I hope you can provide signature support. We will gradually increase the retrieval ratio and http retrieval in the following process .

image image image

Bitengine-reeta commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

Bitengine-reeta commented 1 year ago

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceaxbeztowwv3gwzhawlevtww2yw7x6mtxyahigsq2xn3zjx2cogcc

Address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

Datacap Allocated

500.00TiB

Signer Address

f1jyvhxp4kmwreo22ke4itspraznpudw3uqaink5i

Id

8f69ef04-a2ea-422c-a8d2-06fc8fdac3c7

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceaxbeztowwv3gwzhawlevtww2yw7x6mtxyahigsq2xn3zjx2cogcc

ipollo00 commented 1 year ago

Hi there, @sxxfuture-official How's your HTTP retrieval going? Will you keep inviting more sps for your project?

sxxfuture-official commented 1 year ago

@ipollo00 HTTP retrieval need boost sever, since lotus has been used before, this part still needs time to be overcome, but we will support it eventually. More SPs will be added soon, sure to include the following: f02144581, f02169409, f02169395, f02169396

ipollo00 commented 1 year ago

Fine, Those sps can be reached. Will catch up with the latest update.

ipollo00 commented 1 year ago

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceanywvnem5v7gmahdjrgglmqotx5c7gy3vd7lxa42z2ehfkyxfkm4

Address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

Datacap Allocated

500.00TiB

Signer Address

f1n5wlrrhoxpkgwij25xrtt7w7g2k3fhbthmdn6ri

Id

8f69ef04-a2ea-422c-a8d2-06fc8fdac3c7

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceanywvnem5v7gmahdjrgglmqotx5c7gy3vd7lxa42z2ehfkyxfkm4

herrehesse commented 1 year ago

@sxxfuture-official You state BOOST will be added later, but graphsync should always work as long as the miners have retrievals enabled.

Your application has: Overall Graphsync retrieval success rate: 2.33%

How are you contacting your SP's to get this up to 20-30--->100% ?

sxxfuture-official commented 1 year ago

@herrehesse Yes, we are contacting all related SPs to optimize the retrievability issue, the situation will improve in the near future

cryptowhizzard commented 1 year ago

Scherm­afbeelding 2023-07-21 om 15 38 14

Please correct the sealing percentage. You are > 50% to your own miners.

sxxfuture-official commented 1 year ago

@cryptowhizzard The data has been sent to other SPs through the hard disk. Since the data transfer takes time, the seal progress of our own SPs is faster, and this will be resolved soon.

github-actions[bot] commented 1 year ago

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

sxxfuture-official commented 1 year ago

checker:manualTrigger

filplus-checker-app[bot] commented 1 year ago

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.

sxxfuture-official commented 1 year ago

under sealing

large-datacap-requests[bot] commented 1 year ago

DataCap Allocation requested

Request number 3

Multisig Notary address

f02049625

Client address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

DataCap allocation requested

1000.0TiB

Id

b6c9fa92-d562-4215-81ca-6659228eae4a