search

SIMPLE-AstroDB / SIMPLE-db

BSD 3-Clause "New" or "Revised" License
11 stars 21 forks source link

Ingest byw #435

Closed LishaRamon closed 7 months ago

LishaRamon commented 7 months ago

Ingesting the sources from Rothermich 2024

Link to relevant issue: Closes # 428

For data ingests:

LishaRamon commented 7 months ago

Line 17 Error: "astropy.io.ascii.core.InconsistentTableError: Number of header columns (9) inconsistent with data columns (7) at data line 0"

Header values: ['Source', 'RA', 'Dec', 'Epoch', 'Equinox', 'Shortname', 'Reference', 'Other_ref', 'Comments'] Data values: ['CWISE J000021.45-481314.9', '0.0893808', '-48.2208077', '2015.4041', 'ICRS', 'CWISE 0000-4813', 'Rothermich et al. Submitted']

I understand how it's seeing there's missing data under two of the header columns but not sure how to go about debugging for it to recognize that having a missing value is okay.

kelle commented 7 months ago

I think you should use astropy.io.ascii instead of astropy.Table. It looks like ascii.read does a better job of handling empty cells:

https://docs.astropy.org/en/stable/io/ascii/read.html

LishaRamon commented 7 months ago

Line 17 Error Update: ValueError: ASCII format 'ascii' not in allowed list ['aastex', 'basic', 'cds', 'commented_header', 'csv', 'daophot', 'ecsv', 'fast_basic', 'fast_commented_header', 'fast_csv', 'fast_no_header', 'fast_rdb', 'fast_tab', 'fixed_width', 'fixed_width_no_header', 'fixed_width_two_line', 'html', 'ipac', 'latex', 'mrt', 'no_header', 'qdp', 'rdb', 'rst', 'sextractor', 'tab']

How would I find/recognize the correct 'format'? I assumed csv since it's an excel file, but it breaks the code further.

kelle commented 7 months ago

Here's a video to help you format the link to the Google sheet. You don't need to use pandas, just need to get the URL correct:

https://youtu.be/t6WSY9D_ORQ?feature=shared

kelle commented 7 months ago

If you can't figure out how to read the Google sheet, just use the CSV file from the issue.

LishaRamon commented 7 months ago

Since ascii.read attempted to read data from local files rather from URLs, by using a library like 'requests' it helped get data and create an object that can be passed to ascii.read to understand

kelle commented 7 months ago

Great work!

LishaRamon commented 7 months ago

Not sure why the json files didnt update when I added the changes, but they didn't show up for me to push new ones