search

dkirkby / bossdata

Tools for accessing SDSS BOSS data
MIT License
1 stars 3 forks source link

DR14 and mixed BOSS/eBOSS data access #126

Open ixkael opened 5 years ago

ixkael commented 5 years ago

Hi there,

bossdata is an awesome tool and I have been using it successfully to work with BOSS (lite) DR12 spectra. However, I would like to now use DR14 spectra, both the reprocessed BOSS data and the new eBOSS data. I was able to make some progress by applying some tweaks, including defining BOSS_SAS_PATH=/sas/dr14/eboss BOSS_DATA_URL=http://data.sdss.org BOSS_REDUX_VERSION=v5_10_0

However, only the BOSS spectra seem to be accessible for download that way. I am misunderstanding something (aren't the eBOSS spectra available at this stage?) or mis-using the code?

I would appreciate any help or advice! Thank you, Boris

dkirkby commented 5 years ago

The only difference with my DR14 settings (from here) is:

BOSS_DATA_URL=http://dr14.sdss.org

but I'm not sure if that makes a difference. Perhaps the DR14 lite files you are looking for have not been released? Can you give an example URL of a lite file under:

https://dr14.sdss.org/sas/dr14/eboss/spectro/redux/v5_10_0/spectra/lite/

that bossdata cannot access?

ixkael commented 5 years ago

thank you for this super quick reply.

I tried using this BOSS_DATA_URL but am getting the same issue. A lot of files are missing. For instance:

/sas/dr14/eboss/spectro/redux/v5_10_0/spectra/lite/0584/spec-0584-52049-0512.fits

I should also say: I am running bossfetch on a file obtained with the following query:

bossquery --what OBJ_ID,PLATE,MJD,FIBER,RA,DEC,Z --verbose --quasar-catalog --save qso_cat.fits --max-rows 100000000

Could it be that the file names obtained from this query are incorrect? Or the files have not been released, as you suggested?

dkirkby commented 5 years ago

That 3-digit plate number (0584) looks fishy and the corresponding MJD (52049) is 20-May-2001, so this must be from the original SDSS quasar sample. It looks like those are no longer being carried forward in new releases but, if you really want to include them in your sample, you should be fine to take them from an older release.

ixkael commented 5 years ago

Ah thank you! I digged into that (MJD and spectro source) and figured out an inconsistency in the way I was downloading and reading the files. It is mostly resolved now - I am getting most of the SDSS, BOSS, and eBOSS files correctly, from v5_10_0.

I still have 80k files missing from the 526k quasars though. They seem to be from SDSS or BOSS with fairly random MJD. The targeting flags were not very informative either. Here are a few examples:

sas/dr14/eboss/spectro/redux/v5_10_0/spectra/lite/1086/spec-1086-52525-0162.fits sas/dr14/eboss/spectro/redux/v5_10_0/spectra/lite/2327/spec-2327-53710-0112.fits sas/dr14/eboss/spectro/redux/v5_10_0/spectra/lite/2284/spec-2284-53708-0104.fits sas/dr14/eboss/spectro/redux/v5_10_0/spectra/lite/2583/spec-2583-54095-0351.fits sas/dr14/eboss/spectro/redux/v5_10_0/spectra/lite/2881/spec-2881-54502-0442.fits sas/dr14/eboss/spectro/redux/v5_10_0/spectra/lite/2604/spec-2604-54484-0221.fits

Any idea what those could be?