araichoor
commented
1 year ago if of any usefulness in the future, here are the lines of code I used:
first grab all exposures with TILEID > 0 since Sept. 2022:
proddir = "/global/cfs/cdirs/desi/spectro/redux/daily"
ds = []
for month in [202209, 202210, 202211, 202212]:
mydir = os.path.join(proddir, "exposure_tables", str(month))
fns = sorted(glob(os.path.join(mydir, "exposure_table_{}??.csv".format(month))))
for fn in fns:
d = Table.read(fn)
d = d[d["TILEID"] > 0]
for key in ["BADCAMWORD", "BADAMPS"]:
d[key] = d[key].astype(str)
ds.append(d)
d = vstack(ds)then read the 20221031-20221112 bad guiding exposures (and assess that those have LASTSTEP=="ignore":
badexps = Table.read("/global/cfs/cdirs/desi/users/schlafly/misc/badguidingtileexp.ecsv")
#
for i in range(len(badexps)):
j = np.where(d["EXPID"] == badexps["EXPID"][i])[0][0]
if d["LASTSTEP"][j] != "ignore":
print(d["EXPID"][j], d["NIGHT"][j], d["LASTSTEP"][j], d["COMMENTS"][j])this returns those two exposures, don t know if that matters:
152853 20221110 skysub dither seq|
152855 20221110 skysub dither seq|then reads the exposures-daily.csv exposures with EFFTIME_SPEC>0:
e = Table.read("/global/cfs/cdirs/desi/spectro/redux/daily/exposures-daily.csv")
e = e[(e["NIGHT"] > 20220900) & (e["EFFTIME_SPEC"] > 0)]and now I check for each (night, tileid) if there is a folder and a tile-qa*fits, if we expect one:
for night in np.unique(e["NIGHT"]):
for tileid in np.unique(e["TILEID"][e["NIGHT"] == night]):
# exposures for that night, tileid)
expids = e["EXPID"][(e["TILEID"] == tileid) & (e["NIGHT"] == night)]
# are all the exposures bad guiding? (if yes, we don t expect the folder to exist)
allbad = np.in1d(expids, badexps["EXPID"]).sum() == len(expids)
# recording the laststep, to verify that those don t all are "ignore"
laststeps, counts = np.unique(d["LASTSTEP"][np.in1d(d["EXPID"], expids)], return_counts=True)
all_laststeps = ",".join(["{}={}".format(laststep, count) for laststep, count in zip(laststeps, counts)])
# folder and tileqa file we check
mydir = os.path.join(proddir, "tiles", "cumulative", str(tileid), str(night))
fn = os.path.join(mydir, "tile-qa-{}-thru{}.fits".format(tileid, night))
if not allbad:
if not os.path.isdir(mydir):
print(mydir, all_laststeps)
else:
if not os.path.isfile(fn):
print(fn, all_laststeps)
marcelo-alvarez
akremin
if I didn t mistake, it looks like some folders or tile-qa files are missing in the
daily/tiles/cumulativefolder. this issue is a "enhanced" follow-up of https://desisurvey.slack.com/archives/C01HNN87Y7J/p1670109389351019.from spot-checking https://data.desi.lbl.gov/desi/spectro/redux/daily/run/dashboard/zdashboard.html for few of the cases, it is correctly reported there that files are missing. we apparently have not been careful in monitoring that zdashboard: maybe we should verify that zdashboard page before announcing a night processed?
I did check for nights since 20220901, and account for the bad guiding blacklist from https://github.com/desihub/desisurveyops/issues/74.
I find that the following folders are missing:
and the following tileqa files are missing: