search

CHIMEFRB / datatrail-cli

CHIME/FRB Data Management CLI
https://chimefrb.github.io/datatrail-cli/
MIT License
2 stars 0 forks source link

[BUG] Data trail missing events #44

Closed FengqiuAdamDong closed 12 months ago

FengqiuAdamDong commented 1 year ago

Describe the bug There seems to be missing events in data trail, specifically those associated with J1634+44 339081553 339081550 339081551 339081552 339081557 339081556 339081555 339636297 341265154 341265159 341265166 341265169 341265171 341265172 341265175 341265176 341832014 341832013 341832019 341832017

To Reproduce adamdong@notebook1:/arc/projects/chime_frb/adamdong$ datatrail ps chime.event.intensity.raw 339081553

Expected behavior I queried 339081553 and data trail reported nothing

Screenshots image

Desktop (please complete the following information):

Additional context Add any other context about the problem here.

tjzegmott commented 12 months ago

Hi @MWSammons, I'd like you to have a go at figuring out what happened here. This is a pretty typical example of an issue you'll see with Datatrail. Reply to this thread with what you think happened here and tag me if you have any issues.

MWSammons commented 12 months ago

Right, so it appears that all the events listed are missing because they are unregistered, and unregistered because they could not be attached to the dataset GWG_Candidates.J1634+44. The way the error triggered I guess this means that GWG_Candidates.J1634+44 doesn't exist as a dataset and needs to be created, but I'm not sure how to go about that precisely. @tjzegmott any suggestions ?

tjzegmott commented 12 months ago

@MWSammons let's meet tomorrow and create that dataset together. Does 10:00 work?

MWSammons commented 12 months ago

Sounds great, I'll drop by the project office around then.

MWSammons commented 12 months ago

@FengqiuAdamDong we've created a new dataset for GWG_Candidates.J1634+44 with the same policies as GWG_Candidates. Now that the larger dataset exists those unregistered datasets should be processed in a few days over the weekend and be available to you.

FengqiuAdamDong commented 12 months ago

@MWSammons @tjzegmott , There should have been baseband data in this dataset as well. The GWG_Candidates only had intensity data. Will the baseband data follow the same rules?

tjzegmott commented 12 months ago

@FengqiuAdamDong How long do you need the data kept at Minoc for? Since the baseband data is larger it should probably have policies that reflect that. Do you ever collect baseband GWG_Candidates data?

FengqiuAdamDong commented 12 months ago

Ideally, for J1634+44, it should be kept forever. The data rates are very low. We only have 3 baseband dumps since starting ~1 year ago We never dump baseband for GWG_Candidates.

tjzegmott commented 12 months ago

Okay, we'll keep it Minoc for 100 years also.

MWSammons commented 12 months ago

@FengqiuAdamDong Ok, I've created a dataset for the baseband data too.