search

LArbys / LArCV

Liquid Argon Computer Vision
11 stars 9 forks source link

Supera failure #77

Closed twongjirad closed 7 years ago

twongjirad commented 8 years ago

Seeing a fair amount of Supera jobs failing on the following:

[CRITICAL]<Supera::process_event>^[[00m SuperaCore.inl::L407 Unexpected # of hi-res images: 2 @ plane 0

This refers the following block of code:

// Retrieve cropped full resolution image
auto int_img_v = (::larcv::EventImage2D*)(_larcv_io.get_data(::larcv::kProductImage2D, Form("tpc_int%02d", roi.MCTIndex())));
LARCV_INFO() << "Cropping ROI: " << roi_meta.dump();

auto hires_img = _full_image.crop(roi_meta);
int_img_v->Emplace(std::move(hires_img));
if (int_img_v->Image2DArray().size() > (p + 1)) {
    LARCV_CRITICAL() << "Unexpected # of hi-res images: " << int_img_v->Image2DArray().size() << " @ plane " << p << std::endl;
    throw ::larcv::larbys();
}

Still trying to figure this one out.

drinkingkazu commented 8 years ago

This happens when there are > 2 neutrino interactions in one event. I plan to fix this. Most part of Supera code is designed to handle this actually, but I did not finish everything so I left it as a handled error at the moment. So in that sense it's not a bug.

Our experience is there are several (less than 10) out of ~300 jobs that encounter this problem. We started look into failure modes in initial production and 100% is this case so far. So unless we care this rare occasion I do not think we care too much.

drinkingkazu commented 8 years ago

... and I also assigned myself since I planned to finish implementing handling of 2 neutrinos.

drinkingkazu commented 7 years ago

I think this is handled in new meat slicer. Please re-open otherwise!