Using Northern Tracks v5p2 for stacking, I noticed that some of my signal jobs took way longer than the other ones. Eventually I got the error below. This does not happen frequently. From a batch of 142 jobs only 3 jobs produced this error. When resubmitting the failed jobs, one of them succeeded, the other two failed again.
Traceback (most recent call last):
File "/data/user/ymerckx/Jobs/stacking/stacking_kde.py", line 155, in <module>
(_, _, _, trials) = create_trial_data_file(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ymerckx/github/skyllh/skyllh/core/utils/analysis.py", line 1364, in create_trial_data_file
trials = ana.do_trials(
^^^^^^^^^^^^^^
File "/home/ymerckx/github/skyllh/skyllh/core/analysis.py", line 1218, in do_trials
result_list = parallelize(
^^^^^^^^^^^^
File "/home/ymerckx/github/skyllh/skyllh/core/multiproc.py", line 254, in parallelize
result_list = master_wrapper(
^^^^^^^^^^^^^^^
File "/home/ymerckx/github/skyllh/skyllh/core/multiproc.py", line 227, in master_wrapper
result_list.append(func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^
File "/home/ymerckx/github/skyllh/skyllh/core/analysis.py", line 1153, in do_trial
(n_sig, n_events_list, events_list) = self.generate_pseudo_data(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ymerckx/github/skyllh/skyllh/core/analysis.py", line 1080, in generate_pseudo_data
(n_sig, n_events_list, events_list) = self.generate_signal_events(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ymerckx/github/skyllh/skyllh/core/analysis.py", line 1006, in generate_signal_events
self._sig_generator.generate_signal_events(
File "/home/ymerckx/github/skyllh/skyllh/core/signal_generator.py", line 908, in generate_signal_events
self._draw_valid_sig_events_for_dataset_and_shg(
File "/home/ymerckx/github/skyllh/skyllh/core/signal_generator.py", line 705, in _draw_valid_sig_events_for_dataset_and_shg
signal_event_post_sampling_processing(
File "/home/ymerckx/github/skyllh/skyllh/i3/signal_generation.py", line 384, in signal_event_post_sampling_processing
shg_src_sig_events = shg_sig_events[shg_src_mask]
~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/home/ymerckx/github/skyllh/skyllh/core/storage.py", line 1200, in __getitem__
return self.get_selection(name)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ymerckx/github/skyllh/skyllh/core/storage.py", line 1514, in get_selection
data[fname] = self._data_fields[fname][indices]
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
IndexError: boolean index did not match indexed array along dimension 0; dimension is 1 but corresponding boolean dimension is 2
Using Northern Tracks v5p2 for stacking, I noticed that some of my signal jobs took way longer than the other ones. Eventually I got the error below. This does not happen frequently. From a batch of 142 jobs only 3 jobs produced this error. When resubmitting the failed jobs, one of them succeeded, the other two failed again.