hpc4cmb / toast

Time Ordered Astrophysics Scalable Tools
Other
43 stars 37 forks source link

drone_map test failing with 4 MPI tasks #766

Open sbiquard opened 3 weeks ago

sbiquard commented 3 weeks ago

Hello,

While running the MPI tests on my laptop after compiling from source, I noticed the drone_map test was failing (see extract below). However the CI tests are passing, the difference being that they only run with 2 MPI tasks whereas I used 4...

test_drone_map (toast.tests.ops_pointing_wcs.PointingWCSTest) ... TOAST INFO: Adding patch "small_patch"
TOAST INFO: Rectangular format
TOAST INFO: Creating '/tmp/tmp6lhvedua'
TOAST INFO: Loading schedule from /tmp/tmp6lhvedua/ground_schedule.txt
TOAST INFO: Loaded 23 scans from /tmp/tmp6lhvedua/ground_schedule.txt totaling 4.25 hours.
[3]error Proc 3: Traceback (most recent call last):
(...)
Proc 3:   File "/home/sbiquard/work/software/toastdev/lib/python3.10/site-packages/toast/ops/pixels_wcs.py", line 413, in _exec
    lnmin, lnmax, ltmin, ltmax = scan_range_lonlat(
Proc 3:   File "/home/sbiquard/work/software/toastdev/lib/python3.10/site-packages/toast/timing.py", line 81, in df
    return f(*args, **kwargs)
Proc 3:   File "/home/sbiquard/work/software/toastdev/lib/python3.10/site-packages/toast/pointing_utils.py", line 163, in scan_range_lonlat
    det_lon, det_lat = center_offset_lonlat(
Proc 3:   File "/home/sbiquard/work/software/toastdev/lib/python3.10/site-packages/toast/pointing_utils.py", line 45, in center_offset_lonlat
    raise ValueError(msg)
Proc 3: ValueError: center_offset dimensions (21300, 2) not compatible with 42600 quaternion values

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
sbiquard commented 3 weeks ago

(this is on the toast3 branch)

tskisner commented 3 weeks ago

I also discovered this bug while making drone maps for SO. It is fixed in #765