3Dsynth Issue: 'ValueError: Maximum allowed size exceeded'

[obiwanjacobi47]

Sorry to bother you, but I ran into this issue when running do_RMsynth_3D.py ...

When running the line: ' ./do_RMsynth_3D.py "myStokesQfitsfile" "myStokesUfitsfile" "MyFreqdatfile" ' I receive the error: ' ValueError: Maximum allowed size exceeded '

Terminal in full:

' (base) Jacobs-MacBook-Pro:RMtools_3D jacobaskew$ ./do_RMsynth_3D.py /Users/jacobaskew/Desktop/CSIRO/Data/P2G24/J093120+582050/Images/J093120+582050_Q_cm_pb_cube_sub.fits /Users/jacobaskew/Desktop/CSIRO/Data/P2G24/J093120+582050/Images/J093120+582050_U_cm_pb_cube_sub.fits /Users/jacobaskew/Desktop/CSIRO/Data/P2G24/J093120+582050/data/P2G24_J093120+582050_freqs.dat Traceback (most recent call last): File "./do_RMsynth_3D.py", line 632, in main() File "./do_RMsynth_3D.py", line 616, in main not_rmsf = args.not_RMSF) File "./do_RMsynth_3D.py", line 129, in run_rmsynth phiArr_radm2 = np.linspace(startPhi_radm2, stopPhi_radm2, nChanRM) File "<__array_function__ internals>", line 6, in linspace File "/anaconda3/lib/python3.7/site-packages/numpy/core/function_base.py", line 143, in linspace y = _nx.arange(0, num, dtype=dt).reshape((-1,) + (1,) * ndim(delta)) ValueError: Maximum allowed size exceeded (base) Jacobs-MacBook-Pro:RMtools_3D jacobaskew$ '

The data file sizes are as follows: (I ran ls -l *)

' -rw-r--r-- 1 jacobaskew staff 225288000 6 Dec 15:40 J093120+582050_Q_cm_pb_cube_sub.fits '

' -rw-r--r-- 1 jacobaskew staff 225288000 6 Dec 15:40 J093120+582050_U_cm_pb_cube_sub.fits '

If you need more information or there is anything else I can do to speed up the process of solving this issue then let me know!

Thanks so much I hope this is very simple and I am doing a silly!

[Cameron-Van-Eck]

Hi obiwan:

Sorry to hear you're having this problem. Can you send me your frequency file and the FITS header from one of your files, do help me reproduce this? Easiest way is probably to paste them into a reply here.

In the mean time, I can try to guess where the problem is. From the traceback, it looks like it's trying to create too many slices along the Phi axis. Try manually setting the -l PHIMAX flag (how far out in Faraday depth to compute) and -d DPHI flag (what step size to use in Faraday depth) to values that seem reasonable for your data, to see if that solves the problem.

Cheers, Cameron

[obiwanjacobi47]

Header for Stokes Q: ' SIMPLE = T / conforms to FITS standard
BITPIX = -32 / array data type
NAXIS = 3 / number of array dimensions
NAXIS1 = 1600
NAXIS2 = 1600
NAXIS3 = 22
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H BUNIT = 'JY/BEAM ' / Units are in Jansky per beam
BMAJ = 0.00289906057544038
BMIN = 0.00140187047310063
BPA = 86.9873742674554
EQUINOX = 2000. / J2000
BTYPE = 'Intensity'
TELESCOP= 'EVLA '
OBSERVER= 'Tessa Vernstrom'
OBJECT = 'J093120+582050'
ORIGIN = 'WSClean ' / W-stacking imager written by Andre Offringa
CTYPE1 = 'RA---SIN' / Right ascension angle cosine
CRPIX1 = 801
CRVAL1 = 142.836874999451
CDELT1 = -0.000152777777777778
CUNIT1 = 'deg '
CTYPE2 = 'DEC--SIN' / Declination angle cosine
CRPIX2 = 801
CRVAL2 = 58.3474444455887
CDELT2 = 0.000152777777777778
CUNIT2 = 'deg '
CTYPE3 = 'FREQ ' / Central frequency
CRPIX3 = 1.
CRVAL3 = 1023500000.
CDELT3 = 32000000.0
CUNIT3 = 'Hz '
CTYPE4 = 'STOKES '
CRPIX4 = 1.
CRVAL4 = 2.
CDELT4 = 1.
CUNIT4 = ' '
SPECSYS = 'TOPOCENT'
DATE-OBS= '2019-04-13T08:24:02.4'
WSCDATAC= 'CORRECTED_DATA'
WSCVDATE= '2018-06-11'
WSCVERSI= '2.6 '
WSCWEIGH= 'Briggs''(0.5)'
WSCENVIS= 277644.3136985
WSCFIELD= 0.
WSCGAIN = 0.1
WSCGKRNL= 7.
WSCIMGWG= 1.09446165984584E+17
WSCMAJOR= 4.
WSCMGAIN= 0.75
WSCMINOR= 8000.
WSCNEGCM= 1.
WSCNEGST= 0.
WSCNITER= 8000.
WSCNORMF= 1.09446165984584E+17
WSCNVIS = 858242.
WSCNWLAY= 32.
WSCTHRES= 0.
WSCVWSUM= 1.81043047987216E+17
HISTORY wsclean -weight briggs 0.5 -log-time -no-update-model-required -abs-mem HISTORY 100 -niter 8000 -size 6720 7820 -multiscale -multiscale-scales 0,5,10 -s HISTORY cale 0.55asec -fit-beam -weighting-rank-filter 3.0 -mgain 0.75 -auto-mas HISTORY k 5 -auto-threshold 2 -join-channels -channels-out 22 -nwlayers 32 -pol HISTORY QU -joinpolarizations -squared-channel-joining -spws 0,1,4,5,6,7,10,11,1 HISTORY 2,13,14 -name WSclean/J093120+582050/wsc_J093120+582050_r05_pol J093120+ HISTORY 582050.ms
'

Frequency File: ' 1.0235 1.0555 1.0875 1.1195 1.2795 1.3115 1.3435 1.3755 1.4075 1.4395 1.4715 1.5035 1.6635 1.6955 1.7275 1.7595 1.7915 1.8235 1.8555 1.8875 1.9195 1.9515 '

I will attempt to follow your wisdom while you have a go at this, tyvm for assistance!

[Cameron-Van-Eck]

Aha, I think I see the problem. The frequencies need to be in Hz, and it looks like you've put it in GHz. So RM-tools thinks you have the lowest frequency radio data ever, along with super narrow channels, so it thinks your Faraday depth resolution is amazing (thus needing a very small DPHI), while having sensitivity to very large RMs (thus needing a very large PHI_MAX). So the number of needed phi-planes is crazy large, larger than numpy wants to handle.

I think if you rescale your frequencies everything should work out.

Cheers, Cameron

[obiwanjacobi47]

It is working now to my knowledge, thanks for such a timely and succinct response! I will come back and annoy next time I break it aha!

Cheers Jacob

[Cameron-Van-Eck]

Great! Happy to help.

Cameron