ratt-ru / pfb-imaging

Preconditioned forward/backward clean algorithm
MIT License
7 stars 5 forks source link

Typical MeerKAT image size hits the zarr 2 GB chunk limit #69

Closed IanHeywood closed 1 year ago

IanHeywood commented 2 years ago

When trying to grid some visibilities using my standard image (10240 x 10240) and cell (1.1 arcsec) sizes I run into the following because of the double-sized PSF image:

ValueError: Column PSF has a chunk of dimension (16384, 16384) that will exceed zarr's 2GiB chunk limit

8192 x 8192 images also triggers this. 4096 x 4096 is working but I think support for larger images is essential. Is the chunk size tunable?

Thanks.

landmanbester commented 2 years ago

Ah must have broken that recently, should rechunk before writing. Will add a fix for that. The PSF size relative to the image size can be set with --psf-oversize

IanHeywood commented 1 year ago
########################################] | 100% Completed | 19m 40s
INFO      09:33:40 - GRID               | Writing fits
INFO      09:34:45 - GRID               | All done here.

Success, cheers.