No PNG output for fits input ?

[hjmcc]

hi @ryanhausen thanks for making this great software available to everyone ! I am using FITSMAP to make mosaics for COSMOS. It mostly works great, and is super-fast.

However, when I upgraded to the latest version (0.10.1) I noticed that no input FITS files were converted to PNGs (those output directories were empty). I tried both python 3.9 and 3.10 in an empty Conda environment...couldn't get it to work. I dropped back to an earlier version and it worked again.. Thanks!

[ryanhausen]

@hjmcc thanks for bringing this to my attention!

There was some major changes in 10.1 to help with larger image files. Can you give me a sample file (a google drive link or point me to the public release) and the code that you used?

Thanks!

[hjmcc]

Thanks @ryanhausen! I just checked again with some public data and I can confirm that indeed with a simpler configuration generation of PNG tiles from FITS files doesn't seem to be working (output tile directories are empty). I did the following with fitsmap 0.10.1 installed an empty python environment :

from fitsmap import convert

convert.dir_to_map("./forfitsmap-cosmos", \
                   out_dir="/n07data/hjmcc/exchg/COSMOS/tests/", \
                   procs_per_task = 2,\
                   task_procs = 2 ,\
                   catalog_delim = ',',\
                   norm_kwargs=dict(stretch="linear", min_cut = -0.05, max_cut = 0.1))

that file that I am trying to tile is here:

https://exchg.calet.org/hjmcc/tests/UVISTA_Ks_19_11_18_allpaw_skysub_015_dr4_rc_v2.fits

Apologies if it is too large, I can try with a smaller file. Thanks again for making this software available!

[ryanhausen]

@hjmcc perfect thanks! I am pulling the file now and will confirm the issue. Will hopefully have a fix soon!

[ryanhausen]

@hjmcc I confirmed the bug. I will work on a fix and will comment in here when it's released. In the mean time, I have a work around that should help. Change your code to the following:

from fitsmap import convert

convert.dir_to_map("./forfitsmap-cosmos", \
                   out_dir="/n07data/hjmcc/exchg/COSMOS/tests/", \
                   procs_per_task = 2,\
                   task_procs = 2 ,\
                   catalog_delim = ',',\
                   image_engine=convert.IMG_ENGINE_MPL, # Here's the new argument 
                   norm_kwargs=dict(stretch="linear", min_cut = -0.05, max_cut = 0.1))

The default image_engine in versions previous to 0.10.1 was convert.IMG_ENGINE_MPL where MPL means matplotlib. This is needed to nicely convert 2D fits data into the PNG format. However, in testing, I noticed that convert.IMG_ENGINE_PIL (where PIL means pillow) performs faster for PNG images, and after the deepest zoom all the zoom levels are recursively built using PNG images (new in 0.10.1, significant mem/speed improvement). So, I changed the default image_engine to be pillow. However, the behavior you observed is definitely a bug. You'll need to use image_engine=convert.IMG_ENGINE_MPL going forward anyway because it's required for the simple_norm scaling. I'll include this in the documentation as well.

[hjmcc]

@ryanhausen thank you for the rapid response! I understand better now. I tried your fix, and indeed the PNG files are created. Unfortunately, it seems the output is corrupted. Take a look here: https://exchg.calet.org/hjmcc/COSMOS/tests/. I guess I will stand-by for the proper fix. I used the following command:

convert.dir_to_map("./forfitsmap-cosmos", \
                   out_dir="/n07data/hjmcc/exchg/COSMOS/tests/", \
                   procs_per_task = 8,\
                   task_procs = 2 ,\
                   catalog_delim = ',',\
    image_engine=convert.IMG_ENGINE_MPL,\
                   norm_kwargs=dict(stretch="linear", min_cut = -1.5, max_cut = 5))

I notice the output seems to change based on the number procs used ... something wrong on my side? Were you able to tile the whole mosaic?

thanks again for your help !

[ryanhausen]

@hjmcc Yep, I see the same issue with larger images. I tested on a smaller image because i was a away from the computer that I downloaded your sample image. The issue you're experiencing needs a proper fix and is related to the new parallelization implementation. I need to restructure some code pertaining to matplotlib globals.

I started a branch here: https://github.com/ryanhausen/fitsmap/tree/fits-image-scaling-issue-67

I will start working on it, hopefully will have something released to pypi in 1-2 weeks. Will follow-up here when it's merged and pushed.

Thanks for finding this! and for your willingness to test!

[hjmcc]

@ryanhausen great, no worries! I suspected it was something with parallelisation because I also saw that the results changed when I changed the number of processors. No worries, I will stick with the old version.

Just one other thing I noticed -- it would be nice if possible to be able to set the cut levels independently for each FITS image, if that's not already the case.

[ryanhausen]

@hjmcc I will look into per file norm_kwargs. I can see how it would be helpful for multiple fits files. Thanks for the suggestion!

[ryanhausen]

@hjmcc I have pushed a new version of fitsmap 0.10.2 to pypi. Can you give a try when you have a chance?

A few notes:

First, I removed the image_engine kwarg that I told you to use. the original way you wrote it should work fine.

Second, I added per image norm_kwargs. You can assign the norm_kwargs per image by using a dictionary of file names (not whole paths) for example:

from fitsmap import convert

paths_to_files = [
    "fits_images/F125W.fits",
    "fits_images/F160W.fits",
]

convert.files_to_map.(
    paths_to_files,
    out_dir="path/to/data/map",
    cat_wcs_fits_file="path/to/header_file.fits",
    norm_kwargs={
        "F125W.fits":dict(stretch="log", max_percent=99.5),
        "F160W.fits":dict(stretch="log", max_percent=99.9, min_percent=0.1),
    }
)

If you do it this way, you need to provide one for every fits file, otherwise it will break.

Finally, if you find that your not getting a good speed-up with parallelization, you can change the kwarg img_tile_batch_size in files_to_map or dir_to_map. This param sets how many tiles each worker should do. The default is 1000, but depending on your setup you may want to change it.

[hjmcc]

hi @ryanhausen super! I confirm that I can now convert my COSMOS full mosaic FITS file into PNGs and display them with fitsmap! I haven't had time to check the other options just yet but rest assured I will let you know if I find bugs 😀. Speedup seems to be fine on our 48-core machines.

It's fantastic. Once I get a chance, I plan to run the COSMOS2020 catalogue through it with a selection of images so that people can look up their favourite sources. That will be great, and very useful! I guess you can close this issue.

[ryanhausen]

@hjmcc Wonderful! I am glad that the tool is helpful. I'd love to see the finished product when you're done!

Feel free to submit issues for other feature requests that come to mind.

Closing this issue.