search

lofar-astron / PyBDSF

Python Blob Detector and Source Finder
https://pybdsf.readthedocs.io
GNU General Public License v3.0
51 stars 25 forks source link

Handling beam and frequency of a FITS cube #218

Closed physicistsam closed 3 months ago

physicistsam commented 4 months ago

This is a usage quarry rather than an issue. I am trying to perform source extraction using a fits cube, on each channel. The programme is as follows:

import sys, os, time
import bdsf
os.nice(10)
beam_m=[(3.402592546505e-03, 2.490963285403e-03, -7.818989487638e+01), (2.709791203909e-03, 1.902673183541e-03, -7.818989487638e+01), (2.539184411559e-03, 1.740613144994e-03, -7.818989487638e+01)]
freq_m =[9.985621744792E+08+0*2.853333333333E+08, 9.985621744792E+08+1.0*2.853333333333E+08, 9.985621744792E+08+2.0*2.853333333333E+08]

input_images=['myfile.fits']
for input_image in input_images:
    img = bdsf.process_image(input_image, multichan_opts = True, collapse_mode='single', beam_sp_derive = False,  frequency_sp = freq_m,
                             collapse_ch0 =1,beam_spectrum = beam_m , psf_vary_do=True, atrous_do=True, rms_box=(150,50), 
                             rms_map=True, adaptive_rms_box=False, adaptive_thresh=150, rms_box_bright=(20,7), thresh_isl=5.0, 
                             thresh_pix=7.0, plot_allgaus=True, plot_islands=True)
    #img = bdsf.process_image(input_image, thresh_pix = 10.0, thresh_isl=5.0)
    #img.show_fit()
    # Write the source list catalog. File is named automatically.
    img.write_catalog( format='fits', catalog_type='srl', clobber=True)

    # Write the residual image. File is named automatically.
    img.export_image( img_type='gaus_resid', clobber=True)

    # Write the model image. Filename is specified explicitly.
    img.export_image(img_type='gaus_model', outfile=input_image+'wavelet.model')

    #write the rms map
    img.export_image(img_type='rms', clobber=True)

I expect it to perform source extration on the second channel (out of three), with frequency and the beam given by the array, or it should read the same from the FITS. However, when I see the .log file, it always quotes / mentions the frequency of the zeroth channel and beam that of the mean quoted in the FITS header (the beams for all the bands are also mentioned in the header). Even the rms files produced have the mean beam and zeroth channel frequency. May be I am doing something wrong her. Please let me know how to resolve this. The log file is as follows.


========================================================================
Thu 07-03-2024 09:13:24:: INFO    : Processing myfile.fits
Thu 07-03-2024 09:13:24:: INFO    : PyBDSF version 1.10.3
Thu 07-03-2024 09:13:24:: INFO    : Non-default input parameters:
                                        adaptive_thresh      = 150.0
                                        atrous_do            = True
                                        beam_sp_derive       = False
                                        beam_spectrum        = [(0.003402592546505, 0.002490963285403, -78.18989487638), (0.002709791203909, 0.001902673183541, -78.18989487638), (0.002539184411559, 0.001740613144994, -78.18989487638)]
                                        collapse_ch0         = 1
                                        collapse_mode        = 'single'
                                        filename             = myfile.fits'
                                        multichan_opts       = True
                                        plot_allgaus         = True
                                        plot_islands         = True
                                        psf_vary_do          = True
                                        rms_box              = (150, 50)
                                        rms_box_bright       = (20, 7)
                                        rms_map              = True
                                        thresh_isl           = 5.0
                                        thresh_pix           = 7.0
Thu 07-03-2024 09:13:24:: USERINFO: --> Opened 'myfile.fits'
Thu 07-03-2024 09:13:24:: INFO    : Original data shape of myfile.fits: (3, 1, 12005, 12005)
Thu 07-03-2024 09:13:24:: INFO    : Final data shape (npol, nchan, x, y): (1, 3, 12005, 12005)
Thu 07-03-2024 09:13:24:: USERINFO: Image size .............................. : (12005, 12005) pixels
Thu 07-03-2024 09:13:24:: USERINFO: Number of channels ...................... : 3
Thu 07-03-2024 09:13:24:: USERINFO: Number of Stokes parameters ............. : 1
Thu 07-03-2024 09:13:25:: USERINFO: Beam shape (major, minor, pos angle) .... : (3.07426e-03, 2.16169e-03, -78.2) degrees
Thu 07-03-2024 09:13:25:: INFO    : Equinox not found in image header. Assuming J2000.
Thu 07-03-2024 09:16:09:: USERINFO: Source extraction will be done on channel : 1 (998.562 MHz)
Thu 07-03-2024 09:16:10:: USERINFO: Number of blank pixels .................. : 0 (0.0%)
Thu 07-03-2024 09:17:07:: INFO    : Raw mean (Stokes I) =  0.0046 mJy and raw rms =  2.9345 mJy 
Thu 07-03-2024 09:17:07:: INFO    : sigma clipped mean (Stokes I) =  -0.0006 mJy and  sigma clipped rms =  0.2361 mJy 
Thu 07-03-2024 09:17:14:: USERINFO: Flux from sum of (non-blank) pixels ..... : 9.723 Jy
Thu 07-03-2024 09:17:15:: INFO    : Estimated bmpersrc_th =   69.47
Thu 07-03-2024 09:17:15:: INFO    : Parameter confused is False
Thu 07-03-2024 09:17:15:: USERINFO: --> Calculating background rms and mean images
Thu 07-03-2024 09:17:15:: INFO    : Estimated size of brightest source (pixels) = 8
Thu 07-03-2024 09:17:18:: INFO    : Maximum extent of largest 10-sigma island using clipped rms (pixels) = 545
Thu 07-03-2024 09:18:15:: WARNING : Negative values found in rms map interpolated with spline_rank = 3
Thu 07-03-2024 09:18:16:: WARNING : Using spline_rank = 1 (bilinear interpolation) instead
Thu 07-03-2024 09:18:55:: USERINFO: Using user-specified rms_box ............ : (150, 50) pixels
Thu 07-03-2024 09:18:55:: INFO    : Background rms and mean images computed
Thu 07-03-2024 09:18:55:: USERINFO: --> Using 2D map for background rms
Thu 07-03-2024 09:18:55:: USERINFO: --> Variation in mean image significant
Thu 07-03-2024 09:18:55:: USERINFO: --> Using 2D map for background mean
Thu 07-03-2024 09:18:55:: USERINFO: Min/max values of background rms map .... : (6.49e-05, 8.12e-02) Jy/beam
Thu 07-03-2024 09:18:55:: USERINFO: Min/max values of background mean map ... : (-9.50e-03, 4.47e-03) Jy/beam
Thu 07-03-2024 09:18:56:: USERINFO: --> Expected 5-sigma-clipped false detection rate < fdr_ratio
Thu 07-03-2024 09:18:56:: USERINFO: --> Using sigma-clipping ('hard') thresholding
Thu 07-03-2024 09:18:56:: USERINFO: Minimum number of pixels per island ..... : 22
Thu 07-03-2024 09:19:01:: USERINFO: Number of islands found ................. : 854
Thu 07-03-2024 09:21:35:: USERINFO: Total number of Gaussians fit to image .. : 1057
Thu 07-03-2024 09:21:35:: USERINFO: Total flux density in model ............. : 21.111 Jy
Thu 07-03-2024 09:21:35:: WARNING : Total flux density in model is 2.17 times sum of pixels in input image. Large residuals may remain.
Thu 07-03-2024 09:21:35:: INFO    : Decomposing gaussian residual image into a-trous wavelets
Thu 07-03-2024 09:21:35:: INFO    : Calculating residual image after subtracting reconstructed gaussians
Thu 07-03-2024 09:21:54:: INFO    : Statistics of the Gaussian residual image:
                                            mean: -5.221e-06 (Jy/beam)
                                        std. dev: 2.842e-03 (Jy/beam)
                                            skew: -3.784
                                        kurtosis: 2078.395
Thu 07-03-2024 09:21:54:: INFO    : Using B3 spline filter with J_max = 7
Thu 07-03-2024 09:21:56:: USERINFO:
.
.
.
.
.
.
Thu 07-03-2024 18:57:31:: INFO    : Internally derived parameters:
                                        beam                 : (0.003074256033112387, 0.002161689685955781, -78.18989487637535)
                                        bmpersrc_th          : 69.4650918934994
                                        frequency            : 998562174.4792
                                        indir                : './'
                                        maxpix_isl           : inf
                                        minpix_isl           : 22
                                        outdir               : 'path'
                                        rms_box_bright       : (150, 50)
                                        thresh               : 'hard'

Please let me know if the question is not clear.

darafferty commented 4 months ago

Hmm, indeed it seems that it's ignoring your input frequencies for some reason. I'll try to understand why this is happening.

Regarding the beam_spectrum option, the documentation is rather misleading, as it turns out that this option is only used by the spectral index module. I think you can work around this limitation by specifying the beam using the beam option and giving it the beam of the channel you want to use for source detection (so the second channel). This method might also work for the frequency (use the frequency option to set it that of the second channel).

darafferty commented 3 months ago

The issue with frequency_sp being ignored should now be fixed on the master branch. I've also added a note to the docs about the use of beam_spectrum. Please reopen this issue if you're still having problems.

physicistsam commented 3 months ago

Thanks for the help. However when I try to install the mater branch using 

pip install git+https://github.com/lofar-astron/PyBDSF.git

I am getting following error:

  Building wheel for bdsf (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for bdsf (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [114 lines of output]

      --------------------------------------------------------------------------------
      -- Trying 'Ninja' generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
        Compatibility with CMake < 3.5 will be removed from a future version of
        CMake.

        Update the VERSION argument <min> value or use a ...<max> suffix to tell
        CMake that the project does not need compatibility with older versions.

      Not searching for unused variables given on the command line.

      -- The C compiler identification is GNU 11.4.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- The CXX compiler identification is GNU 11.4.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Configuring done (0.4s)
      -- Generating done (0.0s)
      -- Build files have been written to: /tmp/pip-req-build-2y1ehlp1/_cmake_test_compile/build
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying 'Ninja' generator - success
      --------------------------------------------------------------------------------

      Configuring Project
        Working directory:
          /tmp/pip-req-build-2y1ehlp1/_skbuild/linux-x86_64-3.10/cmake-build
        Command:
          /tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/cmake/data/bin/cmake /tmp/pip-req-build-2y1ehlp1 -G Ninja -DCMAKE_MAKE_PROGRAM:FILEPATH=/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/ninja/data/bin/ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-req-build-2y1ehlp1/_skbuild/linux-x86_64-3.10/cmake-install -DPYTHON_VERSION_STRING:STRING=3.10.4 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/myPATH/bin/python -DPYTHON_INCLUDE_DIR:PATH=/software/common/python/3.10.4/include/python3.10 -DPYTHON_LIBRARY:PATH=/software/common/python/3.10.4/lib/libpython3.10.a -DPython_EXECUTABLE:PATH=/myPATH/bin/python -DPython_ROOT_DIR:PATH=/myPATH -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/software/common/python/3.10.4/include/python3.10 -DPython_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/numpy/core/include -DPython3_EXECUTABLE:PATH=/myPATH/bin/python -DPython3_ROOT_DIR:PATH=/myPATH -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/software/common/python/3.10.4/include/python3.10 -DPython3_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/numpy/core/include -DCMAKE_MAKE_PROGRAM:FILEPATH=/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/ninja/data/bin/ninja -DCMAKE_BUILD_TYPE:STRING=Release

      Not searching for unused variables given on the command line.
      -- The C compiler identification is GNU 11.4.0
      -- The CXX compiler identification is GNU 11.4.0
      -- The Fortran compiler identification is GNU 11.4.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Detecting Fortran compiler ABI info
      -- Detecting Fortran compiler ABI info - done
      -- Check for working Fortran compiler: /usr/bin/gfortran - skipped
      CMake Warning (dev) at /tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:245 (find_package):
        Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
        are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
        the cmake_policy command to set the policy and suppress this warning.

      Call Stack (most recent call first):
        CMakeLists.txt:4 (find_package)
      This warning is for project developers.  Use -Wno-dev to suppress it.

      -- Found PythonInterp: /myPATH/bin/python (found version "3.10.4")
      CMake Warning (dev) at /tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:252 (find_package):
        Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
        are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
        the cmake_policy command to set the policy and suppress this warning.

      Call Stack (most recent call first):
        CMakeLists.txt:4 (find_package)
      This warning is for project developers.  Use -Wno-dev to suppress it.

      -- Found PythonLibs: /software/common/python/3.10.4/lib/libpython3.10.a (found version "3.10.4")
      -- Found Python: /myPATH/bin/python (found version "3.10.4") found components: Interpreter Development.Module
      -- Found F2PY: /tmp/pip-build-env-vg5tsljv/overlay/bin/f2py3 (found version "1.21.6")
      CMake Error at /tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
        Could NOT find Boost (missing: Boost_INCLUDE_DIR python numpy)
      Call Stack (most recent call first):
        /tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
        /tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.28/Modules/FindBoost.cmake:2393 (find_package_handle_standard_args)
        CMakeLists.txt:7 (find_package)

      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/skbuild/setuptools_wrap.py", line 666, in setup
          env = cmkr.configure(
        File "/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/skbuild/cmaker.py", line 357, in configure
          raise SKBuildError(msg)

      An error occurred while configuring with CMake.
        Command:
          /tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/cmake/data/bin/cmake /tmp/pip-req-build-2y1ehlp1 -G Ninja -DCMAKE_MAKE_PROGRAM:FILEPATH=/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/ninja/data/bin/ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-req-build-2y1ehlp1/_skbuild/linux-x86_64-3.10/cmake-install -DPYTHON_VERSION_STRING:STRING=3.10.4 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/myPATH/bin/python -DPYTHON_INCLUDE_DIR:PATH=/software/common/python/3.10.4/include/python3.10 -DPYTHON_LIBRARY:PATH=/software/common/python/3.10.4/lib/libpython3.10.a -DPython_EXECUTABLE:PATH=/myPATH/bin/python -DPython_ROOT_DIR:PATH=/myPATH -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/software/common/python/3.10.4/include/python3.10 -DPython_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/numpy/core/include -DPython3_EXECUTABLE:PATH=/myPATH/bin/python -DPython3_ROOT_DIR:PATH=/myPATH -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/software/common/python/3.10.4/include/python3.10 -DPython3_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/numpy/core/include -DCMAKE_MAKE_PROGRAM:FILEPATH=/tmp/pip-build-env-vg5tsljv/overlay/lib/python3.10/site-packages/ninja/data/bin/ninja -DCMAKE_BUILD_TYPE:STRING=Release
        Source directory:
          /tmp/pip-req-build-2y1ehlp1
        Working directory:
          /tmp/pip-req-build-2y1ehlp1/_skbuild/linux-x86_64-3.10/cmake-build
      Please see CMake's output for more information.

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for bdsf
Failed to build bdsf
ERROR: Could not build wheels for bdsf, which is required to install pyproject.toml-based projects

Standard relased version of pyBDSF is being installed using pip, without any issue. I think this has to do with the boost libraries. I am working in a cluster and virtual environment. Do you think I need to install them locally ?

darafferty commented 3 months ago

Hmm, I'm not sure whether you need them locally. @gmloose do you happen to know?

tammojan commented 3 months ago

To build the latest version, you need to build from source, which in this case fails because the package boost-python is missing. If you download from pip, you probably get pre-built binary wheels so you do not have to compile from source. Could you mention on which operating system you are running (e.g. ubuntu 22.04, cat /etc/os-release should tell you), and which python environment (which python should tell you)?

physicistsam commented 3 months ago

OS: PRETTY_NAME="Ubuntu 22.04.3 LTS" Python environment is the virtual environment I made, ie /myPath/bin/python. Version is Python 3.10.4.

tammojan commented 3 months ago

Thank you. Getting boost-python to work in a virtual-env may be challenging. Perhaps you can set up a virtualenv that uses the system python, and install libboost-python-dev through apt. Alternatively, you could copy the changed python files from https://github.com/lofar-astron/PyBDSF/pull/219 into the pip-installed version of pybdsf in your virtual environment (overwriting the ones from the pip-version).

gmloose commented 3 months ago

If you want to use master, you have to build from source, and you need to install Boost.Python. Something like this is probably enough to get you going.

apt install libboost-python-dev
physicistsam commented 3 months ago

I think its a bit difficult to use apt as I don't have root or sudo access.

tammojan commented 3 months ago

In that case the 'hack' I suggested: download collapse.py , psf_vary.py , shapefit.py and overwrite the existing files in your virtualenv with their updated version

physicistsam commented 3 months ago

Let me try this "hack".

physicistsam commented 3 months ago

I tried the "hack" and it worked. Thanks !