search

PaulHancock / Aegean

The Aegean source finding program and associated tools
http://aegeantools.rtfd.io/
Other
47 stars 14 forks source link

Multicore issue - different source counts when using multiple cores with OSX 10.11.6 #30

Closed Sunmish closed 3 years ago

Sunmish commented 8 years ago

When using multiple cores (2-4 on my machine) I get different source counts (often none at all) when compared to similar source-finding with a single core. Running a test on all the test images yields the following source counts:

1904-66_AIR.fits: 1 core: 51, 4 cores: 0 1904-66_AIT.fits: 1 core: 53, 4 cores: 0 1904-66_ARC.fits: 1 core: 52, 4 cores: 0 1904-66_AZP.fits: 1 core: 51, 4 cores: 0 1904-66_BON.fits: 1 core: 51, 4 cores: 23 1904-66_CAR.fits: 1 core: 51, 4 cores: 0 1904-66_CEA.fits: 1 core: 50, 4 cores: 0 1904-66_COD.fits: 1 core: 51, 4 cores: 23 1904-66_COE.fits: 1 core: 51, 4 cores: 0 1904-66_COO.fits: 1 core: 50, 4 cores: 22 1904-66_COP.fits: 1 core: 51, 4 cores: 24 1904-66_CSC.fits: 1 core: 54, 4 cores: 0 1904-66_CYP.fits: 1 core: 50, 4 cores: 21 1904-66_HPX.fits: 1 core: 58, 4 cores: 0 1904-66_MER.fits: 1 core: 49, 4 cores: 0 1904-66_MOL.fits: 1 core: 49, 4 cores: 21 1904-66_NCP.fits: 1 core: 50, 4 cores: 23 1904-66_PAR.fits: 1 core: 52, 4 cores: 0 1904-66_PCO.fits: 1 core: 54, 4 cores: 0 1904-66_QSC.fits: 1 core: 48, 4 cores: 0 1904-66_SFL.fits: 1 core: 50, 4 cores: 0 1904-66_STG.fits: 1 core: 53, 4 cores: 0 1904-66_SZP.fits: 1 core: 52, 4 cores: 0 1904-66_TAN.fits: 1 core: 48, 4 cores: 0 1904-66_TSC.fits: 1 core: 47, 4 cores: 21 1904-66_ZEA.fits: 1 core: 53, 4 cores: 0 1904-66_ZPN.fits: 1 core: 41, 4 cores: 41

FWIW I'm using a Mac with OSX El Capitan, 10.11.6. The python version is 2.7.11 installed through Anaconda with the latest Aegean version as well.

PaulHancock commented 8 years ago

This is the test code that creates the output listed in @Sunmish's report. convert .py.txt->.py and then run core_test.py.txt

If anyone has similar problems i'm keen to hear about it as I haven't been able to replicate the issue on my own Ubuntu or Windows machines, though I have heard of similar strange things happening on some OSX installs.

jd-au commented 8 years ago

I've also hit this issue on Mac OSX El Capitan (10.11.6). I am able to reproduce it using just the command line tools and the supplied test images:

bane Test/Images/1904-66_AIR.fits 44780:INFO This is BANE v1.4.1-(2016-02-15) 44780:INFO using grid_size (15, 15), box_size (90, 90) 44780:INFO on data shape (192, 192) 44780:INFO using 8 cores 44780:INFO done 44780:INFO Wrote <open file '', mode 'w+b' at 0x10acd6c00> 44780:INFO running second pass to get a better rms 44780:INFO using 8 cores 44780:INFO Wrote Test/Images/1904-66_AIR_bkg.fits 44780:INFO Wrote Test/Images/1904-66_AIR_rms.fits

aegean Test/Images/1904-66_AIR.fits aegean:INFO This is Aegean v2.0b-46-g24246a4-(master)-(2016-07-19) aegean:INFO Found 8 cores aegean:INFO Using 8 cores aegean:INFO Finding sources. source_finder:INFO Calculating curvature source_finder:INFO Calculating background and rms data source_finder:INFO beam = 864.00'' x 864.00'' at 0.00deg source_finder:INFO seedclip=5 source_finder:INFO floodclip=4 aegean:INFO No sources found in image aegean:INFO found 0 sources total

aegean Test/Images/1904-66_AIR.fits --cores=1 aegean:INFO This is Aegean v2.0b-46-g24246a4-(master)-(2016-07-19) aegean:INFO Using 1 cores aegean:INFO Finding sources. source_finder:INFO Calculating curvature source_finder:INFO Calculating background and rms data source_finder:INFO beam = 864.00'' x 864.00'' at 0.00deg source_finder:INFO seedclip=5 source_finder:INFO floodclip=4 ***/anaconda/lib/python2.7/site-packages/numpy/lib/function_base.py:3569: RuntimeWarning: Invalid value encountered in median RuntimeWarning) aegean:INFO found 60 sources total

python --version Python 2.7.12 :: Anaconda 2.4.1 (x86_64)

conda list

packages in environment at ***:

#

_license 1.1 py27_1 abstract-rendering 0.5.1 np110py27_0 AegeanTools-v2.0b-46-g24246a4-master alabaster 0.7.6 py27_0 anaconda 2.4.1 np110py27_0 anaconda-client 1.2.1 py27_0 APLpy 1.0 appnope 0.1.0 py27_0 appscript 1.0.1 py27_0 argcomplete 1.0.0 py27_1 astropy 1.2.1 np111py27_0 babel 2.1.1 py27_0 backports_abc 0.4 py27_0 beautifulsoup4 4.4.1 py27_0 bitarray 0.8.1 py27_0 blaze-core 0.8.3 py27_0 bokeh 0.10.0 py27_0 boto 2.38.0 py27_0 bottleneck 1.0.0 np110py27_0 cdecimal 2.3 py27_0 cffi 1.2.1 py27_0 clyent 1.2.0 py27_0 colorama 0.3.3 py27_0 conda 4.1.11 py27_0 conda-build 1.18.2 py27_0 conda-env 2.5.2 py27_0 configobj 5.0.6 py27_0 cryptography 1.0.2 py27_0 curl 7.45.0 0 cycler 0.9.0 py27_0 cython 0.23.4 py27_1 cytoolz 0.7.4 py27_0 datashape 0.4.7 np110py27_1 decorator 4.0.4 py27_0 docutils 0.12 py27_0 dynd-python 0.7.0 py27_0 emcee 2.2.1 enum34 1.0.4 py27_0 fastcache 1.0.2 py27_0 flask 0.10.1 py27_1 freetype 2.5.5 0 funcsigs 0.4 py27_0 gevent 1.0.1 py27_0 gevent-websocket 0.9.3 py27_0 greenlet 0.4.9 py27_0 grin 1.2.1 py27_1 h5py 2.5.0 np110py27_4 hdf5 1.8.15.1 2 idna 2.0 py27_0 ipaddress 1.0.14 py27_0 ipykernel 4.1.1 py27_0 ipython 4.0.1 py27_0 ipython-notebook 4.0.4 py27_0 ipython-qtconsole 4.0.1 py27_0 ipython_genutils 0.1.0 py27_0 ipywidgets 4.1.0 py27_0 itsdangerous 0.24 py27_0 jbig 2.1 0 jdcal 1.0 py27_0 jedi 0.9.0 py27_0 jinja2 2.8 py27_0 jpeg 8d 1 jsonschema 2.4.0 py27_0 jupyter 1.0.0 py27_1 jupyter_client 4.1.1 py27_0 jupyter_console 4.0.3 py27_0 jupyter_core 4.0.6 py27_0 launcher 1.0.0 3 libdynd 0.7.0 0 libpng 1.6.17 0 libtiff 4.0.6 1 libxml2 2.9.2 0 libxslt 1.1.28 2 llvmlite 0.8.0 py27_0 lmfit 0.9.5 lxml 3.4.4 py27_0 markupsafe 0.23 py27_0 matplotlib 1.5.0 np110py27_0 mistune 0.7.1 py27_0 mkl 11.3.3 0 montage-wrapper 0.9.8 multipledispatch 0.4.8 py27_0 nbconvert 4.0.0 py27_0 nbformat 4.0.1 py27_0 networkx 1.10 py27_0 nltk 3.1 py27_0 node-webkit 0.10.1 0 nose 1.3.7 py27_0 notebook 4.0.6 py27_0 numba 0.22.1 np110py27_0 numexpr 2.4.4 np110py27_0 numpy 1.11.1 py27_0 odo 0.3.4 py27_0 openpyxl 2.2.6 py27_0 openssl 1.0.2h 1 pandas 0.17.1 np110py27_0 path.py 8.1.2 py27_1 patsy 0.4.0 np110py27_0 pep8 1.6.2 py27_0 pexpect 3.3 py27_0 pickleshare 0.5 py27_0 pillow 3.0.0 py27_1 pip 8.1.2 py27_0 ply 3.8 py27_0 pprocess 0.5.1 psutil 3.3.0 py27_0 ptyprocess 0.5 py27_0 py 1.4.30 py27_0 pyasn1 0.1.9 py27_0 pyaudio 0.2.7 py27_0 PyAVM 0.9.2 pycosat 0.6.1 py27_1 pycparser 2.14 py27_0 pycrypto 2.6.1 py27_0 pycurl 7.19.5.1 py27_3 pyFFTW 0.10.4 pyfits 3.4 pyflakes 1.0.0 py27_0 pygments 2.0.2 py27_0 pyopenssl 0.15.1 py27_1 pyparsing 2.0.3 py27_0 pyqt 4.11.4 py27_1 pyregion 1.1.4 pytables 3.2.2 np110py27_0 pytest 2.8.1 py27_0 python 2.7.12 1 python-dateutil 2.4.2 py27_0 python.app 1.2 py27_4 pytz 2015.7 py27_0 pyyaml 3.11 py27_4 pyzmq 14.7.0 py27_1 qt 4.8.7 1 qtconsole 4.1.1 py27_0 readline 6.2 2 redis 2.6.9 0 redis-py 2.10.3 py27_0 requests 2.10.0 py27_0 rope 0.9.4 py27_1 ruamel_yaml 0.11.14 py27_0 scikit-image 0.11.3 np110py27_0 scikit-learn 0.17 np110py27_1 scipy 0.16.0 np110py27_1 setuptools 23.0.0 py27_0 simplegeneric 0.8.1 py27_0 singledispatch 3.4.0.3 py27_0 sip 4.16.9 py27_0 six 1.10.0 py27_0 snowballstemmer 1.2.0 py27_0 sockjs-tornado 1.0.1 py27_0 sphinx 1.3.1 py27_0 sphinx_rtd_theme 0.1.7 py27_0 spyder 2.3.8 py27_0 spyder-app 2.3.8 py27_0 sqlalchemy 1.0.9 py27_0 sqlite 3.13.0 0 ssl_match_hostname 3.4.0.2 py27_0 statsmodels 0.6.1 np110py27_0 sympy 0.7.6.1 py27_0 terminado 0.5 py27_1 tk 8.5.18 0 toolz 0.7.4 py27_0 tornado 4.3 py27_0 traitlets 4.0.0 py27_0 ujson 1.33 py27_0 unicodecsv 0.14.1 py27_0 werkzeug 0.11.2 py27_0 wheel 0.29.0 py27_0 xlrd 0.9.4 py27_0 xlsxwriter 0.7.7 py27_0 xlwings 0.5.0 py27_0 xlwt 1.0.0 py27_0 xz 5.0.5 0 yaml 0.1.6 0 zeromq 4.1.3 0 zlib 1.2.8 3

caseyjlaw commented 6 years ago

We have seen the same problem in our test today. OSX 10.13.3, python 2.7, latest pip installed aegean (2.0.2). When we run the search with most of the parameters at default values, it selects ncores=4 and finds no sources. When run in debug mode, it selects ncores=1 and finds 24 sources. If no solution is known, it would be good to at least set the default ncores to 1.