search

transientskp / pyse

Python Source Extractor
BSD 2-Clause "Simplified" License
11 stars 5 forks source link

Kappa sigma clipping using dask #13

Closed HannoSpreeuw closed 3 years ago

HannoSpreeuw commented 3 years ago

This implements paralllellisation of kappa, sigma clipping in PySE using Dask. The output is numerically equal to the output from the Ray implementation, which is numerically equal to the output of the single threaded implementation of the Fix-no-clipping-when-source-density-is-high branch.

For the optimum workload per thread, the Dask implementation completed clipping in 2.0s on my laptop. For the Ray implementation it took 3.0s.

However, on my more powerful pc the relative difference became somewhat smaller.

HannoSpreeuw commented 3 years ago

Okay, so we have a git-lfs bandwidth problem again that emerged from Travis-CI. We don't pay for git-lfs, so we exceed the free quota very quickly, perhaps after one or two PRs a month. The problem comes from the images in test/data (total 200 MB) that Travis needs. I executed the nosetests from the .travis.yml:

pyse nosetests --with-coverage --cover-package=sourcefinder

and all 105 tests were passed, so I think that Travis-CI would not have complained if we would have had enough git-lfs bandwidth.