Determine and simplify jwst dependencies

[nden]

The JWST dependencies are not listed anywhere and sometimes it's a matter of trial and error to determine everything that's needed for the pipeline to run. This issue makes an attempt to determine the dependencies, try to reduce their number and eventually document them. I put inline comments/questions. Feel free to add more.

Installation dependencies:

• sphinx Is this really necessary from the user point of view? Isn't it sufficient to document the command to build the documentation if they wanted to?
  • jde: Agreed from a normal user-viewpoint. However, it would be nice if when one does the python setup.py build_sphinx, that the dependencies get installed, much as they do with python setup.py test.
• ~namedlist~ #3065 Could this be replaced with namedtuple? What is the functionality that requires install time third party dependence?
  • jde Yes, the functionality can be reproduced internally. Big question though is what are the standards for using/not using third-party? Discuss among yourselves...

Testing dependencies:

• ~backports.tmpfile~ #1510, #1585
  • jde Now that we're python3 only, this can go.
• pytest
• ~pytest_catchlog~ #1585
• requests_mock

Running calspec2:

• astropy
• asdf
• gwcs
• jsonschema
• ~nose~ #1585
• drizzle
• ~fitsblender~ #1689 , #1690 , #1708, #1955 An old version of fitsblender is on pypi, needs to be updated as the one there uses pyfits.- (spacetelescope/fitsblender#15)
• stsci.tools #3145 - the only module from stsci.tools is bitmask. It may be worth copying it here to avoid the dependency. But stsci.image still depends on stsci.tools.
• ~d2to1~ (required by stsci.tools) - spacetelescope/stsci.tools#55
• ~stsci.distutils~ (required by stsci.tools) - spacetelescope/stsci.tools#55
• ~stwcs~ (imported by stsci.tools.fileutil.OpenImage via fitsblender):scream: - #1689 , #1690 , #1708
• stsci.image
• ~stsci.convolve~ (required by stsci.image) Isn't the astropy.convolution.convolve function good enough?
• photutils
• stsci.imagestats It may be possible to replace use of this with astropy.stats.sigma_clipped_stats...
• ~stsci.sphere~ ~Isn't this the depricated package, the current one being spherical_geometry?~ Replaced by spherical_geometry in #1859
• stsci.stimage
• ~six~, since we're py3 only now, we shouldn't need this. Removed in #2382

to be continued...

[jhunkeler]

stsci.sphere references

$ find . -type f -name "*.py" | xargs grep -H 'stsci\.sphere'
./jwst/skymatch/skyimage.py:from stsci.sphere.polygon import SphericalPolygon
./jwst/skymatch/skyimage.py:        :py:class:`~stsci.sphere.polygon.SphericalPolygon`
./jwst/skymatch/skyimage.py:            A :py:class:`~stsci.sphere.polygon.SphericalPolygon` that is
./jwst/skymatch/skyimage.py:            :py:class:`stsci.sphere.polygons.SphericalPolygon`, or
./jwst/skymatch/skyimage.py:            :py:class:`stsci.sphere.polygons.SphericalPolygon`, or
./jwst/skymatch/skyimage.py:        :py:class:`~stsci.sphere.polygon.SphericalPolygon`
./jwst/skymatch/skyimage.py:            A :py:class:`~stsci.sphere.polygon.SphericalPolygon` that is
./jwst/skymatch/skyimage.py:            :py:class:`stsci.sphere.polygons.SphericalPolygon`, or
./jwst/tweakreg/wcsimage.py:from stsci.sphere.polygon import SphericalPolygon
./jwst/tweakreg/wcsimage.py:        :py:class:`~stsci.sphere.polygon.SphericalPolygon`
./jwst/tweakreg/wcsimage.py:            A :py:class:`~stsci.sphere.polygon.SphericalPolygon` that is
./jwst/tweakreg/wcsimage.py:        :py:class:`~stsci.sphere.polygon.SphericalPolygon`
./jwst/tweakreg/wcsimage.py:            A :py:class:`~stsci.sphere.polygon.SphericalPolygon` that is
./jwst/tweakreg/wcsimage.py:        :py:class:`~stsci.sphere.polygon.SphericalPolygon`
./jwst/tweakreg/wcsimage.py:            A :py:class:`~stsci.sphere.polygon.SphericalPolygon` that is```

[nden]

@mcara stsci.sphere is deprecated in favor of spherical_geometry. Could the latter be used as a drop in replacement?

[bernie-simon]

Yes, it should be compatible.

[mcara]

@nden @bernie-simon knows much better than me what is going on with spherical_geometry and he seems to be optimistic. However, I would suggest to wait a little: I could try to replace stsci.sphere with spherical_geometry in stsci.skypac and see if drizzlepac runs fine and if it does - then it is safe to replace. Or, if there are any JWST tests then maybe this can be done directly in jwst.

[nden]

@mcara As far as I can tell the only reason stsci.tools is a dependency is the use of bitmask. Does it make sense to copy it to jwst to avoid stsci.tools?

[mcara]

@nden Do you mean make a copy of the bitmask as a sub-package of jwst? It will be more difficult to maintain, I would think... For example, bitmask is already in the jwst and it is already behind the one in stsci.tools: https://github.com/STScI-JWST/jwst/blob/master/jwst/resample/bitmask.py I wasn't even aware of this before so how could have I updated it?

[jdavies-st]

I notice that fitsblender is still imported in cube_build/ifu_cube.py. @jemorrison ?

Once removed, we can remove it as a dependency.

[mcara]

@nden ... But, if this is what you really want, I guess it could be copied to jwst.

[mcara]

@nden Let me know .

[mcara]

Another option, probably would be to get bitmask into astropy.nddata (or some other place) - see https://github.com/astropy/ccdproc/pull/611#issuecomment-362977219

[jdavies-st]

I like the idea of putting bitmask into astropy.nddata or some other place.

[jdavies-st]

So it turns out that even though we resolved the bitmask issue by using the version on astropy, stsci.tools is a dependency of stsci.image, which we still depend on. So we did not remove the stsci.tools dependency yet. =(

[mcara]

@jdavies-st Well, that's like "second-hand" dependency. It is not a direct dependency. A package manager should be able to resolve all this.