search

LSSTDESC / desc-exp-checker

Crowdsourced image quality control
MIT License
0 stars 2 forks source link

Restructure/rewrite code base and interface with LSST Stack #17

Open yymao opened 6 years ago

yymao commented 6 years ago

As this project continue the evolve, it seems logical to rewrite/restructure the code base for several reasons. The major ones are:

  1. There are much more people in the collaboration who are experienced with Python than those who are experienced with PHP. Using Python would enable more people to contribute to and help maintain this tool.

  2. The Project has many tools in Python that are available for interfacing/accessing/processing the images. Having the back end code in Python enables us to use those existing (or to-be-existing) tools and we don't need to reinvent the wheels.

To achieve (2), the major structural change would be how the image files are being accessed. Right now we (i) use LSST Stack to first pre-generate fits files on disk, (ii) add them into the exposure checker's database, and (iii) use javascript to display the fits files on the front end. These steps in principle can all be merged and the exposure checker can directly use LSST Stack to create a png/jpg file to display on the web interface. This would also allow the exposure checker to access auxiliary information about the images more easily.

To interface with LSST Stack, the work actually involves mostly moving much of the front-end javascript code, which is currently used to fetch and display the images, to the back end, and rewriting them in Python so that we can use LSST Stack.

On the other hand, most of the current back-end php code are actually dealing with users' contribution (api.php, db.php, gallery.php, mydata.php, problems.php, ranking.php, stats.php), and mostly they just need to be directly translated into Python with little or no structural change. (So achieving goal (1) is actually easier.)

On the database side, we then will only need databases that store the users' contribution (the qa and submissions tables), assuming we are using NERSC or any services that provide user management already, in which case we don't the user management database (see this note).

(tagging people who have been in this conversation on Slack @kadrlica @RobertLuptonTheGood @SimonKrughoff @katrinheitmann @rmandelb @drphilmarshall)

yymao commented 6 years ago

Also, if we carry out this plan, Issues #8, #9, #12, #13 will all naturally be fixed.

yymao commented 5 years ago

An update on this issue: I've implemented a cutout service (currently only for Run1.1p coadd images) using the idea outlined in this issue (i.e., using Python + LSST Stack in the backend to access data, and then send images to the frontend to display using NEWT AJAX). One can find the source code at https://github.com/LSSTDESC/cutout-service.

This is a proof-of-concept that our proposal above can work. The major enhancement that still needs to be done is to make this an actual service (i.e., does not start a new python kernel for each request; c.f. https://github.com/LSSTDESC/cutout-service/issues/2). In fact, right now most of the wait time is on starting the interpreter and load modules, rather than on running the image generation code.

Of course the cutout service only demonstrates how to display images, and for the exposure checker we also need to implement the feedback recording machiasm in Python + JS, but this is supposedly not as involved (at least one would hope).