lsst-uk / lasair-project-management

Event handling site for LSST:UK
Apache License 2.0
9 stars 0 forks source link

New Lasair query page #29

Open RoyWilliams opened 5 years ago

RoyWilliams commented 5 years ago

I would really appreciate a close look from you of the new Lasair query page that Ken and I built last week. We now have a truly freeform query mechanism. It is available on the dev system (username: flame, password: !flame). Please look at:

(1) Copy/paste some examples into the query box and click Submit Query. Is it clear what is happening?

(2) The text of query page itself https://lasair-dev.roe.ac.uk/objlist/. Are there other notes or caveats to include?

(3) The Example Queries on the same page. (Some stolen from Joe and Matt Jupyter notebooks.) It would be great to have a fuller explanation of why some of them look the way they do. Or modify them to be more informative.

(4) The schema page https://lasair-dev.roe.ac.uk/schema. I have guessed at which fields to include from sherlock_crossmatches, and guessed at the correct explanation. I have also reduced the ZTF candidate schema.

(5) We would also appreciate your opinion of the new object page, for example https://lasair-dev.roe.ac.uk/object/ZTF18aaxyspx/, which now has non-detections, Sherlock classification/annotation, and JSON output.

RoyWilliams commented 5 years ago

From Matt Nichol:

Looks great! A few initial thoughts:

— running the example queries is mostly clear and straightforward. I did notice that it breaks if I add LIMIT to the query, presumably because of the hard-coded LIMIT 1000. Maybe this doesn’t matter; I don’t know whether people would have any reason to want to specify manual limits.

— The examples named “classified as SNe” and “recent SNe” are slightly misleading in my view. Not a problem for dev version, but when you’re ready to put this on the main page, I would say that a sherlock ‘classification’ means “likely SNe”. And ‘recent’ sounds like a recent discovery (i.e. all candidates within an object are recent), whereas the query returns recent detections (any candidate that is recent regardless of how long the object has been known). This leads me on to the next point…

— Most SN-enthusiasts will be primarily interested in young SNe, i.e. that exploded recently and have rising light curves. Here’s a simple query that returns likely SNe first detected within the past week (using latest mags and mag ranges to quickly assess whether they are brightening or fading). Most of the objects returned do look like young SNe. I think this or something similar would make a good baseline query for SN folk:

SELECT objectId, jdmin-2400000.5 as first_discovered, jdmax-2400000.5 as last_seen, ncand, latestgmag, maggmax, maggmin, latestrmag, magrmax, magrmin FROM objects WHERE jdmin > (unix_timestamp(now())/86400 + 2440587.5) - 7 AND sherlock_classification = 'SN' ORDER BY jdmax DESC

— For sherlock info, I’m a bit confused about the redshift. The table on each object page gives “photoZ”, but the schema page only lists “z” under the sherlock_crossmatches table. If this z can be either photometric or spectroscopic (depending on the catalog or specific object) it’s important to differentiate (and spec should aways take priority). I remember talking about this at the Lasair meeting, and I think Dave said that sherlock does default to spec if available? If so, the sherlock_crossmatches table needs to keep track of phot or spec (e.g. with another column called something like “z_type”) and this should also be displayed on the object pages

— I also notice on many object pages that the redshift given in the bullet points (to calculate absolute magnitude) often differs a bit from that in the crossmatches table right under it. e.g. https://lasair-dev.roe.ac.uk/object/ZTF18acdzzyf/, https://lasair-dev.roe.ac.uk/object/ZTF18aceyjfx/

— minor point: each non-detection appears to be listed multiple times in the candidates table (i.e. identical times and limiting mags). I don’t know whether this is because of multiple exposures in a row, but it’s suspicious that the times are exactly the same.

RoyWilliams commented 5 years ago

From Joe Lyman:

When I add my own 'LIMIT' it clashes with the 'LIMIT 1000', which is appended by the software before query - it would be good if it would honour the user's limit (if its <=1000).

Results available in some plain formatted text format (json or csv etc) would be good.

I couldn't find out how to access non-detections as part of queries. Do these appear as rows in 'candidates' and how would we query on them? A regular query might be to find detections with a recent non-detection.

Related, how true are the non-detections? e.g. is there some minimum rb score in the ZTF steam or 'candidates' insertion, and so would a low rb score "detection" be instead counted as a non-detection.

I think the 'finding recent supernovae' query needs a "AND o.sherlock_classification = 'SN'" adding into it?

For an example of interesting additional 'objects' calculated values, Matt, Stephen, and I discussed in the SN group (but valid for other transients) about having a rising/fading metric in the 'objects' table. An example implementation is

SELECT o.primaryid, o.objectid, o.ncand FROM sherlock_crossmatches AS sc JOIN objects as o ON sc.transient_object_id = o.primaryid WHERE sc.rank = 1 AND sc.association_type = 'SN' AND o.ncand > 8

association_type Type of the matched object (Can be VS, SN, CV, BS, NT, AGN, UNKNOWN) (string)

(5) We would also appreciate your opinion of the new object page,

There seems to be a varying number of multiple rows in the light curve table per non-detection.

smarttgit commented 5 years ago

Immediate issues to sort :

The summary table returned for each should have objectId ra dec discovery MJD discovery mag discovery filter brightest mag brightest mjd discovery filter sherlock_classification redshift (from sherlock cross-match, if available)

smarttgit commented 5 years ago

The PanSTARRS sgscore selection line is not correct and should never be used like this

It says only select the object if Pan-STARRS sgscore < 0.49

Two problems with that 1) There are many large extended galaxies that Pan-STARRS has not catalogued properly. 2) This applies to the nearest Pan-STARRS catalogued object. So the nearest, faint star to an orphan (say 5 away) has no relation to the object.

What we want is the following, but I don't know how to code in SQL

To remove variables from the search, we will use this : AND o.sherlock_classification != 'VS' AND o.sherlock_classification != 'AGN' AND o.sherlock_classification != 'CV' AND o.sherlock_classification != 'BS'

and this is bit I need SQL for :

AND if distpsnr1 < 1 arcsec [meaning, if there is a Pan-STARRS obect within 2 arcsec) then only select if sgscore < 0.49 [i.e. it has to be galaxy] else don't use sgscore at all

smarttgit commented 5 years ago

New query now on the lasair site. Addresses most of the issues above - this seems a robust and useful query for all SN like candidates.

smarttgit commented 5 years ago

Also added a "Nuclear transients" or TDE candidate query

tms-epcc commented 2 years ago

06/JUL/22