BETA TEST: ZTF Forced-Photometry

thespacedoctor commented 10 months ago

This test is to iron out any issues with the presentation of the ZTF force-photometry. The forced photometry in the alert packets comes in raw counts, but it is being converted to μJy on the fly when the webpages render.

From the advice given in 'A New Forced Photometry Service for the Zwicky Transient Facility', I have made these cuts to the force-photometry data:

Removed photometric measurements set to -99999 in the output lightcurve file. This indicates that a measurement could not be computed
Removed epochs with scisigpix > 25 DN (spatial noise-sigma per pixel)
Removed epochs with sciinpseeing > 4 arcsec (very poor seeing)

There will be cases where the baseline flux is not zero, typically when the source producing the difference flux is found in the subtracted reference image (see panel B below from the paper). It is relatively simple to re-normalise this baseline flux to zero for an individual short-lived transient, but much harder to do so in a generic way for all transients. I have not attempted any generic correction, but I can revisit if we think this is a big enough problem.

Roy has switched on ingestion into the dev server, so we should see more forced photometry appearing in the next days. But for now, here are some objects Ken found:

Suggested Checklist (for guidance only ... all feedback welcome):

[ ] Is it immediately clear what the 2 lightcurve plots are showing?
[ ] Are you happy with the default scaling/zoom of the forced and unforced lightcurve plots?
[ ] Are the tooltips connected to the 'Forced Photometry' and 'Unforced Photometry' clear?
[ ] Can you export the lightcurve data easily?
[ ] Any other suggestions for improvement?

RoyWilliams commented 10 months ago

Any chance the two plots could be together with no space between? I am scrolling up and down to compare them because of the 3 inches of white space.

smarttgit commented 10 months ago

Is it immediately clear what the 2 lightcurve plots are showing? yes, but the text displayed on the cursor hover is showing uJy flux twice and not uJy +/- duJy. Also the errors

Are you happy with the default scaling/zoom of the forced and unforced lightcurve plots? Looks fine As Roy says, would be better to have them closer to see them on the one screen shot. Could the legend sit within the plot, would save white space

Are the tooltips connected to the 'Forced Photometry' and 'Unforced Photometry' clear? You mean the little "info" circle and the link to learn more ? I think we should do more on the page https://lasair.readthedocs.io/en/develop/concepts/lightcurve.html To explain the forced and unforced differences etc.

Can you export the lightcurve data easily? Not obvious how to get the forced values.

For long lasting lightcurves - minor tick marks at 10 days if possible

smarttgit commented 10 months ago

And looks great !

smarttgit commented 10 months ago

Agree not to do any renormalising at present. Can revisit.

smarttgit commented 10 months ago

OK, I can see the forced flux in the table and can export. With incomplete numbers in the columns, this will be confusing to users. Maybe this is an issue with ZTF unforced photometry and forced ?

e.g. https://lasair-dev.lsst.ac.uk/objects/ZTF23abivuzt/#top

There are rows with magpsf but no forced microjansky And rows with forced microjansky but no magpsf.

The mag <-> flux conversions all look OK.

When the CSV is downloaded, the flux_status appears next to the flux microjansky which implies it is related to the forced flux. Which it isn't of course - could we move that column to sit next to magpsf and call it 'psf_status`

Not sure what is happening with the miss-matches of psf mags and fluxes - I guess you are just reading from the forced input as you are giving it ?

smarttgit commented 10 months ago

I much prefer a solid line, not the shaded -ve region to illustrate uJy = 0

thespacedoctor commented 10 months ago

There are rows with magpsf but no forced microjansky. And rows with forced microjansky but no magpsf.

@RoyWilliams can verify, but as this is the development server, the ingests have not been happening consistently over the last weeks, and therefore, the forced photometry will not be complete. This explains the rows with magpsf but no forced microjansky.

On the second point of rows with forced microjansky but no magpsf, this is due to there only being a limiting magnitude in the unforced ... the limits are given in a faded grey.

Not sure what is happening with the miss-matches of psf mags and fluxes - I guess you are just reading from the forced input as you are giving it ?

The forced photometry is in an entirely separate table from the unforced photometry in Cassandra. I do a match on MJD and filter-id to associate the forced-phot measurement with the equivalent unforced measurement. This allows me to then present the data in the one table at the bottom of the page.

I will make all of the change requests now and ping you when they're live.

thespacedoctor commented 10 months ago

Change Request Updates

These changes are now live on the dev server:

The whitespace between the plots has been trimmed.
The zero-line has been reverted from a shaded area to a solid line.

The duJy on the forced photometry cursor hover now shows the correct value

I have renamed columns in the HTML alert table, and the plain-text CSV and JSON outputs

Old Column Name	New Column Name
MJD	MJD
filter	filter
magpsf	unforced_mag
magpsf_error	unforced_mag_error
flux_status	unforced_mag_status
microjansky	forced_ujy
microjansky_error	forced_ujy_error

TODO: @RoyWilliams to update the lightcurve docs to explain the difference between forced and unforced photometry.

genghisken commented 9 months ago

Hi all

Chris F is having issues displaying a particular object. Here's the webserver log. 99% sure this is a forced photometry problem. It it reading data from the forced phot but there are no valid data points?? https://lasair-ztf.lsst.ac.uk/objects/ZTF23abrgldj/

ubuntu@lasair-ztf-web:~/mod_wsgi$ tail -f error_log
[Fri Jan 12 13:25:52.887004 2024] [mpm_event:notice] [pid 1863264:tid 140455440247872] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.9.2 Python/3.8 configured -- resuming normal operations
[Fri Jan 12 13:25:52.887085 2024] [core:notice] [pid 1863264:tid 140455440247872] AH00094: Command line: 'apache2 (mod_wsgi-express) -f /home/ubuntu/mod_wsgi/httpd.conf -D MOD_WSGI_KEEP_ALIVE -D MOD_WSGI_WITH_PYTHON_PATH -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE'
[Fri Jan 12 13:26:38.974205 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] Cluster.__init__ called with contact_points specified, but no load_balancing_policy. In the next major version, this will raise an error; please specify a load-balancing policy. (contact_points = ['lasair-ztf-cassandranodes-0', 'lasair-ztf-cassandranodes-1', 'lasair-ztf-cassandranodes-2', 'lasair-ztf-cassandranodes-3', 'lasair-ztf-cassandranodes-4'], lbp = None)
[Fri Jan 12 13:26:38.980346 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] Downgrading core protocol version from 66 to 65 for 10.66.3.222:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
[Fri Jan 12 13:26:38.982021 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] Downgrading core protocol version from 65 to 5 for 10.66.3.222:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
[Fri Jan 12 13:26:39.024645 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] Cluster.__init__ called with contact_points specified, but no load_balancing_policy. In the next major version, this will raise an error; please specify a load-balancing policy. (contact_points = ['lasair-ztf-cassandranodes-0', 'lasair-ztf-cassandranodes-1', 'lasair-ztf-cassandranodes-2', 'lasair-ztf-cassandranodes-3', 'lasair-ztf-cassandranodes-4'], lbp = None)
[Fri Jan 12 13:26:39.027406 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] Downgrading core protocol version from 66 to 65 for 10.66.3.44:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
[Fri Jan 12 13:26:39.028632 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] Downgrading core protocol version from 65 to 5 for 10.66.3.44:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
[Fri Jan 12 13:26:39.115355 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] /home/ubuntu/lasair4/webserver/lasair/apps/object/utils.py:50: SettingWithCopyWarning: 
[Fri Jan 12 13:26:39.115420 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] A value is trying to be set on a copy of a slice from a DataFrame.
[Fri Jan 12 13:26:39.115426 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] Try using .loc[row_indexer,col_indexer] = value instead
[Fri Jan 12 13:26:39.115431 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] 
[Fri Jan 12 13:26:39.115435 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
[Fri Jan 12 13:26:39.115440 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]   rBandNonDetections["name"] = "r-band limiting mag"
[Fri Jan 12 13:26:41.513349 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] Internal Server Error: /objects/ZTF23abrgldj/
[Fri Jan 12 13:26:41.513412 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] Traceback (most recent call last):
[Fri Jan 12 13:26:41.513418 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]   File "/home/ubuntu/.local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 56, in inner
[Fri Jan 12 13:26:41.513422 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]     response = get_response(request)
[Fri Jan 12 13:26:41.513426 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]   File "/home/ubuntu/.local/lib/python3.8/site-packages/django/core/handlers/base.py", line 197, in _get_response
[Fri Jan 12 13:26:41.513430 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]     response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Fri Jan 12 13:26:41.513434 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]   File "/home/ubuntu/lasair4/webserver/lasair/apps/object/views.py", line 59, in object_detail
[Fri Jan 12 13:26:41.513438 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]     fplightcurveHtml, mergedDF = object_difference_lightcurve_forcedphot(data)
[Fri Jan 12 13:26:41.513442 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]   File "/home/ubuntu/lasair4/webserver/lasair/apps/object/utils.py", line 223, in object_difference_lightcurve_forcedphot
[Fri Jan 12 13:26:41.513446 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]     forcedDF.loc[(forcedDF['fid'] == 1), "bcolor"] = "#606e03"
[Fri Jan 12 13:26:41.513450 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]   File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 818, in __setitem__
[Fri Jan 12 13:26:41.513455 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]     iloc._setitem_with_indexer(indexer, value, self.name)
[Fri Jan 12 13:26:41.513458 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]   File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1718, in _setitem_with_indexer
[Fri Jan 12 13:26:41.513463 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030]     raise ValueError(
[Fri Jan 12 13:26:41.513467 2024] [wsgi:error] [pid 1863265:tid 140455039915776] [remote 127.0.0.1:58030] ValueError: cannot set a frame with no defined index and a scalar

thespacedoctor commented 9 months ago

I'll get a hotfix in asap

thespacedoctor commented 9 months ago

The hotfix has been issued and pushed into production.

The issue occurred where there is forced photometry available, but it is ALL crap and therefore gets filtered to zero data points. The code was then trying to plot those zero points and falling over.

genghisken commented 9 months ago

Excellent. I thought it was exactly that! (Have run into this before with Pan-STARRS!)

lsst-uk / lasair4

BETA TEST: ZTF Forced-Photometry #210

Change Request Updates