Closed yuchaz closed 6 years ago
Hi @yuchaz I had a request from a user to update the PullLightcurves notebook, so I did that one last week (sorry - didn't know you were working on it too). Can you have a look at what I did there? (It's on master) I think it was somewhat useful to rearrange a few things and add the cell that highlights the difference between v3 and v4, some of which may be useful for you to carry forward for other future work. I'd rather use that version than the one here, which makes the PR a bit more difficult to merge. Perhaps you can pull in the sims-maf-contrib master version of the PullLightcurves notebook and update the PR?
Some other notes - please add "from future import print_function" to the head of the notebooks, as we may still have people using py2 for a while (although they will be expected to have future, from the LSST python installation). You don't have to add the #changelog notes to the notebook cells, as we intend these for users to go through as examples (and these comments may be confusing for someone coming in and seeing the notebook for the first time) -- adding them to the git commit message would be nice though.
Can you update "runName" to be the same as the opsim database name? In some of the notebooks (ComplexMetrics, for example) the run name is only set in the metric bundle, and not earlier, and so it's easy to get out of sync. It would be good to change the notebook to set this earlier. (if you notice, the runName goes into the plots, and so records what opsim database/ opsim run the plots reflect -- this should be consistent with what the run actually is).
I'm a little puzzled about the multiple chip warning in the CameraGeom notebook - I don't think this should happen and will check with the sims camera geometry package maintainer about the error message. Before I go too far there though, can you please try running the notebook again with Owen's newer docker image (sims 2.5.0)? I'm kind of hoping it's a bug that got fixed.
This is a great start!
Thank you @rhiannonlynne I will edit them :) . And sorry for the PullLightcurves
notebook issue. I should have update my forked repo before creating the pull request
Hi @rhiannonlynne I fixed the notebooks. As for the the multiple chips warning in CameraGeom.ipynb
notebook, it's still there even if I ran with latest MAF (version 2.5).
Hi @yuchaz thanks!
I think I may have been confusing when I was using the OpsimDatabase.opsimVersion information before. Clearly, we should probably write up a little summary of the differences between v3 and v4, which would be helpful. There are also changes between older versions of MAF and the current version, which have been added to handle both V3 and V4 versions of the Opsim output databases. So there's a lot going on! The notebooks need to update from old MAF calls to new MAF calls (the primary thing here is the availability of new kwargs and default values, and moving the sqlWhere from a function in utils to a method on OpsimDatabase). They also need to update to handle the new OpsimDatabase outputs (V4 instead of just V3). These differences are column names, but also that all angles in v4 are in degrees instead of in radians in v3. These changes can be accomodated by updating the kwargs, and using a kwarg set to indicate degrees or radians.
I realize now that this also makes the rest of the notebooks much more complicated. I guess it's good to show the differences, but if you'd prefer to leave everything in the V4 default mode, I think that would be perfectly reasonable as well (in which case, a lot of the comments below here you can ignore) -- but then it'd probably be good to remove the v3/v4 option as well.
So this will seem picky but here's how including the v3/v4 differences affects the updated notebooks:
ComplexMetrics.ipynb: cell 4. In an older version of MAF, the sqlWhere clause was in utils (as you have it here under "V3"), but now it is always a method on OpsimDatabase - whether or not the database itself is an Opsim V3 or Opsim V4 database. So in this case, the sqlconstraint would be created in the same way regardless of whether opsim is v3 or v4. (The "version" under OpsimDatabase.opsimVersion refers to the opsim version itself, not MAF).
DitherStacker.ipynb: cell 12. I like how you're using the v3 vs. v4 version information here, but the difference between the databases also includes a difference between degrees and radians for the angles, so you should include a variable called something like "degrees" (or inDegrees?) which is True for V4 and False for V3. When you set up the HealpixSlicer, there is a kwarg called "latLongDeg" which would be set to this value. And to be honest, it makes the rest of the notebook more complicated because you have to add explicit instantiations of the stackers so that you can set a kwarg ('degrees') equal to the same value. (The way MAF works, these stackers are automatically set up and run to generate the dithered ra/dec values that are asked for by the HealpixSlicer in each cell -- but the automatic setup doesn't work correctly if you have to set any kwargs to anything other than the defaults). It would be good to do this extra illustration, but if you want to skip it, I think it's fine too -- but then I'd say go back and remove the V3/V4 option for expMJD/observationStartMJD as well.
MafCameraGeom.ipynb: Similarly, this one gets a bit more complicated when you completely account for the degrees/radians difference. In particular, the sqlWhere clause in cell 3 should be updated so that the values for the RA/Dec values in the sql constraint are appropriately either degrees (for v4) or radians (for v3). In cell 8, the 'degrees' value should be used to set latLongDeg again (like you did in cell 4) and the stacker to set up the 'ditheredRa' and 'ditheredDec' values should be explicitly set up -- like
stacker = stackers.DefaultDitherStacker(fieldIdCol=fieldIdCol, degrees=degrees)
slicer = slicers.HealpixSlicer(latCol='ditheredDec', lonCol='ditheredRA', latLonCol=degrees, nside=nside)
slicer2 = slicers.HealpixSlicer(latCol='ditheredDec', lonCol='ditheredRA', latLonCol=degrees, nside=nside, useCamera=True, radius=1.9)
bundle1 = metricBundles.MetricBundle(metric, slicer, sqlWhere, stackerList=[stacker], summaryMetrics=summaryMetrics)
bundle2 = metricBundles.MetricBundle(metric,slicer2,sqlWhere, stackerList=[stacker], summaryMetrics=summaryMetrics)
bg = metricBundles.MetricBundleGroup({'NoCamera':bundle1, 'WithCamera':bundle2},opsdb, outDir=outDir, resultsDb=resultsDb)
bg.runAll()
(and where you'd set fieldIdCol as another variable back in cell 3 -- for version 3 it would be 'fieldID' and for version 3 it would be 'fieldId'). Then in cell 10, the sqlWhere would be "%s =2266 and night < 500" % fieldIdCol and again add latLonDeg=degrees in the HealpixSlicer.
Slicers.ipynb: A bit more of the same, but the differences here come up in not just expMJD vs observationStartMJD but also fieldID vs fieldId and the degrees thing again. So then in cell 6, the latLongDeg kwarg for the HealpixSlicer has to be set appropriately and the OpsimFieldSlicer in cell 7 has to have the kwargs related to fieldId name set appropriately (fieldID vs fieldId).
Hi @rhiannonlynne I just fixed what you pointed out. Let me know if there's any other things that need to be fixed.
Looks pretty good, just one more problem with the Dither notebook. Could you either take out the V3 options (i.e. setting the expMJD/degrees for v3 & v4) or add explicit stackers for the dither stackers (like you did in the MafCameraGeom notebook, cell 8)? As it stands currently, the notebook wouldn't work for a v3 database because the stackers don't have the fieldId column + the angles are in degrees -- here's an example v3 database so you can try it out! http://astro-lsst-01.astro.washington.edu:8081/astro/store/pogo4/db_gzip/minion_1016_sqlite.db.gz)
Hi @rhiannonlynne I just fixed Dithers.ipynb. However, there are two things I would like you to ask/check.
Hi @rhiannonlynne I just updated all the notebooks and the description texts on Index. I also did the sphinx documentations but I think I should separate it with different PR and create it when all the descriptions are revised. Beside this, I have several questions to ask:
VectorMetrics.ipynb
, I change the col
in metrics from finSeeing
to seeingFwhmEff
. Is that the correct way to change it? I cannot find the description of finSeeing
since it's not even exist in v3 dataset.Spatial_Coordinates.ipynb
, I changed the ditheredRA
and ditheredDec
to fieldRA
and fieldDec
if its v4 dataset. (because the SummaryAllProps
table in v4 dataset does not have dithered*
columns. But seems like its redundant for v4 because the output is the same as cell [8].HybridCameraGeomStudies.ipynb
does not show anything. However, I used the same sql constraint as that used in ChipNames.ipynb
. Therefore, the result should be the same as cell [8] of ChipNames.ipynb
. Do you know how to solve it?Hi @yuchaz sorry for the delay. Owen is going to deal with more detailed comments, but I wanted to add:
Fix the following notebooks.