Notebooks updated for v3/v4 and python 2/3

[yuchaz]

Fix the following notebooks.

1. Complex Metrics.ipynb
2. Dithers.ipynb
3. MAFCameraGeom.ipynb
4. PullLightCurves.ipynb
5. Slicers.ipynb
6. Stackers.ipynb

[rhiannonlynne]

Hi @yuchaz I had a request from a user to update the PullLightcurves notebook, so I did that one last week (sorry - didn't know you were working on it too). Can you have a look at what I did there? (It's on master) I think it was somewhat useful to rearrange a few things and add the cell that highlights the difference between v3 and v4, some of which may be useful for you to carry forward for other future work. I'd rather use that version than the one here, which makes the PR a bit more difficult to merge. Perhaps you can pull in the sims-maf-contrib master version of the PullLightcurves notebook and update the PR?

Some other notes - please add "from future import print_function" to the head of the notebooks, as we may still have people using py2 for a while (although they will be expected to have future, from the LSST python installation). You don't have to add the #changelog notes to the notebook cells, as we intend these for users to go through as examples (and these comments may be confusing for someone coming in and seeing the notebook for the first time) -- adding them to the git commit message would be nice though.

Can you update "runName" to be the same as the opsim database name? In some of the notebooks (ComplexMetrics, for example) the run name is only set in the metric bundle, and not earlier, and so it's easy to get out of sync. It would be good to change the notebook to set this earlier. (if you notice, the runName goes into the plots, and so records what opsim database/ opsim run the plots reflect -- this should be consistent with what the run actually is).

I'm a little puzzled about the multiple chip warning in the CameraGeom notebook - I don't think this should happen and will check with the sims camera geometry package maintainer about the error message. Before I go too far there though, can you please try running the notebook again with Owen's newer docker image (sims 2.5.0)? I'm kind of hoping it's a bug that got fixed.

This is a great start!

[yuchaz]

Thank you @rhiannonlynne I will edit them :) . And sorry for the PullLightcurves notebook issue. I should have update my forked repo before creating the pull request

[yuchaz]

Hi @rhiannonlynne I fixed the notebooks. As for the the multiple chips warning in CameraGeom.ipynb notebook, it's still there even if I ran with latest MAF (version 2.5).

[rhiannonlynne]

Hi @yuchaz thanks!

I think I may have been confusing when I was using the OpsimDatabase.opsimVersion information before. Clearly, we should probably write up a little summary of the differences between v3 and v4, which would be helpful. There are also changes between older versions of MAF and the current version, which have been added to handle both V3 and V4 versions of the Opsim output databases. So there's a lot going on! The notebooks need to update from old MAF calls to new MAF calls (the primary thing here is the availability of new kwargs and default values, and moving the sqlWhere from a function in utils to a method on OpsimDatabase). They also need to update to handle the new OpsimDatabase outputs (V4 instead of just V3). These differences are column names, but also that all angles in v4 are in degrees instead of in radians in v3. These changes can be accomodated by updating the kwargs, and using a kwarg set to indicate degrees or radians.

I realize now that this also makes the rest of the notebooks much more complicated. I guess it's good to show the differences, but if you'd prefer to leave everything in the V4 default mode, I think that would be perfectly reasonable as well (in which case, a lot of the comments below here you can ignore) -- but then it'd probably be good to remove the v3/v4 option as well.

So this will seem picky but here's how including the v3/v4 differences affects the updated notebooks:

• ComplexMetrics.ipynb: cell 4. In an older version of MAF, the sqlWhere clause was in utils (as you have it here under "V3"), but now it is always a method on OpsimDatabase - whether or not the database itself is an Opsim V3 or Opsim V4 database. So in this case, the sqlconstraint would be created in the same way regardless of whether opsim is v3 or v4. (The "version" under OpsimDatabase.opsimVersion refers to the opsim version itself, not MAF).

• DitherStacker.ipynb: cell 12. I like how you're using the v3 vs. v4 version information here, but the difference between the databases also includes a difference between degrees and radians for the angles, so you should include a variable called something like "degrees" (or inDegrees?) which is True for V4 and False for V3. When you set up the HealpixSlicer, there is a kwarg called "latLongDeg" which would be set to this value. And to be honest, it makes the rest of the notebook more complicated because you have to add explicit instantiations of the stackers so that you can set a kwarg ('degrees') equal to the same value. (The way MAF works, these stackers are automatically set up and run to generate the dithered ra/dec values that are asked for by the HealpixSlicer in each cell -- but the automatic setup doesn't work correctly if you have to set any kwargs to anything other than the defaults). It would be good to do this extra illustration, but if you want to skip it, I think it's fine too -- but then I'd say go back and remove the V3/V4 option for expMJD/observationStartMJD as well.

• MafCameraGeom.ipynb: Similarly, this one gets a bit more complicated when you completely account for the degrees/radians difference. In particular, the sqlWhere clause in cell 3 should be updated so that the values for the RA/Dec values in the sql constraint are appropriately either degrees (for v4) or radians (for v3). In cell 8, the 'degrees' value should be used to set latLongDeg again (like you did in cell 4) and the stacker to set up the 'ditheredRa' and 'ditheredDec' values should be explicitly set up -- like

  stacker = stackers.DefaultDitherStacker(fieldIdCol=fieldIdCol, degrees=degrees)
  slicer = slicers.HealpixSlicer(latCol='ditheredDec', lonCol='ditheredRA', latLonCol=degrees, nside=nside)
  slicer2 = slicers.HealpixSlicer(latCol='ditheredDec', lonCol='ditheredRA', latLonCol=degrees, nside=nside, useCamera=True, radius=1.9)
  bundle1 = metricBundles.MetricBundle(metric, slicer, sqlWhere, stackerList=[stacker], summaryMetrics=summaryMetrics)
  bundle2 = metricBundles.MetricBundle(metric,slicer2,sqlWhere, stackerList=[stacker], summaryMetrics=summaryMetrics)
  bg = metricBundles.MetricBundleGroup({'NoCamera':bundle1, 'WithCamera':bundle2},opsdb, outDir=outDir, resultsDb=resultsDb)
  bg.runAll()

  (and where you'd set fieldIdCol as another variable back in cell 3 -- for version 3 it would be 'fieldID' and for version 3 it would be 'fieldId'). Then in cell 10, the sqlWhere would be "%s =2266 and night < 500" % fieldIdCol and again add latLonDeg=degrees in the HealpixSlicer.

• Slicers.ipynb: A bit more of the same, but the differences here come up in not just expMJD vs observationStartMJD but also fieldID vs fieldId and the degrees thing again. So then in cell 6, the latLongDeg kwarg for the HealpixSlicer has to be set appropriately and the OpsimFieldSlicer in cell 7 has to have the kwargs related to fieldId name set appropriately (fieldID vs fieldId).

[yuchaz]

Hi @rhiannonlynne I just fixed what you pointed out. Let me know if there's any other things that need to be fixed.

[rhiannonlynne]

Looks pretty good, just one more problem with the Dither notebook. Could you either take out the V3 options (i.e. setting the expMJD/degrees for v3 & v4) or add explicit stackers for the dither stackers (like you did in the MafCameraGeom notebook, cell 8)? As it stands currently, the notebook wouldn't work for a v3 database because the stackers don't have the fieldId column + the angles are in degrees -- here's an example v3 database so you can try it out! http://astro-lsst-01.astro.washington.edu:8081/astro/store/pogo4/db_gzip/minion_1016_sqlite.db.gz)

[yuchaz]

Hi @rhiannonlynne I just fixed Dithers.ipynb. However, there are two things I would like you to ask/check.

1. After upgrading the version of MAF to 2.5, some of the API changes therefore cell 8 will fail. I ask @oboberg and he suggested me this quick fix.
2. In this version, I use v3 dataset. And the output in cell 22 is sparse. Is that normal?

[yuchaz]

Hi @rhiannonlynne I just updated all the notebooks and the description texts on Index. I also did the sphinx documentations but I think I should separate it with different PR and create it when all the descriptions are revised. Beside this, I have several questions to ask:

1. In cell [8] of VectorMetrics.ipynb, I change the col in metrics from finSeeing to seeingFwhmEff. Is that the correct way to change it? I cannot find the description of finSeeing since it's not even exist in v3 dataset.
2. In cell [10] of Spatial_Coordinates.ipynb, I changed the ditheredRA and ditheredDec to fieldRA and fieldDec if its v4 dataset. (because the SummaryAllProps table in v4 dataset does not have dithered* columns. But seems like its redundant for v4 because the output is the same as cell [8].
3. The lower third figure in cell [16] of HybridCameraGeomStudies.ipynb does not show anything. However, I used the same sql constraint as that used in ChipNames.ipynb. Therefore, the result should be the same as cell [8] of ChipNames.ipynb. Do you know how to solve it?

[rhiannonlynne]

Hi @yuchaz sorry for the delay. Owen is going to deal with more detailed comments, but I wanted to add:

• if you're accounting for differences between opsim v3 and v4, you need to make sure that everything works for both --- i.e. if you have "expMJD" and "observationStartMJD", then you also need to add the consideration of whether angles are in degrees or not. The problem with altering the column name based on the opsim version, but not the 'degrees', is that you will end up with a notebook that runs on v3 or v4, but is very wrong for v3 (because all the values will be in radians, but the maf code assumes it's in degrees unless told otherwise). If you're in doubt, I would rather you not add any corrections to attempt to handle v3, but just stick to v4. (example where this is not working here - PlottingExamples.ipynb)
  • if you see a mention of the "MAF version" anywhere in the notebook, please delete it. I don't want to have to keep rerunning notebooks to make sure that the version info is updated, so let's put it in one notebook (the Index) and only there, deleting it from elsewhere. Usually this is showing up in the first comment block (something like "make sure you have MAF version > 1.1" etc).
  • watch out for line length .. when the notebooks render in github, long lines are wrapped and hard to read.
  • please replace any "sqlconstraint" kwarg names in the MetricBundles with just "constraint" (i.e. : bundle = mb.MetricBundle(metric, slicer, sqlconstraint=sqlconstraint) should become bundle = mb.MetricBundle(metric, slicer, constraint=sqlconstraint)) because the "sqlconstraint" kwarg name is being changed to simply constraint. You should actually get a warning message in the cell output after using one of these, so they should be easy to spot.