search

LSSTDESC / Monitor

Extract light curves for time-variable cosmological objects
BSD 3-Clause "New" or "Revised" License
3 stars 1 forks source link

Tiny example dataset #4

Open drphilmarshall opened 8 years ago

drphilmarshall commented 8 years ago

I think we need a very small data file checked in to this repo, to enable some basic tests to be run by Travis (and help development). @SimonKrughoff do you have a suitable ForcedSource table snippet that we could use for this purpose? I guess it needs at least two objects in it and at least a handful of epochs, and preferably multiple filters, but the objects needn't be anything special: all we're doing is validating at this point. Thanks!

drphilmarshall commented 8 years ago

I think this issue is the roadblock for Monitor development (which needs to happen fast if we are to be able to make plots of @TomGlanzman and @tony-johnson 's Twinkles outputs before March 7. Does anyone have a `ForcedSource catalog (or catalogs) that can be stripped down to 2 objects in 3 filters observed over N epochs for development purposes? @SimonKrughoff? @jbkalmbach?

rbiswas4 commented 8 years ago

I agree we need this very quickly, for understanding how to build the light curve class. I was going through @SimonKrughoff's plot_mags code as suggested by @jchiang87 and it would help to be able to see the data.

jchiang87 commented 8 years ago

@rbiswas4 Do you have a slac account? If so, you can try using the following output_data directory with Simon's code:

bash-4.1$ pwd -P
/nfs/slac/g/ki/ki18/jchiang/DESC/Twinkles/work/output_data
bash-4.1$ ls
_parent@                  forced/
calexp/                   forcedPhotCcd_metadata/
config/                   icMatch/
deepCoadd/                icSrc/
deepCoadd-results/            processEimage_metadata/
deep_makeCoaddTempExp_metadata/       schema/
deep_safeClipAssembleCoadd_metadata/  src/
bash-4.1$

Unfortunately, to use that code with the butler etc., the full output directory structure is probably needed, i.e., some sample forced source catalogs won't be sufficient.

rbiswas4 commented 8 years ago

@jchiang87

@rbiswas4 Do you have a slac account? This is in the works, but not yet. Your reply just cleared up a bunch of things I was confused about and was composing questions about: things like what the strings like 'output_data' were doing in that script.

Since I do not have an account yet, could you please upload that example dataset to nersc? I can download it from there to start looking at it. We can think about stripping it down to size for an example_data set after that. Thanks!

jchiang87 commented 8 years ago

Ok. I can try. I have a NERSC account, but I haven't tried using it yet.

rbiswas4 commented 8 years ago

Thank you!

On Sat, Feb 27, 2016 at 6:26 PM, James Chiang notifications@github.com wrote:

Ok. I can try. I have a NERSC account, but I haven't tried using it yet.

— Reply to this email directly or view it on GitHub https://github.com/DarkEnergyScienceCollaboration/Monitor/issues/4#issuecomment-189766208 .

jchiang87 commented 8 years ago

I wasn't able to access my NERSC account, so I uploaded the data to lsst-dev.ncsa.illinois.edu instead. The tarball, output_data.tgz, should be read-accessible:

[jchiang@lsst-dev Twinkles]$ pwd -P
/nfs/home/jchiang/Twinkles
[jchiang@lsst-dev Twinkles]$ ls
output_data  output_data.tgz
[jchiang@lsst-dev Twinkles]$ ls output_data/
calexp                               forcedPhotCcd_metadata
config                               icMatch
deepCoadd                            icSrc
deepCoadd-results                    _parent
deep_makeCoaddTempExp_metadata       processEimage_metadata
deep_safeClipAssembleCoadd_metadata  schema
forced                               src
[jchiang@lsst-dev Twinkles]$ ls -l output_data.tgz 
-rw-rw-r-- 1 jchiang jchiang 1260134730 Feb 27 22:28 output_data.tgz
[jchiang@lsst-dev Twinkles]$

HTH.

rbiswas4 commented 8 years ago

I do have an account on lsstdev, but I cannot access this directory because your home directory blocks permissions. But thanks for putting this up on lsstdev!

[rbiswas@lsst-dev ~]$ cd /nfs/home/jchiang/Twinkles
-bash: cd: /nfs/home/jchiang/Twinkles: Permission denied
rbiswas@lsst-dev home]$ cp /nfs/home/jchiang/Twinkles/output_data.tgz .
cp: cannot stat `/nfs/home/jchiang/Twinkles/output_data.tgz': Permission denied
[rbiswas@lsst-dev ~]$ cd /home/
[rbiswas@lsst-dev home]$ ls -ltr|grep jchiang
drwx------  11 jchiang       jchiang        22 Feb 27 22:07 jchiang
jchiang87 commented 8 years ago

Hi @rbiswas4, Can you try again? I just changed the directory permissions. I think you should be able to do cp /nfs/home/jchiang/Twinkles/output_data.tgz <some place local>

rbiswas4 commented 8 years ago

OK. I tried it now and still have that problem:

rbiswas@lsst-dev ~]$ cp /nfs/home/jchiang/Twinkles/output_data.tgz .
cp: cannot stat `/nfs/home/jchiang/Twinkles/output_data.tgz': Permission denied
SimonKrughoff commented 8 years ago

Sorry for my silence on this. Is there anything I can do to help? It sounds like @rbiswas4 is looking into it.

jchiang87 commented 8 years ago

@SimonKrughoff Can you provide @rbiswas4 with an output_data collection? You probably have the gri run handy. The one I put at lsst-dev is just the older r band analysis.

SimonKrughoff commented 8 years ago

O.K. I'll put this together tomorrow.

jbkalmbach commented 8 years ago

Ok, I've got Simon's data, but it's quite large so I'm working on making the smallest, most basic but useful dataset out of it that I can.

jbkalmbach commented 8 years ago

So, the output_data folder provided by @SimonKrughoff only had visits from the same MJD and the same band. @jchiang87 are yours any different? If not, would it possible for anybody to run dm processing on the new sprinkled outputs to help create a better test set? Or at the very least have any output_data folders lying around that span a few days/bands?

jchiang87 commented 8 years ago

Mine is just r-band for 9 visits, They might span more than one night but they won't be significantly different from Simon's, if at all. @tony-johnson should have the workflow engine going soon on the multiband/multiday data that are being generated with phosim. I can look into running the Twinkles cookbook on those data tomorrow to get some files for you sooner.

jbkalmbach commented 8 years ago

Yeah, anything from the actual twinkles data would be great if we can that!

jbkalmbach commented 8 years ago

Ok, small test set of forced photometry output available in 'data/test/forced'. ObjectIds run from 0-375. Now, by using one of those numbers in build_lightcurve(objectId), anybody can run the example notebook on their own.

jbkalmbach commented 8 years ago

I just realized we still need to get the true light curves associated with this test data. So, I reopened this issue until that gets done.