search

LSSTDESC / Twinkles

10 years. 6 filters. 1 tiny patch of sky. Thousands of time-variable cosmological distance probes.
MIT License
13 stars 12 forks source link

update flowchart #30

Open SimonKrughoff opened 8 years ago

SimonKrughoff commented 8 years ago

The flowchart was written to reflect the state of things at the early 2015 DESC meeting.

We should update the current flowchart to reflect the desired workflow. I think @danielsf can probably inform the us on the flow chart.

danielsf commented 8 years ago

It looks about right. The main thing that jumps out at me is that the flowchart makes it look like the CatSim catalogs are generated independently of the OpSim pointings. We will probably not want to do anything with CatSim until we know which pointings we are simulating, since that will constrain our queries to fatboy, and make everything run a bit faster.

drphilmarshall commented 8 years ago

Right. We must be able to make a more accurate flowchart now! @danielsf , would you like to take a stab at minimally editing the flowchart on confluence so that it is as accurate as possible, given what you know (or can find out from @jbkalmbach and @rbiswas4) about how the SN and SL sprinkling work?

danielsf commented 8 years ago

@rbiswas4 and @jbkalmbach might want to check, but I have updated the flow-chart to reflect (my understanding of) the inter-dependency between the pointings, CatSim, and the sprinkler.

drphilmarshall commented 8 years ago

Thanks @danielsf ! The original concept for the flowchart at https://confluence.slac.stanford.edu/display/LSSTDESC/Twinkles+flow+chart was that all code components would be shown in ellipses, and exist as named in the Twinkles repo (or elsewhere). What's the piece of code that actually executes the database query you have written in green? It can't be done by hand, because that would be irreproducible (not to mention tedious :-)). Do we have a missing ellipse?

cwwalter commented 8 years ago

Your confluence based flow-chart "confuses and frightens me" :)

What is going on? Is that a plugin of some sort?

The left side is cutoff for me. Can others see it OK?

drphilmarshall commented 8 years ago

Nope, it's cut for me too. @danielsf I think you'll have to shift things back over a bit.

On Mon, Feb 29, 2016 at 1:50 PM, Chris Walter notifications@github.com wrote:

Your confluence based flow-chart "confuses and frightens me" :)

What is going on? Is that a plugin of some sort?

The left side is cutoff for me. Can others see it OK?

— Reply to this email directly or view it on GitHub https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/30#issuecomment-190412249 .

danielsf commented 8 years ago

Do we really want the flow chart to be this granular? Even for me, the flow chart we have is pedagogically confusing. Would it not be more useful to just say

OpSim simulates pointings We select the pointings we want to simulate We perform a SQL query against the UW database to find all of the objects in those pointings We use the sprinkler to add "artificial" variability sources CatSim translates the results into a form the PhoSim can read We use PhoSim to generate images

Users can then consult the documentation in detail on whichever part of the process they are most interested.

Or did we explicitly decide not to do that?

danielsf commented 8 years ago

Part of my concern is that "CatSim" encompasses both the software that performs the query on the UW database and the software that formats the output of that query into a text file that PhoSim can read, so either we have arrows going in multiple directions into and out of the "CatSim" ellipse, or we have multiple "CatSim" ellipses, and I'm not sure which of those scenarios is the most human readable.

drphilmarshall commented 8 years ago

I see your point. Your comment about "human readabiity" made me realize that the other reason (as well as pedagogy) for making the flowchart was to enable @tony-johnson and @TomGlanzman to start writing the XML files and python task scripts to configure and feed the SLAC workflow engine. This was successful, in that the engine is running PhoSim as we speak! We are not yet running the CatSim parts here, which is why you're getting odd-sounding questions from me :-) @danielsf I think: if from your point of view the diagram teaches the structure of the pipeline as well as possible, it's time to assign this to @tony-johnson and punt it to the next milestone, so that when we do get set up to run the whole simulation pipeline here (still to be issued, I think), we can finish off the flow chart then. Thanks!