Reply to Walter Jaffe

[hugobuddel]

Bernhard Brandl wrote:

Please find attached the PIP design doc with review comments by Walter Jaffe.

Walter is a retired professor at Leiden Observatory who was heavily involved in MIDI and MATISSE at the VLTI, and (long time ago) he has been the head of the software group at STScI. Wlater did an excellent job as internal-external reviewer of the CFO Design & Analysis document in preparation for the system-FDR last year, and I had asked him if he could also have a look at the main PIP document (just considering the "top view").

I hope his feedback is valuable. His comments may be more representative of those of a general FDR board member, that other team-internals.

See E-REP-AST-MET-1006_0-4_Data_Reduction_Library_DesignWJ.pdf. This issue tracks the comments raised by Walter. We don't have a formal obligation to reply (to get the document approved), but we plan to give a response anyway.

Comments are described on three levels:

• Some have their own issue, where I think we should take action, e.g. #154
• Some have a comment here, which do not require action within the scope of the DRLD or our commitments to ESO (most comments are about things we should do 'on the side')
• Some comments about the spectroscopy are not (yet?) described in this issue

Walter wrote

General comments: I see no estimates of necessary man-years. ESO may not require this in this document, but it is very important (and probably large).

Should be kinda in the development plan section. Indeed beyond the scope of the document itself.

You have no discussion of documentation and testing. For a large software project these are typically more than half of the work. What documentation will you deliver (user level, maintainance level)? How will you test the algorithms?

That is all in the DRLVT.

Most of the coding is in C using the extremely awkward CPL and HDRL interfaces. This requires a lot of very specialized manpower for development and maintainance that you need to incorporate in you management plans. If you are also using Python packages, specify how these are integrated into CPL/eso-rex environment.

There will be PyCPL and PyESOREX and EDPS, all official software by ESO. But not fully officially announced, so we need to tiptoe around that.

Do you use any centralized design and maintainence tools, e.g. object-oriented flow graphs, data dictionaries, test data libraries, test reports.

Officially, no. But unofficially (that is, for me personally...) there is all that. Would be good to formalize that though.

What is your code-management strategy?

Ssst. Officially we'll use the SVN by ESO. Unofficially we'll use our own git until we are allowed to use gitlab from ESO.

Would you consider having a parallel fast-prototyping code development environment, probably in Python, for algorithm development in a much friendlier environmment than CPL (i.e. with graphics and interactive input, debugging tools).

I personally do.

Do you have performance goals and resource estimates? How long does it take to reduce one night's observations? How many TBytes of disk space would a normal user need? What type of processor is needed?

There is a section for the data rate in this document.

What is post-processing and what isn't? The term seems to be used informally here, but I do not think that ESO recognizes its meaning. Will you have post-processing tools for processing larger data sets than a single OB, e.g. tracking AO performance during a whole night or week.

Post-processing is more or less "post-calibration processing". We do not officially have any post-processing for larger data sets than a single OB.

Note that the current ESO archive system only indexes observations by a few keywords and it is not possible e.g. to find and retrieve all observations made with seeing < 1" and airmass<1.5 for global quality control. You might consider a parallel database (I have my own python version, and the MATISSE team has its own version) where all of the keywords and data for all commissioning observations are kept.

We def. should!

Comment about fluxes moved https://github.com/AstarVienna/METIS_DRLD/issues/154

Side-comment: MATISSE and other groups are developing fairly generic Python based modelling tools for converting geometric or physical models in to simulated observations, and optimizing the match with respect to the model parameters. It is specially suited for mid-IR observations with databases on dust properties. This is not a deliverable in your project, but you might consider how to integrate it into your reduction environment.

That sounds cool! Definitely should check that framework out.

[hugobuddel]

p23

What is ADI? This is not a well known term for outsiders. What is the distinction between processing and post-processing??

Rephrased to "post-calibration processing" and added a reference to the ADI section. ADI is in parenthesis, so not too important, and the link allows people to learn more.

[hugobuddel]

p 25

I recommend adding a "Generic Engineering Template" with many input parameters to allow testing at a higher level than the technical Instrument Control System interface.

While perhaps a good idea, changing the templates is beyond the scope of the DRLD.

[hugobuddel]

p25

For consistency with the IFU formats why not put these images in extensions.

Indeed, that is changed already.

[hugobuddel]

Use of binary tables in https://github.com/AstarVienna/METIS_DRLD/issues/152

[hugobuddel]

On section 4:

As a reader I would appreciate performance requirements with these levels as these will drive hardware and software design requirements. E.g.. a QC0 pipeline should produce quick-look data quality products in (??10 min??) including FITS outputs that can be viewed by the observers in Near Real Time. ... QC2 pipelines running in the archive environment should produce standard outputs from 1 night's observing in <(??5??) hours...

Perhaps this would be useful. But I don't think we should commit to anything now, unless explicitly asked by ESO to do so.

[hugobuddel]

About the "Science-Grade Desktop Environment":

In practice, eso-REX+Reflex are an entirely inadequate scientific Desktop environment. They are grossly difficient in flexibility, accountability, and graphic aids. All of the newer instrument teams on the VLT have developed "local" tools, typically as Python packages, to fill these needs. This is not an ESO requirement but is necessary, especially for the instrument team, to efficiently process and analyze data. The underlying problem is that ESO does not maintain or distribute this code to users outside the team, so inevitably an insider/outsider relationship develops between users connected to the team/others.

We know.. We expect to use the EDPS.

[hugobuddel]

About reflex:

These tools currently do not provide graphical outputs, nor do they provide architectures to combine multiple data sets. E.g. you need tools to search the archives for all the calibrators observed during one night, assemble them, reduce them consistently, and plot the quality measures (seeing, atmospheric transparency, strehl ratio) as a function of time, flux, air mass,...

We might need to do that as a consortium, but it is beyond the scope of the official pipeline, and that is what we are focusing on now.

[hugobuddel]

p29

Language confusion-- you write "standard data" which I took to be the opposite of "non-standard data". You mean "calibrator data" or "standard target data"

Yes. Already changed.

I would expect a "flux-calibrated image" to be in units of erg/s/pixel or erg/s/micron/pixel or Jy/arcsec^2 or other physical unit. What do you mean by "flux calibrated". Does this include removing sky transmission? Instrument transmission? Conversion to physical units?

Roy actually praises us for not going to erg or Jy, so maybe that is fine. Clarified this sentence a bit, that we calibrate against a standard star.

[hugobuddel]

Some comments on p33 to p34 about LSS, left out of this issue for now.

[hugobuddel]

on 4.3.2 "Static Calibration Database"

What is in the database: do you record meta-data and history trees for each map/table? Are all the older versions retained in case the most recent one is corrupt? Are there dependency relations so that if one datum is corrupt you can locate the ones that are derived from it?

Well.. This is ESO's database, which does kinda contain all that information. But well, not really that well. Another database is beyond our scope.

[hugobuddel]

Two notes on telluric correction on p 48 and 50. Omitted here for now.

[hugobuddel]

THree notes about LSS wavelength calibration on page 54.

[hugobuddel]

On LM-band imaging flatfield:

How often do these need to be done?

I suppose daily. But that is part of the Calibration plan.

Twilight flats will be very tricky in the midIR under operational circumstances.

We'll see... We just follow the calibration plan..

[hugobuddel]

Replies by WK: E-REP-AST-MET-1006_0-4_Data_Reduction_Library_DesignWJ_WK.pdf

[wkausch]

What is your code-management strategy? Ssst. Officially we'll use the SVN by ESO. Unofficially we'll use our own git until we are allowed to use gitlab from ESO.

What does this "Ssst." mean?

[wkausch]

Concerning Walter's comment

"Note that the current ESO archive system only indexes observations by a few keywords and it is not possible e.g. to find and retrieve all observations made with seeing < 1" and airmass<1.5 for global quality control. You might consider a parallel database (I have my own python version, and the MATISSE team has its own version) where all of the keywords and data for all commissioning observations are kept."

It actually is possible to increase the search parameters, e.g. for seeing and/or airmass. There's the instrument specific ESO archive:

https://archive.eso.org/cms/eso-data/instrument-specific-query-forms.html

where one can refine the search very much.

[hugobuddel]

What is your code-management strategy? Ssst. Officially we'll use the SVN by ESO. Unofficially we'll use our own git until we are allowed to use gitlab from ESO.

What does this "Ssst." mean?

Sorry, I was just joking that code-management is a sensitive subject that shouldn't be discussed so openly. I do not mean to write that explicitly to Walter Jaffe. I found it 'interesting' that Walter gave many comments where I though "yes you are right, but I don't know how to reconcile that with our official ESO stance".

Like source control. To me it is insane to use subversion in 202x, so I would not write that down. At the same time we are also not allowed to say we'll use git, because ESO is only experimenting with that. So we need walk a tightrope and write something meaningless like that we'll use ESO's version control system, and then independently of that push to be allowed to use git.

[hugobuddel]

It actually is possible to increase the search parameters, e.g. for seeing and/or airmass. There's the instrument specific ESO archive:

https://archive.eso.org/cms/eso-data/instrument-specific-query-forms.html

where one can refine the search very much.

OK, that is good to hear. Nevertheless, IMO, we should try to get an consortium-wide archive setup in which we can do all those things that Walter talks about ourselves, like share calibration data. But that goes way beyond our current objective of passing FDR.

[wkausch]

Concerning the versioning: No one can forbid us to use git. I would therefore stay vague and answer: "We internally use git for the versioning. In conjunction with ESO we use what they want us to use."

Concerning the internal archive: I agree, what we do within the consortium is beyond any deliverable for ESO. I propose as answer to Walter:

"The ESO instrument specific archive pages (add link) allow much more refined search parameters including technical setup and observational conditions than the standard archive interface. Nevertheless, it is a good idea to have a more user-friendly internal database. Thank you for this proposal."

Maybe he is even happy to hear about the ESO archive possibilities.