Run ESPRESSO Pipeline

[arpita308]

Download, install, and run ESPRESSO pipeline (https://www.eso.org/sci/software/pipelines/espresso/espresso-pipe-recipes.html). We will want to refer and compare our performance to the ESPRESSO pipeline as the end-to-end state of the art in precision pipelines. Many formerly confidential algorithms used by the HARPS/ESPRESSO team are also available for scrutiny in this code so we should know how to look up relevant parts for each module.

[AbbySh]

Have been working on installation - ran into issues and emailed back and worth with an ESO user helper and seemed to fix the issue. However, running into a problem installing one of the dependencies (esopipe-espdr-datademo). Seemed to be installing more than an hour and a half, restarted terminal and still seems to get stuck while installing. If it doesn't install by the end of today I will send another email.

[AbbySh]

I believe it is finally installed!

[AbbySh]

Errors/problems I have run into so far with ESPRESSO:

Download/Installation:

1. Began installation, was then told I needed the Java Development Kit (JDK). I don’t believe it says that I require this anywhere on the instruction page

2. Had further problems, it became clear that the issue I was having would be resolved in the next update to ESPRESSO online. ESO helper then strongly recommended I use MacPorts. I then had to install MacPorts and Xcode command line tools, as well. To be fair, it does say that they recommended using MacPorts on the main installation page for macOS 10.11 - 10.14, but not 10.15 which is the most up to date operating system. Hence, I disregarded the recommendation at first.

3. “[ ERROR ] esorex: Unable to find recipe 'recipe'. Check that the recipe is in the path specified by the '--recipe-dir' option.” After typing esorex recipe data.sof

Positive aspects I have noticed:

Download/Installation:

1. ESO virtual help desk was very helpful. Provided specific instructions for troubleshooting and recommended many different ways to go about installation (aka, didn’t just answer the question I had, but went above and beyond.

2. The online instructions do have some specific information that is helpful (it isn’t assumed that the person downloading is an expert in this)

[AbbySh]

Need to open new issue with ESO helpdesk for problem 3 above

[AbbySh]

Got another reply from them, working on this.

[AbbySh]

Tried running the quick start from the esoreflex user manual as per their recommendation, received the following error (have emailed them again):

java.lang.Exception: Failed to parse "-kepler -1" at ptolemy.actor.gui.ConfigurationApplication.throwArgsException(ConfigurationApplication.java:975) at ptolemy.actor.gui.ConfigurationApplication.(ConfigurationApplication.java:355) at ptolemy.actor.gui.MoMLApplication.(MoMLApplication.java:100) at ptolemy.vergil.VergilApplication.(VergilApplication.java:128) at ptolemy.vergil.VergilApplication.(VergilApplication.java:112) at ptolemy.vergil.VergilApplication$1.run(VergilApplication.java:273) at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:313) at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:770) at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721) at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:715) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85) at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:740) at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203) at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124) at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113) at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109) at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90) Caused by: ptolemy.kernel.util.IllegalActionException: Cannot set parameter -1 when no value is given. at ptolemy.actor.gui.ConfigurationApplication._parseArgs(ConfigurationApplication.java:1481) at ptolemy.vergil.VergilApplication._parseArgs(VergilApplication.java:536) at ptolemy.actor.gui.ConfigurationApplication.(ConfigurationApplication.java:277) ... 17 more

[AbbySh]

If I skip step 1 and move on to step 2, get this! -

Currently attempting run on a single demo data set - will update.

[AbbySh]

The output! Seems to have successfully run.

[AbbySh]

Was single dataset, this file: ESPRE.2018-10-22T02:09:40.652.fits

[DakotahTyler]

Taking over for the ESPRESSO pipeline:

I was able to get the pipeline installed on my personal machine. I did run into the Java (JDK) dependency problem that is mentioned earlier in this thread (11/10/2020) but the installation instructions include multiple different methods so I used the mac ports installation (preferred/suggested) which worked properly and automatically installed the software dependencies. Although I installed the pipeline to my personal machine, I will likely need to repeat this process on my shrek account as the reduction process is demanding with required minimum of 16GB RAM & 1TB free disk space, but suggested 32GB RAM & 4TB free disk space. The demo data set supplied ran fine on my machine but future tests will require more resources.

[DakotahTyler]

General comments on the simple demo set run through:

• Interactive workflow station is very user friendly and the demo tutorial is easy to follow. Side note: it is also possible to run the reduction straight from the terminal and this process is outlined in the FAQ section of the ESPRESSO cookbook

During the run, a window is populated with the current data set being reduced. For this particular run, I included all 5 of the available demo data sets to be reduced consecutively.

At the completion of the first data set, an interactive window pops up for a quick inspection of the reduction so far. I have included a couple of screenshots to show what options are available. One very convenient aspect here is that the user can adjust various recipe parameters and re-run the recipe directly from this page. The user also has the option to continue the reduction (workflow) and disable subsequent inspection windows from popping up for the remaining data sets:

Finally, once the reduction is complete, a "product explorer" window opens where you can inspect all of the products from the reduction, including all keywords and values for the created fits files:

[DakotahTyler]

Notes on ESPRESSO pipeline: The pipeline essentially consists of two separate pipelines; the data reduction pipeline (which produces the reduced spectra from raw data) and the data analysis software (DAS) pipeline (which is used for the various science objectives). The DAS pipeline includes various work flows which I think are worth highlighting.

• Workflows are broken down for optimization based on individual needs/science goals
  • Star I wkf - calculates quick T_eff estimate & metallicity (Fe/H) from comparison of observed line eq width ratio
  • Star II wkf - "quick" RV by comparison of observed spectrum with a synthetic spectrum
  • Star III wkf - accurate RV & stellar activity indicator log(R'HK)
• There are also two workflows for analyzing quasars
  • We may want to think about other likely unintended potential uses for KPF. With some foresight, structuring the pipeline in slightly different ways like this could broaden KPF's utility and maximize our science potential.

[DakotahTyler]

ESPRESSO data - Tau Ceti 198 observations spanning 172 days - Fully reduced and analyzed

• The RVs & associated errors stored under keywords "QC CCF RV" & "QC CCF RV_ERROR" in the CCF_A.fits files
• First a sanity check with the provided plot from the ESPRESSO team via the MoU (MoU plot on top)

  

Looks right

Total velocity disperson of the dataset

I then took a look at the dispersion over smaller time bins (on the order of hours)

calculated dispersions for the different time bins and a standard error on the dispersion

[awhoward]

Sweet!

[petigura]

Cool stuff, @DakotahTyler! I'm wondering if we can see p-modes in the Tau Ceti velocities. For fun, could you compute the LS periodogram of some of the high cadence chunks?

[arpita308]

Closing this issue since ESPRESSO pipeline has been run successfully by both @AbbySh and @DakotahTyler.

See follow up issues for further exploration of ESPRESSO data.