Questions and suggestions wrt Quickstart tutorial

[dfm]

Instrument

No response

What happened?

As part of https://github.com/openjournals/joss-reviews/issues/4503, I'm going through the "Quickstart" tutorial, and here I'm including comments and suggestions.

• I think it would be useful to include a high-level overview of how Eureka works either at the beginning of this document, or on the index page, or both. In particular, I mean that this tutorial suggests that the standard workflow will be to write configuration files and then run a command line script, rather than interfacing with the Python API directly. For example, is it expected that all users would start by copying the demo project and then edit the .ecf files for their own purposes, or is the demo specific to these example data? I think it would be very useful to provide more context here, rather than just jumping in.
• The demo run_eureka.py script has some path manipulation (sys.path.insert(0, f'..{os.sep}..{os.sep}')) which is not required given that the tutorial expects that Eureka is installed and that we're supposed to be working in a separate directory when running the quickstart. This could also cause issues with import paths etc. depending on where the working directory is located. I think it should be removed.
• It would be useful to be clear within the tutorial that the inputdir and outputdir are both relative to topdir (I know that the demo files say this, but it should be in the tutorial text too) because /Stage2 (for example) is not a great path to see! I'd say that overall this interface feels clunky: is there a reason why topdir can't be defined once and then reused in the other stages with the subdirectory names can be set automatically?
• When running run_eureka.py, there were a handful of warnings (deprecation warnings, sqrt of negative number, etc.) buried in the logs (the full log is included below). Are these anything to be worried about? The tutorial should mention what to expect here and when not to worry.
• When comparing to the results figures, I don't get exactly the same results. All of the numbers (e.g. MAD) in the titles are somewhat different, and the axis limits in some of the figures vary significantly. If differences are expected (why?) when running with the same datasets and following the steps exactly as given, the tutorial should make it clear that we expect these differences, and how much difference is too much. Overall we just want to be able to check that things ran ok when following the tutorial on a local machine.

Full output log from run_eureka.py: https://gist.github.com/dfm/266b66102b0ed57c360ec3da17762c08

Error traceback output

No response

What operating system are you using?

macOS 12.6 (M1)

What version of Python are you running?

3.9.7

What Python packages do you have installed?

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[taylorbell57]

Making myself a terse checklist to keep track of what I've already taken care of

• [x] Include a high-level overview of how Eureka works either at the beginning of the quickstart document, or on the index page, or both. Clarify that users will continue to use this same format of interacting with the pipeline
• [x] Remove (sys.path.insert(0, f'..{os.sep}..{os.sep}')) from run_eureka.py
• [x] Clarify in the quickstart that inputdir and outputdir are both relative to topdir.
• [x] Investigate deprecation warnings and sqrt of negative number warnings, and reassure users in quickstart if they can. be ignored.
• [x] Update quickstart figures so that users get the same results as those in the tutorial (to offer a kind of validation test)

As for your comment:

... I'd say that overall this interface feels clunky: is there a reason why topdir can't be defined once and then reused in the other stages with the subdirectory names can be set automatically?

• I get your feeling, but there are good reasons that things are this way. If someone renames a folder or moves it to a different hard drive, then assuming too much about the file structure and its permanence breaks things. And someone may want to email me their Stage 3 output folder and let me try running Stages 4-5 myself, and then the topdir and subdirectories will change. Or if I rename a particular run to some other name manually to remind me that this folder is my best reduction, then I need to be able to pass that renamed folder's name into Eureka. Previously I had written things to have a lot of assumptions about folder names, and we ran into a tonne of issues. This level of specificity and automation seems to have worked well though. In general, you set the same topdir in each of your ECFs, set the inputdir and outputdirs to just Stage N-1 and Stage N, and then you're good to go. If you want to analyze an old Stage3 output for example, then you can change inputdir in Stage 4 to "Stage3/S32022-10-01 wasp39b_run1" or something like that to specify that the code should not use the most recent folder but instead that specific folder.