Is "Tutorial 2: Advanced Models and Fitting" reproducible?

[sangchengfang]

I run the script of "Tutorial 2" downloaded from https://tobiscode.github.io/disstans/tutorials/tutorial_2.html, but I cannot get the same results as shown on the tutorial webpage. I did not figure it out. Is "Tutorial 2: Advanced Models and Fitting" reproducible since we set np.random.default_rng(0)? The errors of constant and linear are also worrying. Please help me out, thanks.

My output:

python3.11 tutorial_2_detect_transients.py
Fitting station models: 100%|###############################################################################################################################################| 1/1 [00:00<00:00,  2.93station/s]
Evaluating station models: 100%|############################################################################################################################################| 1/1 [00:00<00:00,  1.90station/s]
TUT: Res_noreg                          Mean  Standard Deviation
Total-Displacement_Model_Total  1.705244e-08            2.042829
Fitting station models: 100%|###############################################################################################################################################| 1/1 [00:00<00:00,  3.10station/s]
Evaluating station models: 100%|############################################################################################################################################| 1/1 [00:00<00:00,  1.89station/s]
TUT: Res_L2                             Mean  Standard Deviation
Total-Displacement_Model_Total  2.080029e-09            2.087589
Fitting station models: 100%|###############################################################################################################################################| 1/1 [00:02<00:00,  2.52s/station]
Evaluating station models: 100%|############################################################################################################################################| 1/1 [00:00<00:00,  1.04station/s]
TUT: Res_L1                         Mean  Standard Deviation
Total-Displacement_Model_Total  0.000003            2.121951
Fitting station models: 100%|###############################################################################################################################################| 1/1 [00:33<00:00, 33.46s/station]
Evaluating station models: 100%|############################################################################################################################################| 1/1 [00:00<00:00,  1.72station/s]
TUT: Res_L0                             Mean  Standard Deviation
Total-Displacement_Model_Total  9.514120e-08            2.117921
Number of nonzero spline parameters:
L2: 264
L1: 41
L0: 34
Percent Error Constant:              -44.024%
Percent Error Linear:                 20.780%
Percent Error Annual Amplitude:      -0.234%
Percent Error Semi-Annual Amplitude:  0.062%
Absolute Error Annual Phase:          0.917°
Absolute Error Semi-Annual Phase:    -0.982°
Transient Velocity: 0.121516 mm/D

This is my output of creating the conda environment:

conda env create -f environment.yml
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
    current version: 23.9.0
    latest version: 23.10.0

Please update conda by running

    $ conda update -n base -c conda-forge conda

Downloading and Extracting Packages:

Preparing transaction: done                                                                                                                                                               
Verifying transaction: done                                                                                                                                                               
Executing transaction: done                                                                                                                                                               
Installing pip dependencies: - Ran pip subprocess with arguments:                                                                                                                         
['/home/scf/anaconda3/envs/test/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/scf/condaenv.0x7j7h7x.requirements.txt', '--exists-action=b']                                     
Pip subprocess output:                                                                                                                                                                    
Collecting git+https://github.com/tbenthompson/okada_wrapper.git (from -r /home/scf/condaenv.0x7j7h7x.requirements.txt (line 1))
  Cloning https://github.com/tbenthompson/okada_wrapper.git to /tmp/pip-req-build-6p5insxy
  Resolved https://github.com/tbenthompson/okada_wrapper.git to commit 7c9453828dc7a315ad5d6ba07cf9ff3e76fb0d29
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: numpy in ./anaconda3/envs/test/lib/python3.11/site-packages (from okada-wrapper==18.12.7.3->-r /home/scf/condaenv.0x7j7h7x.requirements.txt (line 1)) (1.26.2)
Building wheels for collected packages: okada-wrapper
  Building wheel for okada-wrapper (setup.py): started
  Building wheel for okada-wrapper (setup.py): finished with status 'done'
  Created wheel for okada-wrapper: filename=okada_wrapper-18.12.7.3-cp311-cp311-linux_x86_64.whl size=62168 sha256=ed614663bc46e8a0e4fd835e618c7f318a5278289b33f9dc87c5409109687bd4
  Stored in directory: /tmp/pip-ephem-wheel-cache-irtcqodn/wheels/52/d1/03/1c361a1945abf0c5709dfa5d5748454363c99e4820e4f3844e
Successfully built okada-wrapper
Installing collected packages: okada-wrapper
Successfully installed okada-wrapper-18.12.7.3

done
#
# To activate this environment, use
#
#     $ conda activate test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

❯ conda activate test
❯ pip install disstans
Collecting disstans
  Using cached disstans-1.1.1-cp311-cp311-linux_x86_64.whl
Installing collected packages: disstans
Successfully installed disstans-1.1.1

[sangchengfang]

I solved this by updating to dev version.

[tobiscode]

Hi,

yes, all the tutorials are designed to be reproducible (at least to within machine precision). There were differences because you compared the live version of the tutorial (from github.io, which is always based on the main branch and currently is version 2.0 Release Candidate) with the code from PyPI, which always tracks the latest stable release (which was version 1.1.1).

You can always look at the tutorial state as it was for version 1.1.1 if you clone the repository, checkout the v1.1.1 tag, and then look at the documentation locally (see README/Documentation). That should match the PyPI code. Alternatively, as you said, you can switch your code to the new version (2.0rc0 on branch main) and then it matches the live website as well. (The dev branch is already ahead of the website again so it might also not always match).

Small addendum: I'll be releasing the final version 2.0 very soon and then also PyPI will have the code that matches the website.

Hope that helped, Tobias