JuliaDynamics / DelayEmbeddings.jl

Delay coordinates embeddings and optimizing them
MIT License
27 stars 8 forks source link

Unified embedding tests are brittle #95

Open Datseris opened 3 years ago

Datseris commented 3 years ago

We have established the validity of our unified embedding methods because we even wrote a paper utilizing them in detail and comparing them with each other. There isn't any doubt that they work. However, the way we have written our tests is brittle, because (due to chaos) running the test on different machines may make them fail. For example while working on PR #93 I've seen things like:

Multivariate example: Test Failed at c:\Users\m300808\.julia\dev\DelayEmbeddings\test\unified\test_pecuzal_embedding.jl:72
  Expression: -0.356 < Ls[2] < -0.355
   Evaluated: -0.356 < -0.32831887625247136 < -0.355

Testing pecuzal_method.jl...
Initializing PECUZAL algorithm for univariate input...
Starting 1-th embedding cycle...
Starting 2-th embedding cycle...
Starting 3-th embedding cycle...
Algorithm stopped due to increasing L-values. VALID embedding achieved ✓.
  5.931637 seconds (15.70 M allocations: 2.972 GiB, 14.40% gc time)
Univariate example: Test Failed at c:\Users\m300808\.julia\dev\DelayEmbeddings\test\unified\test_pecuzal_embedding.jl:19
  Expression: -0.728 < Ls[1] < -0.727
   Evaluated: -0.728 < -0.7839819804977819 < -0.727

and even

Initializing PECUZAL algorithm for multivariate input...
Starting 1-th embedding cycle...
Starting 2-th embedding cycle...
Starting 3-th embedding cycle...
Algorithm stopped due to increasing L-values. VALID embedding achieved ✓.
 21.312898 seconds (71.51 M allocations: 11.922 GiB, 8.30% gc time)
Multivariate example: Test Failed at c:\Users\m300808\.julia\dev\DelayEmbeddings\test\unified\test_pecuzal_embedding.jl:55
  Expression: length(ts_vals) == 5
   Evaluated: 3 == 5

(i.e. a lower-dimensional embedding is produced sometimes)

We need to re-write tests in a different manner. First increase the error margins, but also come up with tests that are not so brittle and depend sensitively on the chaotic nature of trajectories. Furthermore, this issue needs to be solved in conjuction with #94 , which would make trajectories no longer simulated but rather available as .csv files from the get go.

@hkraemer I'm also cc'ing you for reference.

Datseris commented 3 years ago

Unfortunately PECUZAL is the most brittle, seems to have the most failures when running on different PC.

hkraemer commented 3 years ago

Hey @Datseris , I have also encountered an issue with the Uzal-algorithm, which also effects PECUZAL. I will take care of all of this once I'm back from vacation, so expect me to fix this by mid of August.

Datseris commented 3 years ago

Hi Hauke, sounds cool, no stress!

hkraemer commented 2 years ago

In #103 I also revised the test for mdop & garcia_almeida, which is why I think you can close this one here. However: I did dive deeper into the code. Especially garcia_almeida used to give some (physically) wired results as far as I can remember.

Datseris commented 2 years ago

Let's keep this open until #94 is resolved as well (i.e., all tested timeseries are loaded from csv files).