Benchmarking examples and tests

[simbilod]

Many projects provide benchmarks to analytical calculations, literature, or other codes

It would be good to include a section in the docs dedicated to showing agreement between femwell and other sources

Internally, we could also build tests around these benchmarks

[HelgeGehring]

Sounds amazing! It's really time to start adding tests :) The heater examples and the lithium niobate phase shifter already reproduce papers, but more would be great!

[blair-morrison]

Funnily enough I was chatting about this with Joaqin at OFC. There are likely many relevant benchmarks, but on the pure optical side step index fibers are a nice starting point due to the presence of analytic solutions. Section 3 of the Zhu and Brown 2002 Optics Express on Finite Difference eigenmode with index averaging has a couple of examples. Likewise the Yu and Chang 2004 Optics Express benchmarks to one of the papers from Hadley, though this isn't really a typical Si waveguide geometry it might still be a relevant comparison (if the boundary conditions are not an issue).

[HelgeGehring]

Sounds great, thanks!

The step index fiber is I think a little harder than it seems, as the core is round in those calculations. If we want to take that as a benchmark we should also take care of that. If we take a lot of elements, we could approximate the circle again, but then the question is where a potential error comes from. But still if we show the error is small it would still be a nice benchmark :) There's also ways to have curved elements in the mesh (@simbilod can gmsh do that?)

I really like the waveguide example your brought up, that'll be a great benchmark! Seems like a SiN waveguide. The boundary conditions for this one are on the roadmap. As the symmetry is here not broken by a substrate, the modes will probably be just TE or TM and not something in-between, which effectively simplifies the equations. So it would be great to have an additional example where that symmetrie is somehow broken (air-clad or rib waveguide? would you know a paper for that @blair-morrison ?)

I think https://helgegehring.github.io/femwell/photonics/examples/leaky_mode.html can also be a great benchmark, @bhnzhang mentioned that they solved it analytically in the paper I think.

[simbilod]

Yes Gmsh has 2nd order mesh elements "to generate a curvilinear mesh and optimize it to produce provably valid curved elements". It's just a flag so should be easy to implement

[HelgeGehring]

Nice, for that we probably need an API which supports circles?

[simbilod]

Yes the plan for meshwell is to allow shapely polygons (which are nice to use), or regular gmsh occ kernel objects including circles, splines, boxes, etc.

[HelgeGehring]

Great!

[blair-morrison]

So it would be great to have an additional example where that symmetrie is somehow broken (air-clad or rib waveguide? would you know a paper for that @blair-morrison ?)

Off the top of my head a good reference for this is still the Hadley paper [1]. As well as two analytic cases (a "low" and "high" index contrast), there is a numerical example of a silicon rib with an air top clad. The solver he used was compared to a few other numerical methods all looking at the same geometry, so I guess it was a benchmark used at the time in the literature. The benchmarking part in the Hadley paper referred to a paper from the 90s [2] that was comparing about 10 different techniques, some of which varied quite significantly!

[1] G. R. Hadley, J. Lightwave Technol. 20, 1219 (2002). 10.1109/JLT.2002.800371

[2] C. Vassallo, Optical and Quantum Electronics 29, 95 (1997). 10.1023/A:1018537602159

[HelgeGehring]

Amazing! I somehow scrolled over the air-clad example. Thanks! I'll start implementing these test-cases in the next days!

I'm happy about everything more. I'm especially thinking of quite simple cases which can validate boundary conditions. Something like cases which have a translational symmetry in one direction and can also be solved in one dimension, e.g. just a slab

Also for the heat solver we it would be nice to have some analytically solved examples, I didn't search for that yet, but that would be the next thing

[HelgeGehring]

The first benchmark is online: https://helgegehring.github.io/femwell/benchmarks/mode_solver.html

[HelgeGehring]

For now I prioritize #44 over additional benchmarks, as there the boundary conditions will be implemented in a more general form, which is required for the other benchmarks.

[HelgeGehring]

Just added a second benchmark which has mixed boundary conditions: https://helgegehring.github.io/femwell/benchmarks/mode_solver_rectangle.html

@blair-morrison @simbilod

[blair-morrison]

This is great to see. I think what would be really useful now is picking a particular geometry and then performing a convergence test with various mesh settings. This would allow for a better comparison between slepc and scipy in terms of speed for a given calculation, and also provide some guidance for users of the tradeoff of speed vs accuracy for a representative structure.

[HelgeGehring]

Yeah, there totally needs to be more guidance on how to chose the mesh!

I've implemented an automatic mesh refinement in #56 as I think it's often not intuitive to chose the right grid. The example uses the first order element as it's easier to see the refinement, but it also works with second order, I'll add more examples

[HelgeGehring]

See https://helgegehring.github.io/femwell/photonics/examples/refinement.html