search

facebookresearch / fmmax

Fourier modal method with Jax
MIT License
95 stars 9 forks source link

Data import/export functionality #77

Closed smartalecH closed 8 months ago

smartalecH commented 8 months ago

It would be great to have robust and standardized tooling for data import/export.

For example, it would be useful to be able to cache to disk LayerSolveResults scattering matrices, and amplitudes corresponding to a particular basis.

Pickle files aren't very robust (or secure). Perhaps Hdf5?

mfschubert commented 8 months ago

I have been using json for this:

https://github.com/invrs-io/totypes/blob/main/src/totypes/json_utils.py

smartalecH commented 8 months ago

Oh interesting... how big do these get? json isn't compressed or serialized in anyway.

mfschubert commented 8 months ago

They are about 1.4x the size of the object in memory. (I actually test for it: https://github.com/invrs-io/totypes/blob/main/tests/test_json_utils.py#L93)

Not a huge overhead, and the benefit is the file can be almost visually inspected.