search

facebookresearch / fmmax

Fourier modal method with Jax
MIT License
101 stars 11 forks source link

Data import/export functionality #77

Closed smartalecH closed 10 months ago

smartalecH commented 10 months ago

It would be great to have robust and standardized tooling for data import/export.

For example, it would be useful to be able to cache to disk LayerSolveResults scattering matrices, and amplitudes corresponding to a particular basis.

Pickle files aren't very robust (or secure). Perhaps Hdf5?

mfschubert commented 10 months ago

I have been using json for this:

https://github.com/invrs-io/totypes/blob/main/src/totypes/json_utils.py

smartalecH commented 10 months ago

Oh interesting... how big do these get? json isn't compressed or serialized in anyway.

mfschubert commented 10 months ago

They are about 1.4x the size of the object in memory. (I actually test for it: https://github.com/invrs-io/totypes/blob/main/tests/test_json_utils.py#L93)

Not a huge overhead, and the benefit is the file can be almost visually inspected.