asdf-format / asdf

ASDF (Advanced Scientific Data Format) is a next generation interchange format for scientific data
http://asdf.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
523 stars 57 forks source link

Consider a new design for the info and search methods that avoids conversion of nodes when the lazy_tree option is used #1795

Open perrygreenfield opened 2 months ago

perrygreenfield commented 2 months ago

The new lazy_tree PR #1733 now provides the ability to delay conversion of nodes until they are needed as well as giving finer control of memory use. Unfortunately, any use of the info or search methods forces conversion of all nodes making them costly to use for large data sets. Since much (perhaps all) of the information for these methods is in the raw yaml, it may be possible for a different implementation to avoid dependencies on conversion to show the needed information. When resources permit, this option should be investigated.