New load-on-demand library implementation for optical components and materials

[rdgraham]

I was getting tired of the long time that it takes to import pyoptools and the memory used -- mostly due to loading the whole material and component libraries on import. I have prepared this PR to help with that. It turned into a bit of a project. I have a new clean implementation of the optical component library and materials library.

Please see the list of changes below and docstrings for new usage details.

Changes:

• Optical component access through library.py is now on-demand rather than loading all on import, and does not require loading the whole optical library into memory. To achieve this I transitioned the library files to .json, which allows the use of ijson library, which is a fast iterative 'SAX-style' parser. (Incidentally, support for nested structures should also be helpful for more naturally supporting asphere definitions in the future).

• I made a fresh import of the latest zemax catalogs from Thorlabs and Edmund. The importer has been tidied up, made to output .json and make a .csv listing all failed/skipped imports. I did notice that there didn't seem to be code there for the cylindrical lens importing, so I implemented that. And also there were some numerical issues with importing ball lenses which I fixed.

• Verified all optics load in a new unittest. Now have over 9000 optics available.

• Library access is now available with dictionary-style lookup directly from the library module. In addition to attribute access via the manufacturer name as before.

• The only API change as far as I can see is that user specified libraries are not loaded automatically from the users home dir. But .json libraries can can be explicitly added with library.add() method. I don't think the original design was good because it allowed for different behavior of the library depending on installation and files outside the package directory.

• Materials are now loaded on-demand, with caching supported. Dictionary style lookup is now supported similar to the optical component libraries.

• I merged in the latest database from refractiveindex.info, including the organic compounds. So now those optical plastics are available. Previously, the generic compounds were not accessible directly. Now they are available through .organic and .inorganic attributes, with an optional syntax to specify the exact reference to load. This is all described in the docstrings.

• Migrated material aliases to a .json format and updated so as to support all the names used in the optical component libraries as of now. Components do now respect the glass manufacturer priority specified in the GCAT field.

• updated the main tutorial, added examples covering searching for optics and materials

• migrated from pkg_resources to the now recommended importlib_resources

By the way, I realize there are a lot of files in this commit on account of updating the libraries. The main things of interest though are the new library.py, new material.py, updated zemax_import and new generic optic factory in _comp_lib/optic_factory. Also the unittests under /tests. For review, it may be useful to restrict the diff to just python files with something like git difftool fastlib2 origin/master -- '***.py'

[ramezquitao]

@rdgraham thanks a lot for this contribution. This is something I was planning to do because the long start up times, but you beat me to it. I will check and accept the pull request during this week. Couple questions:

• The material library from https://refractiveindex.info do not have some generic materials like BK7 of F_SILICA that the Thorlabs library uses. This was fixed by creating an "aliases.cfg" file to be able to redirect them to the correct material in the library (in this case the N-BK7 or SiO2_Malitson materials). Is this still working?
• I think that managing the component libraries as one huge file is inconvenient when you need to add new components, so I was planning to split the component library files in to library folders, where each component has it own independent file. This means that to add a new component to the library would be just as easy as adding a new file, and not editing a huge one. Using your new approach, do you think this is still feasible?

Thanks again for your contribution

[rdgraham]

Hi,

• For the material aliases, yes these continue to work, verified for all the available optics. I migrated the aliases file to .json (pyoptools/raytrace/mat_lib/data/aliases.json) and there is support for pointing towards glasses as well as inorganic and organic compounds. The aliases are the first thing that it checks and take priority.
• No problem to add additional lens catalog files. The optics will automatically be available directly at the base level, or through attribute access based on the filename. The latter might be a tiny bit faster but is mostly for backwards compatibility.
• Currently it will not walk into sub-directories, just looks at all the .json files in pyoptools/raytrace/library/catalogs/, though I guess it would be possible to do that.
• Currently there are just two .json files, for Edmund and Thorlabs, corresponding to their original zemax files. As the library grows I would suggest keeping one file corresponding to each original source file since having each component being it its own file would be thousands of files. If data management becomes a problem, note that since it is .json there are various tools available that can be used to manage and merge multiple files. For example command line tool jq (haven't use that myself though).