add numpy interface for arrayTrace functions

[rhambach]

Dear Indranil Sinharoy,

first, thanks a lot for this great project! I have used the arrayTrace() functionality for my work. Recently, I have added some functions that use numpy arrays as arguments and pass them directly to the DLL. This reduces the overhead of the python wrapper considerably. Minor bugs have been fixed too. Maybe you find these additions useful.

Best regards, Ralf Hambach.

---

Details:

• Idea: During profiling, most of the time used for ArrayTrace is spent in the python-wrapper, filling the RAYDATAARRAY. Instead of executing the loop in Python, I added functions, that pass numpy arrays to the DLL and perform the loop in C.

• Performance: I have tested the performance of ArrayTrace for the CookTriplet Sample file. The numpy version is considerably faster, but still a factor 5 slower than a GeometricImageSimulation in Zemax. Checking also the time that is spent in the PostArrayTraceMessage shows, that the Python overhead gets negligible (for many rays).

  rays	1 Mio	10 Mio
  pyzdde	6.4 s	50 s
  pyzdde+numpy	1.1 s	9 s
  PostArrayTraceMessage	-	~8 s
  Zemax	<1 s	2 s

• Notes: So far I only tested the code on a single machine, so maybe some minor changes are necessary.

[indranilsinharoy]

Dear Ralf Hambach,

This is too good! Thank you very much. I have quickly taken a look at your additions, and all of it look really nice. Over the next few days I will try to understand the additions/changes in more details and merge them into the main branch. (I am a little short on time at the moment as I am trying to finish my thesis).

I am very sure that so many people will really love this.

Thank you very much once again.

Best regards, Indranil.

[indranilsinharoy]

Dear Ralf Hambach,

I am extremely sorry that I haven't gotten around to merge your excellent pull request yet. I have been keeping really very busy. I assure you that your pull request is not out of my mind. I will get to it ASAP. I hope you will understand.

Thank you, Best regards, Indranil.

[rhambach]

Thanks for your notice. There is really no problem. All the best, Ralf.

[indranilsinharoy]

I need to do some extra work to merge the new code (which are great, by the way) seamlessly into the existing PyZDDE code base without breaking old code. I am working on it locally, but will take some time (as I am currently very busy with my thesis). [This is mostly a reminder to myself that I have to do this work when I get some bandwidth]

Things that needs attention (I will add more as I understand more):

1. The module arraytrace has been removed (more specifically, the file arraytrace.py has been removed under the directory pyzdde). Instead, there are two new mechanisms for array based ray tracing -- either using the module arraytrace.raystruct_interface or using arraytrace.numpy_interface. The module arraytrace.raystruct_interface essentially contains all the functions from arraytrace.py. As a consequence, old code trying to do import pyzdde.arraytrace as at will fail. One solution could be to reintroduce arraytrace.py under pyzdde. This will just act as a wrapper module that will either automatically (if Numpy presence is detected) or otherwise dispatch calls to either arraytrace.raystruct_interface or arraytrace.numpy_interface

[bauman3]

Sorry for my newbie-ishness, but does this mean that if I installed PyZDDE within the last couple weeks (which I have) that arraytrace will not work? I'm guessing that this is true because my kernel freezes up when I try to call zArrayTrace. I have to admit that I don't understand the workarounds.

[indranilsinharoy]

@bauman3 I haven't not merged the changes from @rhambach to the main branch yet (as I explained above ... it will take a little more work). So, the problem that you seeing is certainly not due to the numpy interface. I have observed that some times (I have faced this about 3-4 times) the array tracing module just doesn't respond (I am really not sure why this happens). But, restarting my computer resolved the problem each time. I have written about it here #63 . Please restart and try.

[rhambach]

Dear Indranil Sinharoy,

after switching to python 3, I understood your problems with the proposed code in this pull request. I updated the code to run on my Windows machine with python 2.7, and 3.4. I also tested the DDE link with Zemax 13 and ZOS 16.5. Attached to this message, you will find the compiled code for 64bit systems, in case the compilation makes problems: pyzdde/arraytrace/x64/Release/ArrayTrace.dll (download: x64.zip).

I hope this makes it easier to incorporate the changes... Best regards, Ralf.

PS. The conflicts arise as I also had the problem with the regexp's for float numbers being too tight in zGetPOP(). I solved this issue sligtly different by using a general regexp proposed by http://docs.python.org/2/library/re.html#simulating-scanf. In principle this might be done similarly in other file input functions, too.

---

Summary what works and what does not work on my machine:

Works

• runs with python 2.7 and python 3.4
• backward compatible:
  • import pyzdde.arraytrace as at still works and loads the old interface.
  • The new interface is only obtained by loading import pyzdde.arraytrace.numpy_interface as nt explicitly. I would not propose to make the choice automatic (if numpy is present), as the numpy interface uses slightly different parameters to use numpy arrays efficiently (e.g. pos as 3xN array instead of separate x,y,z arrays)

Does not work

• Some of the unittests still fail. This is probably due to different handling of single and array raytrace by Zemax and should be handled in the pyzdde interface to avoid confunsion.

failed unittest	Zemax 13	ZOS 16.5	comment
pyZDDEunittest.py		test_zModifySettings	self.ln.zModifySettings(sfilename,'UN1_OPERAND', 'ZERN') returns -2 (incorrect version number)
arrayTraceTest.py	test_zGetTraceArrayOPD	test_zGetTraceArrayOPD	Test fails for real raytrace. The DDE arrayTrace function from Zemax does not return surface normal if OPD is requested.
arrayTraceTest.py		test_zGetTraceDirectNumpy test_zGetTraceDirectRaystruct	ZOS 16.5 handles error and vigcodes different using either single raytrace or arraytrace.

• The arraytrace (both the old interface and the numpy interface) get stuck using python 3.6 (timeout error). Deleting all cached files does not help for me.

[indranilsinharoy]

Hi @rhambach ,

Thank you very much for all your effort. I promise to dedicate some time this weekend to try to understand and integrate your pull request. I'll most likely need your help; and I'm sorry that I wasn't able to merge your earlier pull-request. Thank you so much for your time and second pull-request.

Best regards, Indranil.