Error controlling rotation from InsteaDMatic

[thomasaarholt]

This afternoon I attempted to set up insteaDMatic to acquire a rotation series in DM, using instamatic to control the tilt. InsteaDMatic on its own works well, but I'm having trouble with the instamatic.temserver_fei communciation. I get the following error when I click "Start" in DM.

To clarify: GMS is installed on the FEI Titan PC, which also has instamatic on it.

Error (The exception occurs when I click "Start" in GMS):

(py3) C:\Users\supervisor\Documents\instamatic-master>instamatic.temserver_fei
Config directory: C:\Users\supervisor\AppData\Roaming\instamatic\config
Microscope: fei (server)
Starting TEM server (localhost:8088 on pid=11224)
Please select the type of sample stage before moving on.
Press <ENTER> to continue...Connected to TEM server (localhost:8088)
Camera    : gatan (stream)

Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\supervisor\Anaconda3\envs\py3\lib\threading.py", line 917, in _
bootstrap_inner
    self.run()
  File "C:\Users\supervisor\Anaconda3\envs\py3\lib\threading.py", line 865, in r
un
    self._target(*self._args, **self._kwargs)
  File "c:\users\supervisor\documents\instamatic-master\instamatic\server\tem_se
rver.py", line 87, in handle
    data = pickle.loads(data)
_pickle.UnpicklingError: unpickling stack underflow

I'm currently investigating, I believe it has to do with the buffer size in tem_server.py It didn't.

[thomasaarholt]

I seem to also get the following error after pressing Enter after the question to do so. I think I may not have been pressing Enter earlier, so this may be the main error:

(py3) C:\Users\supervisor>instamatic.temserver_fei
Config directory: C:\Users\supervisor\AppData\Roaming\instamatic\config
Microscope: fei (server)
Starting TEM server (localhost:8088 on pid=10244)
Please select the type of sample stage before moving on.
Press <ENTER> to continue...Connected to TEM server (localhost:8088)
Camera    : gatan (stream)

Mode: LM
High tension: 300 kV
Current density: 0.00 pA/cm2
GunShift(x=-0.00474000000000002, y=-0.02772)
GunTilt(x=0.00208000000000058, y=0.0361651581014722)
BeamShift(x=0.0016840273190014494, y=0.008138850352565362)
BeamTilt(x=0.14769908569359824, y=0.16022082212813415)
ImageShift1(x=0.0, y=0.0)
ImageShift2(x=0, y=0)
DiffShift(x=-0.3243322135387545, y=0.033904784539488086)
Stage(x=-0.1, y=0.2, z=-0.1, a=11.4, b=0.0)
Magnification(value=7200, index=20)
DiffFocus(value=0.0)
Brightness(value=6.01274709277361)
SpotSize(3)
Saved alignments: ()

Traceback (most recent call last):
  File "C:\Users\supervisor\Anaconda3\envs\py3\Scripts\instamatic.temserver_fei-
script.py", line 11, in <module>
    load_entry_point('instamatic', 'console_scripts', 'instamatic.temserver_fei'
)()
  File "c:\users\supervisor\documents\instamatic-master\instamatic\server\TEMSer
ver_FEI.py", line 59, in main
    s.bind((HOST, PORT))
OSError: [WinError 10049] The requested address is not valid in its context
SUCCESS: The process with PID 10744 (child process of PID 10244) has been termin
ated.
SUCCESS: The process with PID 10244 (child process of PID 8968) has been termina
ted.

[thomasaarholt]

Right, I have resolved the above by setting the fei_server_host variable in the global.yaml config to localhost. Now I'm getting a (slightly) more sensible error which looks a lot like the original error with the exception of the last bit:

(py3) C:\Users\supervisor\Documents\instamatic-master>instamatic.temserver_fei
Config directory: C:\Users\supervisor\AppData\Roaming\instamatic\config
Microscope: fei (server)
Starting TEM server (localhost:8088 on pid=11224)
Please select the type of sample stage before moving on.
Press <ENTER> to continue...Connected to TEM server (localhost:8088)
Camera    : gatan (stream)

Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\supervisor\Anaconda3\envs\py3\lib\threading.py", line 917, in _
bootstrap_inner
    self.run()
  File "C:\Users\supervisor\Anaconda3\envs\py3\lib\threading.py", line 865, in r
un
    self._target(*self._args, **self._kwargs)
  File "c:\users\supervisor\documents\instamatic-master\instamatic\server\tem_se
rver.py", line 87, in handle
    data = pickle.loads(data)
_pickle.UnpicklingError: invalid load key, '4'

[thomasaarholt]

It is proving hard to debug, since any print statements I include aren't shown, which I think is due to there being multiple threads? If you have any suggestions, I am all ears.

[asdfdsa]

I am not sure, but is it the issue of port? Since you are running Instamatic on the same computer as insteaDMatic, it makes sense to change the fei_server_host to localhost, but then in InsteaDMatic you also need to make sure that it is trying to connect to the correct HOST and PORT.

Another thing that seems strange to me is that, I see that you are running instamatic.temserver_fei, but why does the output of the error message just look as if you are running instamtic.temserver?

[asdfdsa]

I mean this line in InsteaDMatic: https://github.com/stefsmeets/InsteaDMatic/blob/f0deed4ea81ee22b7da26739e4ba09dfdec4bedc/insteadmatic.s#L97

[thomasaarholt]

I changed that line to match the instamatic "tem server", which comes out as localhost 8088 for me (fourth line in log in the top post).

What you're saying about instamatic.temserver_fei vs instamatic.temserver is very strange and a very good point. If you compare the output of the last two logs I pasted here, one refers to the former and the second to the latter. I might have done some silly mumbo jumbo. Will take a fresh look tomorrow.

Edit: That could explain the lack of print messages too.

[stefsmeets]

Make sure you set use_tem_server: False in global.yaml, because you are running DM on the microscope computer. On our setup in Stockholm we had a separate computer for DM, and temserver_fei was used as a gateway to connect to temserver running on the TEM computer (just to control the rotation). In your case running temserver is not necessary, I think.

[stefsmeets]

Edit: That could explain the lack of print messages too.

Yes, that's the problem with the servers/threading, but necessary for process isolation. If you set use_tem_server: False / use_cam_server: False, and then run instamatic.controller, you will open an IPython environment with 'direct access' to all variables, which I find useful for debugging.

What is happening in your case is that when starting temserver_fei it tries to connect to temserver, and notices it can't connect, so it starts it in a subprocess. In this situation, I have redirected all output to /dev/null to avoid spamming stdout. You can always start instamatic.temserver in a separate console if you want to see the print statements, it will have the same behaviour, but the output is not hidden. I always do this, because I like to see the output and it allows me to quickly restart instamatic without re-establishing the TEM connection (which can be finicky, not to mention slow).

[asdfdsa]

I thought that using instamatic server could also be useful since probably you want the access of rotation speed, or is there already rotation speed control from DM?

[stefsmeets]

In principle yes, but temserver communicates using pickle to serialize, whereas temserver_fei does not, and only takes a very simple command to start rotation with a specific speed. This is needed to have a simple access from DM via netcat. Maybe temserver_fei was a poor choice for a name... :-)

temserver_fei itself opens a connection to the microscope (it initializes a TEMController object), but with use_tem_server: True, it will try to connect to it via a socket (which is a little bit pointless if it's running on the TEM PC).

[thomasaarholt]

Morning folks. Right, I'm making progress. Turning off use_tem_server helped, and now I'm no longer getting the pickle error. It did reveal an error that I fixed with #25.

Having added #25, I get a communication error (InsteaDMatic Controlling Tilt 3.txt ) in instamatic. I thought to try and start both use_tem_server and fei_server (I can't see a way to turn the latter off in the config, but that's ok), and make sure that the DM script was pointing at the correct port, since the stage wasn't rotating.

Now, when I press start, the stage correctly rotates, but the DM script seems to finish too early. Once the stage reaches the final rotation value, instamatic raises the following error: InsteaDMatic Controlling Tilt 4.txt

I'm wondering if part of the issue is that the rotation speed isn't set correctly, and that the script is checking the rotation angle too often, so that angle at t1 = angle at t2, and so it thinks that the rotation has stopped. I noticed that there are some settings for calibrated_rotation_axis in the DM script. Since we're on a Titan I went with the themisZ setting of -171, but maybe that isn't right?

[stefsmeets]

OK, there is 1 part that I can answer quickly:

Since we're on a Titan I went with the themisZ setting of -171, but maybe that isn't right?

This needs to be calibrated for your particular microscope, but it is not essential, because it can be very easily found afterwards. We use our own scripts, edtools.find_rotation_axis works in conjunction with XDS and can be found here: https://github.com/stefsmeets/edtools, but REDp or PETS can also do it. The value can be added to the script to write it to the input file, so it's not necessary to update the value every time.

[stefsmeets]

I'm wondering if part of the issue is that the rotation speed isn't set correctly, and that the script is checking the rotation angle too often, so that angle at t1 = angle at t2, and so it thinks that the rotation has stopped.

Yes, a common issue, also on our side, particularly when the TEM caches the stage position (very annoying!). For this, you can update the check_angle_interval to a little bit larger number: https://github.com/stefsmeets/InsteaDMatic/blob/b9585825697d81a4b77e2977d68dc2031e5536a2/insteadmatic.s#L104

Probably it needs to be tuned a little bit, i.e., as low as possible, because you want the experiment to stop when rotation is finished, but long enough so that the value updates. On our Themis, we found 0.3 s to work well.

[stefsmeets]

Now, when I press start, the stage correctly rotates, but the DM script seems to finish too early. Once the stage reaches the final rotation value, instamatic raises the following error: InsteaDMatic Controlling Tilt 4.txt

This error is a bit odd, but let me know if the above works to fix this. It seems like the socket is already closed, maybe because it thinks rotation 'finishes' too fast.

BTW, thanks for all the effort in reporting bugs here! I only have limited access to TEMs other than our own (both JEOL), so I am unable to test everything. I'm happy to see that we seem to be getting closer to getting everything to work :-)

[thomasaarholt]

As long as you don't mind, I'll happily keep reporting things I find! In general I try to let people know if I find bugs, but sometimes I feel like I'm just pestering! Glad to see it's appreciated!