search

henrypinkard / Pygellan

[DEPREACATED] Python interface for data-driven microscopy with Micro-manager/Micro-Magellan
BSD 3-Clause "New" or "Revised" License
14 stars 3 forks source link

Accessing micromanager plugins from pygellan #25

Open henrypinkard opened 4 years ago

henrypinkard commented 4 years ago

This would require:

@nicost @bryantchhun

henrypinkard commented 4 years ago

PR for moving ZMQServer from Magellan to Micro-manager: https://github.com/nicost/micro-manager/pull/89

henrypinkard commented 4 years ago

@bryantChhun preliminary version of this ready if youre interested to test

https://github.com/nicost/micro-manager/pull/90

bryantChhun commented 4 years ago

@henrypinkard I'll try to look at this this week. I currently don't have a windows build environment for mm2-gamma available so it might take a some work to get there. I assume I could compile specific jars and copy those to our scope with a recent nightly build?

nicost commented 4 years ago

@bryantChhun I'll build a MMJ.jar from this code. All you will need to do is copy the MMJ.jar to a (recent) installation of 2.0-gamma (into the plugins/micro-manager folder), and this code should run. I'll let you know when I have it ready.

henrypinkard commented 4 years ago

Do either of you know a use case for the fourth bullet for testing? Something where you get a java object (i.e. not a primitive or tagged image), then you want to pass that object as an argument to another function

bryantChhun commented 4 years ago

What about putting a datastore into a display?

henrypinkard commented 4 years ago

Yeah sounds good. I'm not familiar with that API. Would you mind posting an example so I can test?

nicost commented 4 years ago

The micro-manager web-site has some examples on the new api, that still are reasonably close to how things work: https://micro-manager.org/wiki/Version_2.0_API_How_do_I I found that in the current zmq-bridge/Pygellan the more deeply nested classes such as Datastore and DataViewer are not found. I was going to make an example, but currently (latest git version of zmq-server and Pygellan), things do not work on my laptop (editing this response, since I realize that there is a left-over Magellan plugin, after removing that I get the following exception:)

                                [       ] java.lang.RuntimeException: java.io.FileNotFoundException: \Program Files\Micro-Manager-2.0gamma\file (The system cannot find the file specified) in Thread[ZMQ Server master,6,main]
                                [       ]   at org.micromanager.internal.zmq.ZMQServer.initializeAPIClasses(ZMQServer.java:200)
                                [       ]   at org.micromanager.internal.zmq.ZMQServer.parseAndExecuteCommand(ZMQServer.java:92)
                                [       ]   at org.micromanager.internal.zmq.ZMQServer.lambda$initialize$3(ZMQServer.java:58)
                                [       ]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                                [       ]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                                [       ]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                                [       ]   at java.lang.Thread.run(Thread.java:748)
                                [       ] Caused by: java.io.FileNotFoundException: \Program Files\Micro-Manager-2.0gamma\file (The system cannot find the file specified)
                                [       ]   at java.util.zip.ZipFile.open(Native Method)
                                [       ]   at java.util.zip.ZipFile.<init>(ZipFile.java:219)
                                [       ]   at java.util.zip.ZipFile.<init>(ZipFile.java:149)
                                [       ]   at java.util.jar.JarFile.<init>(JarFile.java:166)
                                [       ]   at java.util.jar.JarFile.<init>(JarFile.java:103)
                                [       ]   at org.micromanager.internal.zmq.ZMQServer.initializeAPIClasses(ZMQServer.java:184)
                                [       ]   at org.micromanager.internal.zmq.ZMQServer.parseAndExecuteCommand(ZMQServer.java:92)
                                [       ]   at org.micromanager.internal.zmq.ZMQServer.lambda$initialize$3(ZMQServer.java:58)
                                [       ]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                                [       ]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                                [       ]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                                [       ]   at java.lang.Thread.run(Thread.java:748)
nicost commented 4 years ago

The above problem was caused by splitting the jar file name on ":" (which is part of the Windows path name, so results in the wrong path). Fixed in PR #2.

nicost commented 4 years ago

Interesting, the following is all working for me now. Not sure what the difference is with the system on my Mac, but I do think that the class detection could be made more robust.

The below is an example of the use of the 2.0 api. One question is whether it makes sense to provide convenience translation from the MM Image object to Python/numpy structures. Creating an example getting live Images from MM and sticking them in Napari would be awesome.

from pygellan.acquire import PygellanBridge
bridge=PygellanBridge()
mm=bridge.get_studio()
mmc=bridge.get_core()
mm.live().snap(True)
Out[7]: [JavaObjectShadow for : org.micromanager.data.internal.DefaultImage]
dv=mm.displays().get_active_data_viewer()
dp=dv.get_data_provider()
img=dp.get_any_image()
img.get_height()
img.get_metadata()
md=img.get_metadata()
md.get_camera()
Out[14]: 'Camera'
henrypinkard commented 4 years ago

I finished adding the functionality to send shadowed Java objects back across the bridge as arguments in functions, so I think that should be all the changes needed. Also cleaned up the class detection system so its at least easier to debug should future problems crop up.

@bryantChhun this should be everything needed for your use case. I added this example showing how create an object of a class within your plugin. Let me know if you run into any problems.

@nicost Regarding the MM Image object: I think it is a better use of future effort to focus on developing new and simpler Python APIs for control and more powerful acquisition features (i.e. new acquisition engine) than to make existing APIs designed for Java more convenient, and I plan to allocate my effort as such. That said, if you want to dig into this and send a PR I'd be happy to add it.

bryantChhun commented 4 years ago

Nico, when you get a chance can you compile the MMJ_.jar and find a way to send it to me? Google drive? (gmail will probably filter it)

nicost commented 4 years ago

@bryantChhun I sent a separate email with the location of the jar.

@henrypinkard I am not really in a position to do any actual work on the Python side, since that is all new to me and certainly would lead to code that end up being inconvenient. However, an important point of this project is enabling data processing in Python of Micro-Manager data and to then stop at the last step (translating the datastructure in Java that actually holds the Image data), seems a pity. @bryantChhun do you want to have a look at this?

bryantChhun commented 4 years ago

No problem. Thanks for making the .jar (i got the email). Will try to look at this today.

henrypinkard commented 4 years ago

You won't be able to do this with just changes on the Python side. It requires changes Java too. Essentially you just need to write code that serializes the relevant parts of the object, and then code on the other side that deserializes it into an appropriate python object. I've already done this for TaggedImage, so you can just use that as an example. This is however beyond the scope of my purpose for creating this project, which was to enable experiments that are not possible with the current micro-manager API

bryantChhun commented 4 years ago

Ok a few issues:

Micromanager appears to remember the last "run server on port 4827" state. If you close mm with the box checked, it will stay checked the next startup. This poses problems as such:

When i am connected, I can not create a Studio and receive the following error:

>>> bridge = PygellanBridge()
>>> mm = bridge.get_studio()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bryant.chhun/PycharmProjects/Pygellan/pygellan/acquire.py", line 103, in get_studio
    return self.construct_java_object('org.micromanager.Studio', new_socket=True)
  File "/Users/bryant.chhun/PycharmProjects/Pygellan/pygellan/acquire.py", line 67, in construct_java_object
    raise Exception(response['value'])
Exception: Couldnt find class with nameorg.micromanager.Studio
>>>

setup My environment has pygellan installed by pip install -e . (from your most recent master clone) I deleted the magellan.jar from the micro-manager install replaced the old MMJ_.jar with the new one

nicost commented 4 years ago

Not sure I understand. The "Image" object already makes its way to Pygellan, and Pygellan can access methods that give pixel-type, height, width, as well as give access to the pixels themselves, so it seems that all that is needed is some kind of convenience function to map this into a numpy array (as well as a way to bring a numpy array back into an Image object). Come to think about it, why is the special treatment needed on the Java side for TaggedImage? I guess that it is mainly to avoid one more step on the Python side? I understand your reluctance having anything to do with the 2.0 api, but there is a lot of nice stuff there that is useful to many.

bryantChhun commented 4 years ago

Not sure I understand. The "Image" object already makes its way to Pygellan, and Pygellan can access methods that give pixel-type, height, width, as well as give access to the pixels themselves, so it seems that all that is needed is some kind of convenience function to map this into a numpy array (as well as a way to bring a numpy array back into an Image object). Come to think about it, why is the special treatment needed on the Java side for TaggedImage? I guess that it is mainly to avoid one more step on the Python side? I understand your reluctance having anything to do with the 2.0 api, but there is a lot of nice stuff there that is useful to many.

I think those "image" object properties are returning primitives, which are mapped to python well. But what I will test (when I can get it running) is whether the dataprovider.getImage() will return a byte array or buffer to the python process. Another thing i'll test is the condition 4 as mentioned above.

henrypinkard commented 4 years ago

@bryantChhun I don't think restarts are actually needed. The server is running regardless on the java side (once the box is checked) and can connect to any amount of clients. You shouldnt need to restart mm.

I think that error is something to do with classpath loading. I wonder if it is particular to MMJ. Are you able to run bridge.get_core()?

henrypinkard commented 4 years ago

Yes, exactly @bryantChhun . Regarding TaggedImage, it gets special treatment because the Java Shadow classes dont send over fields (only function names). This might be possible to change (the shadow classes were less well developed when I made this), but for now it is explicit traslated

bryantChhun commented 4 years ago

@henrypinkard what I mean by needing to restart is: if you, for any reason, toggle the tools->option box on/off, it will permanently disable pygellan connections regardless of button state until you restart micro-manager. It is probably some GUI button state issue that isn't well linked to the existence of a zmq.socket.

bridge.get_core() returns a slightly different problem:

>>> mmc = bridge.get_core()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bryant.chhun/PycharmProjects/Pygellan/pygellan/acquire.py", line 96, in get_core
    return self.construct_java_object('mmcorej.CMMCore', new_socket=True)
  File "/Users/bryant.chhun/PycharmProjects/Pygellan/pygellan/acquire.py", line 67, in construct_java_object
    raise Exception(response['value'])
Exception: Couldnt find class with namemmcorej.CMMCore
>>>
henrypinkard commented 4 years ago

I see. That's more of a bug with the button itself then.

Okay, that means you are having a problem with the classpath loading code. I'll need to add some additional logging

nicost commented 4 years ago

The code stopping and starting the zmq server is in MMStudio:

public void runZMQServer() {
      if (zmqServer_ == null) {
         zmqServer_ = new ZMQServer(studio_);
      }
      zmqServer_.initialize(ZMQServer.DEFAULT_PORT_NUMBER);
      logs().logMessage("Initialized ZMQ Server on port: " + ZMQServer.DEFAULT_PORT_NUMBER);
   }

   public void pauseZMQServer() {
      if (zmqServer_ != null) {
         zmqServer_.close();
         logs().logMessage("Paused ZMQ Server");
      }
   }

Suggestions on improving this are welcome.

bryantChhun commented 4 years ago

I'm running on Mac btw. Also @nicost this is possibly known and documented somewhere, but i am generally unable to launch mm2-gamma from macOSX without first replacing the ImageJ app with an older one from 2015. ImageJ will always launch, but only the older copy will subsequently launch micro-manager.

nicost commented 4 years ago

@bryantChhun Best to move the MacOSX one to micro-manager mailing list or github issue.

I am able to reproduce the pygellan bridge issue: running the same code under Netbeans everyhting works fine, but when running it stand-alone (which I had not done yet before), the bridge opens, but appears to have no classes. This will be rough to debug, since you can not use a debugger;) Doubtlessly a class-loader and.or Java permissions issue, I'll see where I can get.

nicost commented 4 years ago

OK, this has me stumped for now. The code gets its list of packages using a system call (Package.getPackages()) that uses a protected method to get the classloader of the calling class (ClassLoader.getClassLoader(Reflection.getCallerClass()); ),

Our code tries to get hold of the same classloader using: Thread.currentThread().getContextClassLoader(); and then finds the resources that this classloader knows about the class names (paths) that we had discovered from the Packages. Remarkably, this all works great in Netbeans, but when running it by itself, the classloader finds no resources for any of the classes.
I do not understand how this is possible if the two class loaders are indeed the same, but it also looks like the two classloaders are identical, hence, I am at a loss for now. Any insights appreciated. I pushed logging code to my zmq branch.

nicost commented 4 years ago

Classloader issue indeed. I fixed it by using the ImageJ classloader. Will do a PR (or @henrypinkard you can merge) and put a new MMJ_.jar up soon.

henrypinkard commented 4 years ago

Nice! I merged that in

Also fixed the bug with toggling the zmqserver option on and off

bryantChhun commented 4 years ago

when I run nico's lines above, they all work now! but I'm trying to take it one step further:

from pygellan.acquire import PygellanBridge
bridge=PygellanBridge()
mm=bridge.get_studio()
mmc=bridge.get_core()
mm.live().snap(True)
Out[7]: [JavaObjectShadow for : org.micromanager.data.internal.DefaultImage]
dv=mm.displays().get_active_data_viewer()
dp=dv.get_data_provider()
img=dp.get_any_image()
img.get_height()
img.get_metadata()
md=img.get_metadata()
md.get_camera()

image = img.get_raw_pixels_copy()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <lambda>
  File "/Users/bryant.chhun/PycharmProjects/Pygellan/pygellan/acquire.py", line 240, in _translate_call
    return self._deserialize_return(reply)
  File "/Users/bryant.chhun/PycharmProjects/Pygellan/pygellan/acquire.py", line 253, in _deserialize_return
    raise Exception(json_return['value'])
Exception: Internal class accidentally exposed

I see this example here: https://github.com/henrypinkard/Pygellan/blob/master/examples/micromanager_core.py but that only allows one to get the most recent snap image (as acquired by mmc)

Question: is there a way to use the studio's machinery to retrieve one or several images by coordinate?

nicost commented 4 years ago

@bryantChhun I think the question is more how to get a complete Image object over to the python side, and I think that @henrypinkard suggested that the best way is to do a "translation" as he does for TaggedImage (which needs code both on the Java and Python side of things). I think that it should be pretty straightforward, but my lack of Python competence makes it hard to do. Most everything else should work (although we should test generation of Coords and thing like that and see what else is missing (if any).

henrypinkard commented 4 years ago

@bryantChhun that was a bug which was just that I'd forgotten to support short[] arrays along with the other types. Fixes in newest pygellan and micromanager.

On second thought after seeing that example, I'm not sure a translation is necessary, since the Image API provides functions for getting pixels and metadata. @nicost Do you think anything is needed beyond that?

In fact, it might make sense to remove the special treatment of TaggedImages and just allow lazy access to their tags and pix fields. That would make the bridge totally agnostic to classes beyond just primitives, which would be nice because it would be generic to whatever java is on the other side

nicost commented 4 years ago

Awesome!

I agree that it would be nice to keep the bridge agnostic to classes beyond primitives, seems cleaner.

It may be nice to provide convenience Python functions to "translate" TaggedImage and Image into easily usable Python data structures, but those could be completely independent of Pygellan and be available as examples (or possibly as another file within the Pygellan package). Even though it sounds this should be easy at this point, I would certainly benefit!

henrypinkard commented 4 years ago

I just changed it to do that. Now the bridge only serializes primitives, arrays, and things with a direct correspondence in python (for now that Lists and JSONObjects).

I've done that with an example right now for the core:

tagged_image = core.get_tagged_image()

#pixels by default come out as a 1D array. We can reshape them into an image
pixels = np.reshape(tagged_image.pix, newshape=[tagged_image.tags['Height'], tagged_image.tags['Width']])

My feeling is that this is straightforward enough that a similar example with Image would be sufficient. There may indeed be more convenient ways of translating larger datastructures (sets of images, for example), but I think this is for another layer that sits above the bridge since the bridge is API agnostic and these convenience conversions wouldn't be

nicost commented 4 years ago

Yes! Totally works:

pythonImage=np.reshape(img.get_raw_pixels(), newshape=[img.get_width(), img.get_height()])

and showing this in napari is then as straight forward as:

with napari.gui_qt():
   ...:    viewer = napari.view_image(nIm)

and now hoping that it is possible to dynamically/continuously feed images into napari.

For now, great demo! Let's kick the tires a bit more and see where things break. I guess that pushing data back into MM form Python is an interesting use-case. @bryantChhun the latest jar is at the usual place.

bryantChhun commented 4 years ago

@nicost @henrypinkard Great! Nice work guys. I'll be pretty busy today and tomorrow. Will try to get to this on the weekend or next week.

Also, streaming images to napari has its limits due to napari design. Simply adding images will usually trigger napari refresh, which has other downstream signaling effects (z-slicing for example). There are ways around this though -- all on the python side. For now I think the java side is in good shape, but will probably need some stress testing.

henrypinkard commented 4 years ago

Sounds good. Thanks to both of you for all your help with this!!

I'm excited about the potential for cool stuff that can sit above this "plumbing" layer, both as other modules in pygellan and other packages.

After we've decided that everything works well, I'll put together some additional documentation for the various ways one can make use of the bridge

nicost commented 4 years ago

I'll get @marktsuchida in the loop, and ask him whether it is OK to push this into micromanager/master. Two questions:

nicost commented 4 years ago

One more remark: The current setup does not work correctly on my Mac running MM under Netbeans. It does work, however, when running MM directly. I think this is due to classloader issues, that I don't think are worth going into at this points in time (but it is funny that thinsg work fine on Windows running MM under Netbeans).

henrypinkard commented 4 years ago

Yes I'm fine with that for first one

Actually, I think the name is relevant. There are two other main parts of this project that are specific to magellan--pygellan.data for reading magellan data, and other parts of pygellan.acquire which are for controlling magellan API. I plan to expand the latter part quite a bit, in ways that will be closely linked and will require modification/refactoring of the bridge (very rough outline is on the issues). This could all perhaps be folded into micro-manager main repo (a parallel API to the studio for smart/high throughput acquisition), but unless that happens I'd prefer to keep as is. However, in spite of its relevance, I do think Pygellan is a kind of a stupid name :)

And that third point is weird. Latest code works for both studio and core on my mac with netbeans. Though I did notice some classloader weirdness for certain Jars last night (The openCL one I think?)

nicost commented 4 years ago

OK, let's test the code a bit longer and move it into the master branch in a week or two.

Let's forget about the third point for now. Hopefully no one else will run into it, and we more or less know this is a classloader problem.

bryantChhun commented 4 years ago

I might make the counter argument that, because pygellan has components specific to megellan, we should separate those components from the “zmq server” components.

So that instead, this internalization of zmq server interacts with a different pypi package like “mmzmq”. And that pygellan uses “mmzmq” to talk with micromanager.

On Thu, Feb 13, 2020 at 11:47 AM Henry Pinkard notifications@github.com wrote:

Yes I'm fine with that for first one

Actually, I think the name is relevant. There are two other main parts of this project that are specific to magellan--pygellan.data for reading magellan data, and other parts of pygellan.acquire which are for controlling magellan API. I plan to expand the latter part quite a bit, in ways that will be closely linked and will require modification/refactoring of the bridge (very rough outline is on the issues). This could all perhaps be folded into micro-manager main repo (a parallel API to the studio for smart/high throughput acquisition), but unless that happens I'd prefer to keep as is. However, in spite of its relevance, I do think Pygellan is a kind of a stupid name :)

And that third point is weird. Latest code works for both studio and core on my mac with netbeans. Though I did notice some classloader weirdness for certain Jars last night (The openCL one I think?)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/henrypinkard/Pygellan/issues/25?email_source=notifications&email_token=AGDVUYEBSOYSWYZ5XHEPTCTRCWPVVA5CNFSM4KKPKJ5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELWLYGA#issuecomment-585939992, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGDVUYHDBUZVWFLQIFQEZS3RCWPVVANCNFSM4KKPKJ5A .

henrypinkard commented 4 years ago

I agree with you that ultimately the most sensible thing might be to split these out into plugin-agnostic and plugin-integrated modules (Or maybe even a faster version of py4j), but I don't think this is yet ready to make that determination for two reasons. First, the ZMQ parts of this are in my mind still a work in progress. In order to enable the functionality described in the issues, I will need to add different types of socket architectures (i.e. not Java server python client). These may be fairly tightly coupled to the magellan acquisition engine. Hard to say for me right now which layer they would belong in, because there is still a lot to figure out. I'd rather not duplicate effort by splitting things now, and then have to do so again in a different way. Second, it is not yet clear to me whether Magellan (or components of Magellan, specifically the acquisition engine and file format) will remain as a plugin or be merged into micro-manager. Since this is all very much a work in progress, I'd like to defer these decisions until I have a clearer idea of how all the pieces will work.

henrypinkard commented 4 years ago

@nicost ready to merge into master?

bryantChhun commented 4 years ago

@nicost @henrypinkard Sorry it's been a while. I think this issues thread is quite a bit larger than it should be and should be branched out -- so the topic I bring above maybe should go in a new issue. Going to make that now.