Refactoring + ICE support + Ms pixel buffer support + screen/plate/well hierarchy

[Rylern]

Initial refactoring of the extension. I'm creating a pull request now to agree on some design decisions:

• I organized packages by feature, not by layer (see this link). For example, the OmeroExtension class installs 3 actions/menus:

  • An action sending annotations
  • A browser menu
  • An action to open a connection manager

  Each of this feature has one package dedicated to it, located in the same package as the OmeroExtension class. The common package is an exception, as it contains functions used by every other parts of the project. I'll see if it's possible to integrate some parts of it in separate feature packages. With the Javadoc comments, it should be easy to understand how the project is structured.

• I used FXML files each time it was possible, even for very short UI elements.
• Each time a function takes some time to complete (for example when making HTTP requests), I made the function asynchronous using the Java CompletableFuture. The functions are then executed in a random thread.
  • Since asynchronous functions are executed in a random thread, I tried to make all non-UI classes thread-safe, which means every public functions of theses classes should work no matter the thread used to call them. I used the synchronized keyword for that.
  • When an asynchronous function is complete and I want to update UI elements, I used Platform.runLater() to come back to the UI thread.
  • When a JavaFX property or an ObservableList is modified from a thread, the changes are propagated to any listening class on the same thread, which can be a problem if a UI class listens to a property/list changed from a non-UI thread (because UI elements can only be updated from the UI thread). To solve this problem, I created someUIElementModel classes that propagate changes made to a property/list from any thread to the UI thread.
• I changed the HTTP requests to use the new Java HTTP client. It natively uses CompletableFutures.
• I used this convention to name packages.
• The texts displayed on the UI are all in one string.properties file.
• Unless explicitly written in the documentation, all public functions never return null. When a function is not guaranteed to return a result, the Java Optional API is used.

A few other things:

• The ICE API was added to the project and is used if OMERO Java files are present in the classpath. I tested i with 8-bit, 16-bit images and it seems to be working. However, opening large files makes the viewer very slow, so I'm working on this right now.
• In the browser, a label indicates if raw pixel access is available.
• I didn't implement unit tests yet, so this version is not ready to be released.
• I also used the preferences API to store the server URLs so that the user doesn't have to write them each time the application is started.
• I will add support for screens/wells/plates.
• Every issues in the issue tab should have been addressed.
• Should we use Java modules?

[petebankhead]

Thank you for doing all this, and the fantastic explanation of everything! I've started to check out the code in more detail but it might take me a little while.

Quick answer to the questions:

Wouldn't it better to make QuPath ImageServers asynchronous? Opening an OMERO image requires to perform several HTTP requests, which for now make the application unresponsive during a few seconds.

Possibly...

I tentatively think it's better to leave that up to the calling code. One reason is that we want the API simple for people to use server.readRegion(...) in scripts (which generally run in a background thread anyway). And we definitely can't use any JavaFX features in ImageServer since it should be UI-independent. I feel like making the ImageServer asynchronous might add complexity and I'm not sure of the specific benefits.

At least, I think it makes sense to retain the current approach for now. But I'm strongly in favor of considering how to rethink the entire ImageServer concept (alongside considering ImgLib2, multidimensional image 'blocks' rather than 2D 'regions', and perhaps writing an entirely new viewer...). So please do record all ideas for improvements :)

I saw that OMERO was also providing screens/wells/plates, is that something this extension should support?

Ideally yes! Since I think these usually aren't RGB images, the previous extension couldn't support the images anyway - so the lack of this feature didn't matter. But it would be a welcome addition.

[Rylern]

I created a commit with the ICE support and updated the text above with the changes

[Rylern]

I updated the pull requests with the changes you requested. This version relies on the Bio-Formats pull request (this is why the GitHub checks failed).

Organization & scripting

I reorganized the package structure. There are now less packages, and every UI elements are in a gui package.

Do you have ideas of example scripts I should include?

Switching raw and web pixel access

I changed the way of accessing pixel values. I created the PixelAPI interface, which provides information (e.g. supported images) about a pixel API (Ice or web for now). A PixelAPI is created by a WebClient.

The user can choose an available pixel API in the browser before opening an image. Every OMERO server supports the web API, but not all support the Ice API (for example servers that don't require authentication like IDR).

The default pixel API is the one that provides access to raw pixel values (which is only Ice for now), or the web API if none is available. This means that if a user:

• Browses an OMERO server that supports both Ice and web.
• Opens an image with the web API, and saves it.
• Closes QuPath, and later opens the saved image without reselecting the web API.

then the image will be opened with the Ice API. Is that a problem?

If the user tries to open an image with an unavailable API, the image server won't be created (which will show an error dialog), so the bug you mentioned shouldn't appear anymore.

PixelAPI is used to retrieve information about a pixel API, but can't actually read pixel values. For that, I created the PixelAPIReader interface. I separated PixelAPI and PixelAPIReader because there is only one instance of PixelAPI per OMERO server, while there can be several instances of PixelAPIReader per OMERO server (if multiple images are opened at the same time for example).

The browser gives some info below the thumbnail. It would help to add more dimension info, as seen under the 'Image' tab in QuPath Dimensions (CZT) and also an estimate of the uncompressed size. Particularly for raw pixel access this helps determine how long it's likely to take to open the image.

The number of channels, z-slices, and timepoints are indicated, but I didn't find a way to get the uncompressed size from the web API.

[petebankhead]

Thanks, I'll read properly when I have more time - for now I'll just try to answer the questions.

I reorganized the package structure. There are now less packages, and every UI elements are in a gui package.

Sounds good!

Do you have ideas of example scripts I should include?

I guess the main one would be to open an image, access pixels or metadata, and close the image.

This may require login (ideally not requiring the password to be in the script).

Also, it should be possible to run a script over OMERO images from a command line, e.g. see https://qupath.readthedocs.io/en/0.4/docs/advanced/command_line.html#subcommands

The following is a rough example - even just printing the image data should demonstrate that the image can be opened.

QuPath script -p myproject/project.qpproj -c "println getCurrentImageData()"

One might also want to connect to a server and list the images contained, selecting which to open that way.

In the future, there will likely be a need for other things (e.g. requesting or sending objects). Basically, if it can be done through the UI then there is a strong chance a user will want to script it some day...

then the image will be opened with the Ice API. Is that a problem?

Yes, it's important that we try to ensure the user always receives the same pixels for the same image in their project. If they request the web version, they should always receive that - because it will be JPEG-compressed, and therefore have different pixel values.

If it switches to use ICE, then the results of tasks like cell detection will be different (and also the performance will be different).

The choice of server might be encoded in the string args that are associated with ImageServer / ServerBuilder.

Switching between implementations that give raw pixel access may be permitted.

The number of channels, z-slices, and timepoints are indicated, but I didn't find a way to get the uncompressed size from the web API.

You can calculate this by multiplying all the other dimensions, and then multiplying the result by the bytes per pixel - see here for an example.