Zarr

[jburel]

Add notebooks and scripts to use zarr:

• pixel_classification_zarr.ipynb notebook

  • load the metadata from OMERO
  • read the binary from S3 (ebi)
  • analyse the data using ilastik
  • save the results back to OMERO as zarr zip
  • read the zip and display using itk

• pixel_classification_zarr_idr.ipynb notebook

  • read the binary from S3 (ebi)
  • analyse the data using ilastik
  • display the results using itk

[jburel]

• I think it is fine to add extra dependencies. I will put a comment indicating that this could be removed.
• Ideally the s3 connection should come from a common lib since we will need the same approach in other repo.
• I will remove the saving step as .zip. I will keep a copy if we need to re-inject it
• The logging, @pwalczysko pointed that out. I need to pass parameter to ilastik to turn that off.

[pwalczysko]

Using repo2docker, and the idr notebook. See below - took quite a while.

%time input_data = load_from_s3(image_id)

CPU times: user 5.58 s, sys: 3.1 s, total: 8.68 s
Wall time: 31.2 s

[jburel]

This is faster than loading directly using getPlanes (~x3 faster) you are running the docker image locally

[pwalczysko]

repo2docker, idr notebook, the step

Load each image as an 5D-numpy array and analyze

Note, this error repeated on Mybinder later https://github.com/ome/omero-guide-ilastik/pull/11#issuecomment-619971488

Got a message from Jupyter - I would not say that the message is accurrate (did not say OK, just closed the window, the process seem to be getting on nicely)

[pwalczysko]

Weel, now, the workflow is stuck, I am getting

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-2993c68bb682> in <module>
      1 # Load the model linked to the dataset
----> 2 home = expanduser("~")
      3 model_file = home+"/notebooks/pipelines/pixel-class-133.ilp"
      4 # Re-order the array tczyz -> tzyxc
      5 input_data = input_data.swapaxes(1, 2).swapaxes(2, 3).swapaxes(3, 4)

NameError: name 'expanduser' is not defined

Possibly the kernel really was restarted, and the procedure is missing some definitions done in the previous cells ?

[jburel]

You need to start again from the first cell. Strange that the kernel dies. After which cell?

[pwalczysko]

@jburel After

Load each image as an 5D-numpy array and analyze. - interestingly, the second time around, this worked fine. But, after I have done it the second time, probably something expired again, after I wanted to close the connection, see below

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-10-8dd0cd799055> in <module>
----> 1 conn.close()

NameError: name 'conn' is not defined

[pwalczysko]

repo2docker worked principally fine for both notebooks. But, as the waiting times can be substantial, and also smooth run is not guaranteed, maybe there could be some provision for keeping the session alive ? I think I have expired twice, in each notebook once. Once I have crashed, but cannot repeat, see above (in the idr notebook).

[jburel]

@joshmoore removed the zip zarr section and added info in env file

[pwalczysko]

Mybinder

Load each image as an 5D-numpy array and analyze. step of the non-idr notebook.

I suspect that the kernel is hanging again, the cell has star for a long time now, no new output is incoming.

Oh, sorry, it made it in the end.

[pwalczysko]

mybinder

Load the results saved as zip and read the data for further analysis.

Again, a long wait. I suspect this will result again in me loosing the connection to the OMERO.server.

[jburel]

@pwalczysko I have added enableKeepAlive this should hopefully avoid the problem

[pwalczysko]

Mybinder

View the first channel so we can compare with the 3D view of the results of the pixel classification.

And again, see below - just after the data from the idr notebook loaded and I started to turn the 3D view of the original data around, the message popped up

As expected, after I got rid of the window, next cell is erroring.

NameError: name 'expanduser' is not defined

Again, this was fine after the second run of the same notebook.

[pwalczysko]

Alas, note that the stoppage can also vary - more than 4 times longer this time (from mybinder)

%time input_data = load_from_s3(image_id)

CPU times: user 4.31 s, sys: 651 ms, total: 4.96 s
Wall time: 2min 30s

[pwalczysko]

Mybinder worked principally as well. The problems encountedred were matching the repo2docker epxperience exactly - one unexpected kernel restart. I did not completely time out on mybinder, but only just.

[jburel]

Thanks for the feedback and testing, I have made the changes, merging and moving forward with next round