search

UQ-RCC / ipp

The IPP is a Web portal for biologists to interact with High Performance Compute (HPC) systems. Designed and funded by the IMB Microscopy Facility & the Research Computing Centre (RCC)
https://rcc.uq.edu.au/
MIT License
3 stars 0 forks source link

Metadata scrapers #50

Open IMBMicroscopy opened 2 years ago

IMBMicroscopy commented 2 years ago

Implement custom metadata scrapers to read additional metadata for: Deconvolution required fields such as Instrument model, Channel name, modality,pinhole size, emission wavelength, objective RI, NA, Mag etc These scrapers will also be used to autodetect file format, channel name etc for file conversion, PSF processing etc

DrSpringy commented 2 years ago

We need to define the list of required additional metadata as per our template:

Microscope File Formats and Metadata.xlsx

DrSpringy commented 2 years ago

Back Projected Pinhole radius calculator.xlsx

We need to calculate automatically the back projected pinhole diameter based on the microscope model, objective mag, additional zoom, pinhole size etc

DrSpringy commented 2 years ago

From Marc Bruce: I haven't found the bad NA read dataset, but at this point I think it was a missed read (data present but not parsed) rather than a wrong read. I can share some of my metadata-parsing-in-progress; partially manually re-parsing the original metadata reported by Bio-Formats, partially full file parsing for some formats where the metadata is somewhat more accessible.

I'll have to expand the core API for the file format writer library that's within the batch plugin now for fast file writing. It supports ims, IJ-compatible tiff, and OME-tiff (with hand-written TIFF headers and OME XML since the native Java writer by Bio-Formats is still super slow for no reason). Even the older IJ-tiff direct from IJ API should read back correctly.

Basic metadata from the deconvolution is currently inserted in ims and ome-tiff for theoretical PSFs. Attaching generic data to ome-tiff is tricky since their XML is so "flexible" that it doesn't permit unknown fields.

I've also got WIP code ready to go once the next gen OME-Zarr formats are finalized.

Partial correction - batch GUI still uses IJ API to write a bare tiff (so it should have the correct dimension header already).

How are you saving files out via the API currently?

Also, your yearly update PO came in. There shouldn't be any rush from a license point of view since the last update was December if I recall. I'll prepare the necessary files and get back to you in a few days.

DrSpringy commented 1 year ago
Screenshot 2022-12-09 at 1 04 38 pm Screenshot 2022-12-09 at 1 04 51 pm Screenshot 2022-12-09 at 1 04 43 pm
markendr commented 3 months ago

JS, NC, ND, ME zoom call on 20-Mar-2024:

Need to workshop further to determine suitable scope for 2H2024, aiming to improve..