`WaveorderReader` uses `glob` to read a folder of ome-tiffs

[talonchandler]

The WaveorderReader scrambles the position order when it reads a folder of ome-tiffs.

For example:

$ python
>>> from waveorder.io import WaveorderReader
>>> reader = WaveorderReader('/hpc/projects/compmicro/raw_data/hummingbird/all_21_3/') # will take ~30 seconds
>>> reader.reader.files[0:3]
['...MMStack_232.ome.tif', '...MMStack_177.ome.tif', '...MMStack_57.ome.tif']

This line seems to be the culprit, since glob docs say that the files will be returned in arbitrary order.

A possible solution is to wrap the glob in a natsort.natsorted function. But maybe theres a better way that uses the MM metadata instead of the file names?

[ziw-liu]

I think the MM metadata (IndexMap) is used to get the correct ordering of the data axes.

AFAIK reader.files is only used to extract metadata that is shared across the dataset (from reader.files[0], which assumes that things like pixel size do not change within the dataset). Although it is possible to re-order based on the time stamp, I cannot see when the arbitrary ordering will become a problem. @talonchandler can you explain the use case where this change have to happen within the reader?

[talonchandler]

Thanks Ziwen, this might not be the root issue that I thought it was. I think you're right that reader.files is only used to extract shared metadata across the dataset.

I was trying to debug an ordering issue for a collaborator (they're reporting unexpected input-output ordering from WaveorderReader), and I thought I'd found the issue in reader.files---at the very least it's unexpected that that reader.files has a random order, even if it's not causing a bug.

Let's leave this open for now, and I'll report back after I get a better handle on their dataset.

[talonchandler]

The root issue was a misunderstanding between Micromanager's position order and ImageJ's position order. If you drag the folder of ome.tifs into ImageJ, it will open in an order that depends on the file system. But if you drag the folder of ome.tifs into MicroManager, it will open in the "correct" MicroManager order.

WaveorderReader opens in the same order as MicroManager, so it is opening the files in the correct order.

I will still suggest renaming reader.reader.files to reader.reader._files since this added an extra layer of uncertainty to my debugging. What do you think?

[ziw-liu]

I will still suggest renaming reader.reader.files to reader.reader._files

Do you want this change to happen in waveorder.io or iohub (or both)?

[talonchandler]

I'm flexible...if we're migrating soon then just iohub is fine.

[talonchandler]

Closing since this is fixed in iohub and we're migrating soon.