allow user to specify their own "channel name" in an hdf5 cutout

[alexbaden]

User story: I have a generic script that takes a single channel in an HDF5 file, does some data analysis, and writes it out. I wrote my script for OCP v0.6 which used "CUTOUT" as every channel name. Or, maybe I am an AT user and want to run my script across multiple channels. I'll either have to change the channel name in the source code and recompile, specify it as a command line parameter, or do some sort of fancy HDF5 search to figure out what the group for my data in the HDF5 is (e.g. /DAPI/CUTOUT or /EM/CUTOUT or /WHATEVER/CUTOUT ).

Proposal: Provide the following interface: /ocp/ca/<>/<>/hdf5{<>}/<>

So one could write: /ocp/ca/kasthuri11/image/hdf5{DATA}/<> and get data in /DATA/CUTOUT

I am writing this because it's going to wreak havoc with how I've deved the surfaces stuff. And I already have to take in a ton of command line parameters, so I'm trying to limit that when possible. I believe we only allow one cutout per HDF5 file, so that shouldn't be a problem, but one imagine a few solutions for that. What say you? I could probably implement, but would certainly need @kunallillaney help.

( @jovo @willgray @j6k4m8 @Aeusman -- any of you use this? )

[wrgr]

Hmm...we definitely used to support pulling down multiple channels at once. Interesting idea.

On Mon, Aug 24, 2015 at 7:31 PM Alex Baden notifications@github.com wrote:

User story: I have a generic script that takes a single channel in an HDF5 file, does some data analysis, and writes it out. I wrote my script for OCP v0.6 which used "CUTOUT" as every channel name. Or, maybe I am an AT user and want to run my script across multiple channels. I'll either have to change the channel name in the source code and recompile, specify it as a command line parameter, or do some sort of fancy HDF5 search to figure out what the group for my data in the HDF5 is (e.g. /DAPI/CUTOUT or /EM/CUTOUT or /WHATEVER/CUTOUT ).

Proposal: Provide the following interface: /ocp/ca/<>/<>/hdf5{<>}/<>

So one could write: /ocp/ca/kasthuri11/image/hdf5{DATA}/<> and get data in /DATA/CUTOUT

I am writing this because it's going to wreak havoc with how I've deved the surfaces stuff. And I already have to take in a ton of command line parameters, so I'm trying to limit that when possible. I believe we only allow one cutout per HDF5 file, so that shouldn't be a problem, but one imagine a few solutions for that. What say you? I could probably implement, but would certainly need @kunallillaney https://github.com/kunallillaney help.

( @jovo https://github.com/jovo @willgray https://github.com/willgray @j6k4m8 https://github.com/j6k4m8 @Aeusman https://github.com/Aeusman -- any of you use this? )

— Reply to this email directly or view it on GitHub https://github.com/openconnectome/open-connectome/issues/107.

[alexbaden]

Right -- I'm not entirely sure how we left off multiple channels in one HDF5. But, if we did support multiple channels in one HDF5, we could assume adding the {} negates that -- as in, if you add {CUTOUT} to your REST argument, you only get the first channel in the list. This is more of a power user feature and not really an everyday script feature.

On Mon, Aug 24, 2015 at 4:37 PM William Gray notifications@github.com wrote:

Hmm...we definitely used to support pulling down multiple channels at once. Interesting idea.

On Mon, Aug 24, 2015 at 7:31 PM Alex Baden notifications@github.com wrote:

User story: I have a generic script that takes a single channel in an HDF5 file, does some data analysis, and writes it out. I wrote my script for OCP v0.6 which used "CUTOUT" as every channel name. Or, maybe I am an AT user and want to run my script across multiple channels. I'll either have to change the channel name in the source code and recompile, specify it as a command line parameter, or do some sort of fancy HDF5 search to figure out what the group for my data in the HDF5 is (e.g. /DAPI/CUTOUT or /EM/CUTOUT or /WHATEVER/CUTOUT ).

Proposal: Provide the following interface: /ocp/ca/<>/<>/hdf5{<>}/<>

So one could write: /ocp/ca/kasthuri11/image/hdf5{DATA}/<> and get data in /DATA/CUTOUT

I am writing this because it's going to wreak havoc with how I've deved the surfaces stuff. And I already have to take in a ton of command line parameters, so I'm trying to limit that when possible. I believe we only allow one cutout per HDF5 file, so that shouldn't be a problem, but one imagine a few solutions for that. What say you? I could probably implement, but would certainly need @kunallillaney < https://github.com/kunallillaney> help.

( @jovo https://github.com/jovo @willgray <https://github.com/willgray

@j6k4m8 https://github.com/j6k4m8 @Aeusman <https://github.com/Aeusman

-- any of you use this? )

— Reply to this email directly or view it on GitHub https://github.com/openconnectome/open-connectome/issues/107.

— Reply to this email directly or view it on GitHub https://github.com/openconnectome/open-connectome/issues/107#issuecomment-134414578 .

[jovo]

lets put on the agenda for the lighting round tomorrow...

On Mon, Aug 24, 2015 at 7:39 PM, Alex Baden notifications@github.com wrote:

Right -- I'm not entirely sure how we left off multiple channels in one HDF5. But, if we did support multiple channels in one HDF5, we could assume adding the {} negates that -- as in, if you add {CUTOUT} to your REST argument, you only get the first channel in the list. This is more of a power user feature and not really an everyday script feature.

On Mon, Aug 24, 2015 at 4:37 PM William Gray notifications@github.com wrote:

Hmm...we definitely used to support pulling down multiple channels at once. Interesting idea.

On Mon, Aug 24, 2015 at 7:31 PM Alex Baden notifications@github.com wrote:

User story: I have a generic script that takes a single channel in an HDF5 file, does some data analysis, and writes it out. I wrote my script for OCP v0.6 which used "CUTOUT" as every channel name. Or, maybe I am an AT user and want to run my script across multiple channels. I'll either have to change the channel name in the source code and recompile, specify it as a command line parameter, or do some sort of fancy HDF5 search to figure out what the group for my data in the HDF5 is (e.g. /DAPI/CUTOUT or /EM/CUTOUT or /WHATEVER/CUTOUT ).

Proposal: Provide the following interface: /ocp/ca/<>/<>/hdf5{<>}/<>

So one could write: /ocp/ca/kasthuri11/image/hdf5{DATA}/<> and get data in /DATA/CUTOUT

I am writing this because it's going to wreak havoc with how I've deved the surfaces stuff. And I already have to take in a ton of command line parameters, so I'm trying to limit that when possible. I believe we only allow one cutout per HDF5 file, so that shouldn't be a problem, but one imagine a few solutions for that. What say you? I could probably implement, but would certainly need @kunallillaney < https://github.com/kunallillaney> help.

( @jovo https://github.com/jovo @willgray < https://github.com/willgray

@j6k4m8 https://github.com/j6k4m8 @Aeusman < https://github.com/Aeusman

-- any of you use this? )

— Reply to this email directly or view it on GitHub https://github.com/openconnectome/open-connectome/issues/107.

— Reply to this email directly or view it on GitHub < https://github.com/openconnectome/open-connectome/issues/107#issuecomment-134414578

.

— Reply to this email directly or view it on GitHub https://github.com/openconnectome/open-connectome/issues/107#issuecomment-134414871 .

the glass is all full: half water, half air. openconnecto.me, jovo.me, office hours https://www.google.com/calendar/embed?src=vk2j3ugqcitvq223g7ivvq4hqo%40group.calendar.google.com&ctz=America/New_York

[kunallillaney]

I am confused what is being attempted here. You can cutout mutiple channels in the same hdf5 file or you can only cut one mentioned channel. If you don't mention one then it will give you the default channel. What else is needed?

[alexbaden]

The ability to download a single HDF5 file individually for multiple channels and have the file appear identical, even though the channel names differ.

On Mon, Aug 24, 2015 at 10:25 PM Kunal Lillaney notifications@github.com wrote:

I am confused what is being attempted here. You can cutout mutiple channels in the same hdf5 file or you can only cut one mentioned channel. If you don't mention one then it will give you the default channel. What else is needed?

— Reply to this email directly or view it on GitHub https://github.com/openconnectome/open-connectome/issues/107#issuecomment-134479678 .

[kunallillaney]

We changed this to your specs for CAJAL. I don't get why is so hard to read the channel name from the HDF5 file. You can use npz service if this is hard.