flatironinstitute / mountainlab-js

MountainLab is data processing, sharing and visualization software for scientists. It is built around MountainSort, spike sorting software, but is designed to be more generally applicable.
Other
43 stars 30 forks source link

Implement mda command in new mountainlab #38

Open magland opened 6 years ago

magland commented 6 years ago

should it be called ml-mda?

wysota commented 6 years ago

I'd vote for simply mda.

Edit: unless it's not a full replacement for 'old mda'.

magland commented 6 years ago

It will be a full replacement. So I'll plan on 'mda'.

magland commented 6 years ago

@wysota. I implemented it and decided to call it mda-info. I will leave this open until you can verify that it is working.

For example, you can test it with this convenient universal kbucket command: mda-info kbucket://b5ecdf1474c5/datasets/synth_datasets/datasets/synth_16ch_30min/firings_true.mda

And also try it on a local .mda file of course.

alexmorley commented 6 years ago
$ mda-info kbucket://b5ecdf1474c5/datasets/synth_datasets/datasets/synth_16ch_30min/firings_true.mda
Invalid number of dimensions **: 255
$ mda-info `ml-prv-locate dataset/raw.mda.prv`
{
    "num_bytes_per_entry": 4,
    "num_dims": 2,
    "dims": [
        4,
        18000000
    ],
    "dtype": "float32",
    "header_size": 20
}
alexmorley commented 6 years ago

Should mda-info be a dependency instead? Or moved to system-packages?

wysota commented 6 years ago

It would be nice to have some basic error detection.

$ ./mda-info README.txt Invalid number of dimensions **: 543518313

Something along the lines of "Hey, this doesn't look like an mda file. Use -f to process the file anyway".

wysota commented 6 years ago

Also, should mda-info also handle these cases (would not be an "info" command then)?

mda get_chunk file.mda chunk_out.mda --index=0,100 --size=4x50 mda create file.mda --dtype=int16 --size=4x1000 mda set_chunk target_file.mda chunk.mda --index=0,100