search

klusta-team / klustaviewa

LEGACY - Graphical interface for spike sorting manual stage in Python
Other
29 stars 17 forks source link

Memory mapping with files in the old format (Klusters) #32

Open rossant opened 10 years ago

rossant commented 10 years ago

There would be no more mandatory conversion before opening the files.

nippoo commented 10 years ago

How would we deal with saving? Would we have to continue supporting this code?

rossant commented 10 years ago

Hmm. I can see three options:

  1. Full identical support of KWIK & Klusters (save multiple clusterings, cluster colors, groups, etc.). Complex since we need to define new file formats with these metadata.
  2. Supporting saving the clusters only (no colors, groups). Only the .clu file is modified.
  3. Read-only support: when loading files in the old format, the user would be prompted to either open the old files quickly in read-only mode, or to convert them to the new file format (slow conversion required first).

I'm leaning toward 3. I think 1 is too much work and not worth it, and 2 is not a good idea (you would want the cluster groups too, at least).

@kdharris101 what do you think? (the question was: whether to support the old file format in KlustaViewa without mandatory conversion, conversion which is a huge drawback according to many users who contacted me).

kdharris101 commented 10 years ago

Hi Cyrille,

I’m sorry I didn’t understand the context (or missed a previous email). Can you explain in more detail?

k

From: Cyrille Rossant [mailto:notifications@github.com] Sent: 06 December 2013 09:11 To: klusta-team/klustaviewa Cc: Harris, Kenneth Subject: Re: [klustaviewa] Memory mapping with files in the old format (Klusters) (#32)

Hmm. I can see three options:

  1. Full identical support of KWIK & Klusters (save multiple clusterings, cluster colors, groups, etc.). Complex since we need to define new file formats with these metadata.
  2. Supporting saving the clusters only (no colors, groups). Only the .clu file is modified.
  3. Read-only support: when loading files in the old format, the user would be prompted to either open the old files quickly in read-only mode, or to convert them to the new file format (slow conversion required first).

I'm leaning toward 3. I think 1 is too much work and not worth it, and 2 is not a good idea (you would want the cluster groups too, at least).

@kdharris101https://github.com/kdharris101 what do you think? (the question was: whether to support the old file format in KlustaViewa without mandatory conversion, conversion which is a huge drawback according to many users who contacted me).

— Reply to this email directly or view it on GitHubhttps://github.com/klusta-team/klustaviewa/issues/32#issuecomment-29973469.

rossant commented 10 years ago

@kdharris101 I'd like to add the possibility to open files in the old file format without any conversion. I recently discovered a very simple technique in Python that allows to do that efficiently, both in terms of loading times and memory consumption. Many people are complaining about the need to convert their terabytes of data before being able to open them in KlustaViewa. I think people will have files in the old format for still quite a few years...

A possibility would be to only support a read-only mode for the old files.