metadata format and entry

[DavidTingley]

This is really more of a place to start discussion than code production. Currently, we have two places where metadata is stored at the individual session level (animal metadata is duplicated elsewhere) 1) the .xml and 2) the *.sessionInfo.mat file. Going forward we should establish a single internal lab format that is easy for newcomers to adopt and extensible to new data types (imaging, opto, patch, etc).

Some things we should figure out:

• Should entry be text or GUI based?
• Do we store the data in a xml or .mat file?
• Should we have different priorities for different parts of the metadata (i.e. required, suggested, optional)?
• others?

[dlevenstein]

-I think we need something that is GUI based -An update to the .xml would be ideal - the xml already works with neuroscope and has a gui editor. All we need to do is add the fields we need, update the GUI, and improve LoadParameters to reflect the new fields. (and get rid of depreciated fields) -with the GUI we can enforce required/suggested/optional -we already have a list fields that we'd like to add that brendon worked on pretty extensively, can be refined as we go

[samamckenzie]

I have been talking with June about this. I think we should make one .mat file that has everything we want and then build a GUI to populate it. This should be in addition to and somewhat redundant with the XML for now and perhaps one day we drop the XML, or the .mat, but for now I think we need both

On Thu, Aug 17, 2017 at 6:50 PM, Dan Levenstein notifications@github.com wrote:

-I think we need something that is GUI based -An update to the .xml would be ideal - the xml already works with neuroscope and has a gui editor. All we need to do is add the fields we need, update the GUI, and improve LoadParameters to reflect the new fields. (and get rid of depreciated fields) -with the GUI we can enforce required/suggested/optional -we already have a list fields that we'd like to add that brendon worked on pretty extensively, can be refined as we go

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323215077, or mute the thread https://github.com/notifications/unsubscribe-auth/ANnfgOEUI0bC6Mhswv-DfOCsRSqhaMewks5sZMOpgaJpZM4O62X5 .

[brendonw1]

As it is now, the .SessionMetadata includes the animal metadata so they are unified.

Furthermore I made it so .xml is not redundant but is BASED on the .mats, so no risk of mistakes. Making the animal metadata lets you create the .xml from it... which I wrote code for. So you can record that way on day 1 before you have a session to make metadata for. So you make animal metadata once and make the .xml from that so it's NOT redundant, but the.mat is more primary... which is how it should be

Then when making session metadata you can just incorporate that animal metadata

Finally, I think GUIs are fine but actually very inefficient as far as time... ie every single day you have to open the GUI. So if you do that then make it so there's another way to do it for people who find text more efficient than GUI... in my case I just have a .m file that's a script for creating the SessionMetadata.mat

On Thu, Aug 17, 2017 at 6:52 PM samamckenzie notifications@github.com wrote:

I have been talking with June about this. I think we should make one .mat file that has everything we want and then build a GUI to populate it. This should be in addition to and somewhat redundant with the XML for now and perhaps one day we drop the XML, or the .mat, but for now I think we need both

On Thu, Aug 17, 2017 at 6:50 PM, Dan Levenstein notifications@github.com wrote:

-I think we need something that is GUI based -An update to the .xml would be ideal - the xml already works with neuroscope and has a gui editor. All we need to do is add the fields we need, update the GUI, and improve LoadParameters to reflect the new fields. (and get rid of depreciated fields) -with the GUI we can enforce required/suggested/optional -we already have a list fields that we'd like to add that brendon worked on pretty extensively, can be refined as we go

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323215077 , or mute the thread https://github.com/notifications/unsubscribe-auth/ANnfgOEUI0bC6Mhswv- DfOCsRSqhaMewks5sZMOpgaJpZM4O62X5 .

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323215392, or mute the thread https://github.com/notifications/unsubscribe-auth/ADXrTT2-AmJ6-zRG3fbo6s0IuhjduUd6ks5sZMQagaJpZM4O62X5 .

[samamckenzie]

how do you update the metadata for the unique properties of a session? here a prepopulated GUI would be nice

On Thu, Aug 17, 2017 at 7:15 PM, Brendon Watson notifications@github.com wrote:

As it is now, the .SessionMetadata includes the animal metadata so they are unified.

Furthermore I made it so .xml is not redundant but is BASED on the .mats, so no risk of mistakes. Making the animal metadata lets you create the .xml from it... which I wrote code for. So you can record that way on day 1 before you have a session to make metadata for. So you make animal metadata once and make the .xml from that so it's NOT redundant, but the.mat is more primary... which is how it should be

Then when making session metadata you can just incorporate that animal metadata

Finally, I think GUIs are fine but actually very inefficient as far as time... ie every single day you have to open the GUI. So if you do that then make it so there's another way to do it for people who find text more efficient than GUI... in my case I just have a .m file that's a script for creating the SessionMetadata.mat

On Thu, Aug 17, 2017 at 6:52 PM samamckenzie notifications@github.com wrote:

I have been talking with June about this. I think we should make one .mat file that has everything we want and then build a GUI to populate it. This should be in addition to and somewhat redundant with the XML for now and perhaps one day we drop the XML, or the .mat, but for now I think we need both

On Thu, Aug 17, 2017 at 6:50 PM, Dan Levenstein < notifications@github.com> wrote:

-I think we need something that is GUI based -An update to the .xml would be ideal - the xml already works with neuroscope and has a gui editor. All we need to do is add the fields we need, update the GUI, and improve LoadParameters to reflect the new fields. (and get rid of depreciated fields) -with the GUI we can enforce required/suggested/optional -we already have a list fields that we'd like to add that brendon worked on pretty extensively, can be refined as we go

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323215077 , or mute the thread https://github.com/notifications/unsubscribe-auth/ANnfgOEUI0bC6Mhswv- DfOCsRSqhaMewks5sZMOpgaJpZM4O62X5 .

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323215392 , or mute the thread https://github.com/notifications/unsubscribe-auth/ADXrTT2-AmJ6- zRG3fbo6s0IuhjduUd6ks5sZMQagaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323218980, or mute the thread https://github.com/notifications/unsubscribe-auth/ANnfgEBY1ustb9UvtVLaG-QUHbYtPzsjks5sZMmfgaJpZM4O62X5 .

[brendonw1]

You can do that or I just copy the .m file that made the one from the previous day (sitting in the previous folder, or actually in the folder above as a template) and copy that. But yeah you can do it with a gui.

On Thu, Aug 17, 2017 at 4:26 PM, samamckenzie notifications@github.com wrote:

how do you update the metadata for the unique properties of a session? here a prepopulated GUI would be nice

On Thu, Aug 17, 2017 at 7:15 PM, Brendon Watson notifications@github.com wrote:

As it is now, the .SessionMetadata includes the animal metadata so they are unified.

Furthermore I made it so .xml is not redundant but is BASED on the .mats, so no risk of mistakes. Making the animal metadata lets you create the .xml from it... which I wrote code for. So you can record that way on day 1 before you have a session to make metadata for. So you make animal metadata once and make the .xml from that so it's NOT redundant, but the.mat is more primary... which is how it should be

Then when making session metadata you can just incorporate that animal metadata

Finally, I think GUIs are fine but actually very inefficient as far as time... ie every single day you have to open the GUI. So if you do that then make it so there's another way to do it for people who find text more efficient than GUI... in my case I just have a .m file that's a script for creating the SessionMetadata.mat

On Thu, Aug 17, 2017 at 6:52 PM samamckenzie notifications@github.com wrote:

I have been talking with June about this. I think we should make one .mat file that has everything we want and then build a GUI to populate it. This should be in addition to and somewhat redundant with the XML for now and perhaps one day we drop the XML, or the .mat, but for now I think we need both

On Thu, Aug 17, 2017 at 6:50 PM, Dan Levenstein < notifications@github.com> wrote:

-I think we need something that is GUI based -An update to the .xml would be ideal - the xml already works with neuroscope and has a gui editor. All we need to do is add the fields we need, update the GUI, and improve LoadParameters to reflect the new fields. (and get rid of depreciated fields) -with the GUI we can enforce required/suggested/optional -we already have a list fields that we'd like to add that brendon worked on pretty extensively, can be refined as we go

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323215077 , or mute the thread https://github.com/notifications/unsubscribe- auth/ANnfgOEUI0bC6Mhswv- DfOCsRSqhaMewks5sZMOpgaJpZM4O62X5 .

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323215392 , or mute the thread https://github.com/notifications/unsubscribe-auth/ADXrTT2-AmJ6- zRG3fbo6s0IuhjduUd6ks5sZMQagaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323218980 , or mute the thread https://github.com/notifications/unsubscribe- auth/ANnfgEBY1ustb9UvtVLaG-QUHbYtPzsjks5sZMmfgaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323220574, or mute the thread https://github.com/notifications/unsubscribe-auth/ADXrTUKGonM60wNHAFtj7eaFMomqv9Wzks5sZMwvgaJpZM4O62X5 .

[juneshuoyang]

This can be done by building an import function on the GUI. Each time, a user can import an existing file, modify it, and save it to a new location via the GUI.

On Thu, Aug 17, 2017 at 7:29 PM, Brendon Watson notifications@github.com wrote:

You can do that or I just copy the .m file that made the one from the previous day (sitting in the previous folder, or actually in the folder above as a template) and copy that. But yeah you can do it with a gui.

On Thu, Aug 17, 2017 at 4:26 PM, samamckenzie notifications@github.com wrote:

how do you update the metadata for the unique properties of a session? here a prepopulated GUI would be nice

On Thu, Aug 17, 2017 at 7:15 PM, Brendon Watson < notifications@github.com> wrote:

As it is now, the .SessionMetadata includes the animal metadata so they are unified.

Furthermore I made it so .xml is not redundant but is BASED on the .mats, so no risk of mistakes. Making the animal metadata lets you create the .xml from it... which I wrote code for. So you can record that way on day 1 before you have a session to make metadata for. So you make animal metadata once and make the .xml from that so it's NOT redundant, but the.mat is more primary... which is how it should be

Then when making session metadata you can just incorporate that animal metadata

Finally, I think GUIs are fine but actually very inefficient as far as time... ie every single day you have to open the GUI. So if you do that then make it so there's another way to do it for people who find text more efficient than GUI... in my case I just have a .m file that's a script for creating the SessionMetadata.mat

On Thu, Aug 17, 2017 at 6:52 PM samamckenzie <notifications@github.com

wrote:

I have been talking with June about this. I think we should make one .mat file that has everything we want and then build a GUI to populate it. This should be in addition to and somewhat redundant with the XML for now and perhaps one day we drop the XML, or the .mat, but for now I think we need both

On Thu, Aug 17, 2017 at 6:50 PM, Dan Levenstein < notifications@github.com> wrote:

-I think we need something that is GUI based -An update to the .xml would be ideal - the xml already works with neuroscope and has a gui editor. All we need to do is add the fields we need, update the GUI, and improve LoadParameters to reflect the new fields. (and get rid of depreciated fields) -with the GUI we can enforce required/suggested/optional -we already have a list fields that we'd like to add that brendon worked on pretty extensively, can be refined as we go

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323215077 , or mute the thread https://github.com/notifications/unsubscribe- auth/ANnfgOEUI0bC6Mhswv- DfOCsRSqhaMewks5sZMOpgaJpZM4O62X5 .

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323215392 , or mute the thread https://github.com/notifications/unsubscribe-auth/ADXrTT2-AmJ6- zRG3fbo6s0IuhjduUd6ks5sZMQagaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323218980 , or mute the thread https://github.com/notifications/unsubscribe- auth/ANnfgEBY1ustb9UvtVLaG-QUHbYtPzsjks5sZMmfgaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323220574 , or mute the thread https://github.com/notifications/unsubscribe-auth/ ADXrTUKGonM60wNHAFtj7eaFMomqv9Wzks5sZMwvgaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323221009, or mute the thread https://github.com/notifications/unsubscribe-auth/AM4sx23xsEzYt9KSHsI_XJao82i5ITUHks5sZMzugaJpZM4O62X5 .

[brendonw1]

Yes. What I set up was a system that does the following when you tell it to, ie after a new recording:

1) Copies .dats from the recording computer (via a mapped drive on wherever you're running MATLAB) 2) Concatenates .dats, makes .lfp (Stores lengths of each dat, timestamps etc etc) 3) Then searches for a template .SessionMetadata in the animal folder and copies it down into the new session folder

• you can open the gui or texteditor to edit the session metadata by opening one or the other at that point

On Thu, Aug 17, 2017 at 4:35 PM, juneshuoyang notifications@github.com wrote:

This can be done by building an import function on the GUI. Each time, a user can import an existing file, modify it, and save it to a new location via the GUI.

On Thu, Aug 17, 2017 at 7:29 PM, Brendon Watson notifications@github.com

wrote:

You can do that or I just copy the .m file that made the one from the previous day (sitting in the previous folder, or actually in the folder above as a template) and copy that. But yeah you can do it with a gui.

On Thu, Aug 17, 2017 at 4:26 PM, samamckenzie notifications@github.com wrote:

how do you update the metadata for the unique properties of a session? here a prepopulated GUI would be nice

On Thu, Aug 17, 2017 at 7:15 PM, Brendon Watson < notifications@github.com> wrote:

As it is now, the .SessionMetadata includes the animal metadata so they are unified.

Furthermore I made it so .xml is not redundant but is BASED on the .mats, so no risk of mistakes. Making the animal metadata lets you create the .xml from it... which I wrote code for. So you can record that way on day 1 before you have a session to make metadata for. So you make animal metadata once and make the .xml from that so it's NOT redundant, but the.mat is more primary... which is how it should be

Then when making session metadata you can just incorporate that animal metadata

Finally, I think GUIs are fine but actually very inefficient as far as time... ie every single day you have to open the GUI. So if you do that then make it so there's another way to do it for people who find text more efficient than GUI... in my case I just have a .m file that's a script for creating the SessionMetadata.mat

On Thu, Aug 17, 2017 at 6:52 PM samamckenzie < notifications@github.com

wrote:

I have been talking with June about this. I think we should make one .mat file that has everything we want and then build a GUI to populate it. This should be in addition to and somewhat redundant with the XML for now and perhaps one day we drop the XML, or the .mat, but for now I think we need both

On Thu, Aug 17, 2017 at 6:50 PM, Dan Levenstein < notifications@github.com> wrote:

-I think we need something that is GUI based -An update to the .xml would be ideal - the xml already works with neuroscope and has a gui editor. All we need to do is add the fields we need, update the GUI, and improve LoadParameters to reflect the new fields. (and get rid of depreciated fields) -with the GUI we can enforce required/suggested/optional -we already have a list fields that we'd like to add that brendon worked on pretty extensively, can be refined as we go

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323215077 , or mute the thread https://github.com/notifications/unsubscribe- auth/ANnfgOEUI0bC6Mhswv- DfOCsRSqhaMewks5sZMOpgaJpZM4O62X5 .

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323215392 , or mute the thread https://github.com/notifications/unsubscribe-auth/ADXrTT2-AmJ6- zRG3fbo6s0IuhjduUd6ks5sZMQagaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323218980 , or mute the thread https://github.com/notifications/unsubscribe- auth/ANnfgEBY1ustb9UvtVLaG-QUHbYtPzsjks5sZMmfgaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323220574 , or mute the thread https://github.com/notifications/unsubscribe-auth/ ADXrTUKGonM60wNHAFtj7eaFMomqv9Wzks5sZMwvgaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323221009 , or mute the thread https://github.com/notifications/unsubscribe-auth/AM4sx23xsEzYt9KSHsI_ XJao82i5ITUHks5sZMzugaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323221792, or mute the thread https://github.com/notifications/unsubscribe-auth/ADXrTXREZpM_29rNJU-BoA2WxSL-Q7DGks5sZM4ogaJpZM4O62X5 .

[dlevenstein]

Import function is very good. I can't say what's good for data collection, and obviously we should do what needs to be done to make that process work well. But I think GUI is necessary for any metadata added post-recording during analysis. I.e. adding badchannels or other channel tags. Or for perusing recordings in a database. Also anything we do needs to be able to import from the old xmls so we can import the metadata from all the legacy datasets

[dlevenstein]

Also the system needs to be robust/flexible enough that it can still work if you're not doing the precise data collection pipeline imagined, as I'm sure others will have different needs

[dlevenstein]

any who - I think if we had a good GUI for the .mat, which automatically produced and synced an xml, that could work

[juneshuoyang]

Just what I am thinking now, as I need to go over more dataset folders to see how each lab member fills out the metadata.

We need to figure out what are the common properties and unique properties of each session.

For common properties, we may need to setup a rule to unify user inputs.

Otherwise, it will require a bunch of data cleaning afterwards, as SQL will recognize Bob and Robert as two different persons.

On Thu, Aug 17, 2017 at 7:26 PM, samamckenzie notifications@github.com wrote:

how do you update the metadata for the unique properties of a session? here a prepopulated GUI would be nice

On Thu, Aug 17, 2017 at 7:15 PM, Brendon Watson notifications@github.com wrote:

As it is now, the .SessionMetadata includes the animal metadata so they are unified.

Furthermore I made it so .xml is not redundant but is BASED on the .mats, so no risk of mistakes. Making the animal metadata lets you create the .xml from it... which I wrote code for. So you can record that way on day 1 before you have a session to make metadata for. So you make animal metadata once and make the .xml from that so it's NOT redundant, but the.mat is more primary... which is how it should be

Then when making session metadata you can just incorporate that animal metadata

Finally, I think GUIs are fine but actually very inefficient as far as time... ie every single day you have to open the GUI. So if you do that then make it so there's another way to do it for people who find text more efficient than GUI... in my case I just have a .m file that's a script for creating the SessionMetadata.mat

On Thu, Aug 17, 2017 at 6:52 PM samamckenzie notifications@github.com wrote:

I have been talking with June about this. I think we should make one .mat file that has everything we want and then build a GUI to populate it. This should be in addition to and somewhat redundant with the XML for now and perhaps one day we drop the XML, or the .mat, but for now I think we need both

On Thu, Aug 17, 2017 at 6:50 PM, Dan Levenstein < notifications@github.com> wrote:

-I think we need something that is GUI based -An update to the .xml would be ideal - the xml already works with neuroscope and has a gui editor. All we need to do is add the fields we need, update the GUI, and improve LoadParameters to reflect the new fields. (and get rid of depreciated fields) -with the GUI we can enforce required/suggested/optional -we already have a list fields that we'd like to add that brendon worked on pretty extensively, can be refined as we go

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323215077 , or mute the thread https://github.com/notifications/unsubscribe- auth/ANnfgOEUI0bC6Mhswv- DfOCsRSqhaMewks5sZMOpgaJpZM4O62X5 .

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86# issuecomment-323215392 , or mute the thread https://github.com/notifications/unsubscribe-auth/ADXrTT2-AmJ6- zRG3fbo6s0IuhjduUd6ks5sZMQagaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323218980 , or mute the thread https://github.com/notifications/unsubscribe- auth/ANnfgEBY1ustb9UvtVLaG-QUHbYtPzsjks5sZMmfgaJpZM4O62X5 .

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323220574, or mute the thread https://github.com/notifications/unsubscribe-auth/AM4sx3r0FepwaCfEldeQvAb5iYEtwuCTks5sZMwvgaJpZM4O62X5 .

[dlevenstein]

maybe we can give a lot of things dropdown menus, with the ability to add new options that then gets added to everyone's dropdown menu (via buzcode) when they make a new metadata file

this would be especially good for region, experimenter etc

as a side note, for region we probably need like 3 fields: Gross Anatomical Region (HPC,CTX), Region (CA1, mPFC), SubRegion(L5,Pyramidal Layer, etc). I think this matches what they have at Allen... we could even just use the Allen map regions

[brendonw1]

Sorry if I was overly attached to my previous setup. I like it but not everyone will. We just need something that works - hopefully from all angles

As for June's question - Has anyone used the metadata other than me? I would try to get my data to June but with Ratgers2 down I can't. Maybe Dan L has some in the william dataset?

What I at least attempted to do was to set up both a system that works now but which is also sitting inside a general framework that will allow future users to add new classes of information. I also tried to tie into the flow of actually recording data. If June or anyone wants me to walk you through what I've done I'm happy to do so. On Thu, Aug 17, 2017 at 8:00 PM Dan Levenstein notifications@github.com wrote:

maybe we can give a lot of things dropdown menus, with the ability to add new options that then gets added to everyone's dropdown menu (via buzcode) when they make a new metadata file

this would be especially good for region etc

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub https://github.com/buzsakilab/buzcode/issues/86#issuecomment-323225219, or mute the thread https://github.com/notifications/unsubscribe-auth/ADXrTRAMUaiFz4YUABRQmmAQuhJfKuM0ks5sZNQ7gaJpZM4O62X5 .