DataFrame [WIP] - Githubissues

G-Node / nix

Neuroscience information exchange format

https://readthedocs.org/projects/nixio/

Other

67 stars 36 forks source link

DataFrame [WIP] #708

Closed gicmo closed 6 years ago

gicmo commented 7 years ago

This is a test balloon to see where if (I am sure it will) breaks on other platforms (win?). It is mostly done. Mostly. Need more tests, as always. Feedback on the API would also be good. Some bits are missing and some are to be discussed:

Appending rows (currently done via rows(NEW_SIZE))?
FS backend and resizing of rows.

gicmo commented 7 years ago

Todo List:

[ ] More tests
[ ] API review and changes
[x] increase the format version number
[x] Documentation
[x] Group's DataFrame access functions
[x] Cell inherit from Variant
[x] read/write Column with string instead of index
[x] check for supported DataType on creation

jgrewe commented 7 years ago

Do we also want to offer some sugar? Things like to and from csv? Can be added later on, of course.

gicmo commented 7 years ago

Do we also want to offer some sugar? Things like to and from csv? Can be added later on, of course.

Yeah, I would add that later. But sounds not like a bad idea. The csv import/export could be used by the FS backend as well.

gicmo commented 7 years ago

Do we want a per-column "type" (not DataType, but as in the second parameter of the DataArray/Block/Group)? A "definition"?

achilleas-k commented 7 years ago

Good question. I would expect the DataFrame to have a single type and description attribute like any other object, but since the data isn't homogeneous, it might useful to be able to specify a description for each. But it also gets a bit messy, doesn't it? Having attributes for each column in this way kinda breaks the idea of a unified table object.

I would lean towards not having type and description per column unless I hear a very compelling argument or example.

gicmo commented 7 years ago

This is getting close. I guess some more tests are needed, but besides that it is basically ready. Review away ;)

gicmo commented 7 years ago

Is there any straightforward way to delete a specific row or column?

Column might be tricky, I am not sure we can alter it, from the docu:

When a dataset is created, the storage datatype is defined. After the dataset or attribute is created, the datatype cannot be changed.

Rows of course should be doable. You think something like DataFrame::deleteRow(ndsize_t index)?

achilleas-k commented 7 years ago

Yeah, delete row should be supported like you mentioned, just an index, or a range that deletes and shifts the rest "up".

As for column, yeah, I wasn't thinking implementation and deleting a common would be a pain. Probably easier to create a new one and copy the rest over, which is horrible.