search

ELVIS-Project / vis-framework

Thoroughly modern symbolic musical data analysis suite.
http://elvisproject.ca/
31 stars 6 forks source link

Post Data Generation Processing Tools #384

Closed musicus closed 8 years ago

musicus commented 8 years ago

We need to develop post-data generation processing tools. Thus far we generate data, but we do not have tools to wrangle that data.This issue needs to branch out into numerous issues for the types of data wrangling tools needed.

alexandermorgan commented 8 years ago

I'm not sure exactly what you have in mind here. So far pandas has met all of our data-crunching needs with the small exception of dendrogram visualizations which we use scipy for. I'm also not sure what data you consider to be post-data. On some level, other than importing a piece, all of our analytical steps amount to post-processing data.

alexandermorgan commented 8 years ago

Right, these are all things that we can and do use Pandas for. If there's something that Pandas can't do then we could make an issue for that specifically, but otherwise I think we can close this.

alexandermorgan commented 8 years ago

I still don't understand this issue or #399. If people are coming up with ways of post-processing VIS data using Excel or any other program outside of the framework I don't see why we would document it. All the things that Catherine, Sam, or Emily have coded or worked out in Excel could have been done with Python and Pandas, they just decided to use other tools that they are more familiar with. Once the data leaves the framework, the user is free to do whatever they want with it.

minamouse commented 8 years ago

I think the idea is to show users the best and simplest ways of doing post-processing, as well as just to demonstrate the range of possibilities, especially to non-programmers (or to those who are relatively new to it). I've noticed we get a fair amount of questions about whether or not something is possible in VIS, and often times it's something that is fairly obvious to do, but because most people aren't familiar with pandas, it seems complicated and convoluted to do it, and often not worth it.

It just seems like a shame to be using something as great as pandas when a lot of it is lost on users, because we're letting them invent roundabout ways of getting the results that they want, when there is a much simpler way of doing it. Obviously users can and will end up doing whatever they want or are most comfortable with, but I think we should definitely be providing them with easy ways of learning how we, as VIS-perts (new name for VIS experts?), process results, since we're most familiar with the best ways of doing it.

As a side note, I think leading people in the direction of thinking like a VIS-programmer could also be beneficial to us. If users are post-processing data like we do, it seems more likely that there will be scripts that they've written that will fit into the framework, that we can eventually maybe also start adding to the scripts folder in vis or something like that.

minamouse commented 8 years ago

I've added some of my scripts and one of Sam's. I think that's all I really had, unless there's something else specific you wanted me to add here, @musicus ?