Removing the Datasets page

[Aidan275]

Thinking for a couple reasons we should remove the dataset page and integrate it into the analysis page by changing the selection from radio to check box.

At the moment on the new analysis page when selecting a file or dataset for analysis, the datasets aren't displaying in a nice way when navigating folders (Since datasets don't have a path).

It also adds an additional step for users, which is really useless since the one thing they can do with a dataset is analyse it.

It should be possible to do, the only probably I can think of is when creating a dataset and waiting for it to be available on S3 for Watson to access but this can be avoided by sending the actual concatenated text with the API call, not the URL to the text file.

Thoughts?

[d3nd0]

Maybe. As long as we retain the ability to delete, edit and view datasets.

[Aidan275]

I'm suggesting scraping them all together. Seeing as a user creates a dataset, performs an analysis on it then has no other use for it.

[d3nd0]

So the user selects the files to include in analysis on-the-fly instead of using a separate dataset process? If so, I'd personally agree with that.

[Aidan275]

Yep.

Datasets will only really be useful if a user can do more than one thing with them. Maybe if we manage to add options when performing analyses they would be handy, saving the user from having to select multiple files each time they want to perform a slightly different analysis on the same set of files, but at the moment I don't see this happening.

I suppose a possible analysis option could be an option to exclude given words from the analysis, which wouldn't be too difficult. It would just involve downloading the text to the client, searching the text for the key words and removing them before sending the text to Watson for analysis. So datasets could be handy then...

I guess instead of just removing the datasets we could come up with a better way of listing them in the list of files/datasets available when creating an analysis.

What're everyone's opinions?

[rogersBen]

I think the datasets are a good way to analyse multiple files, we aren't 100% sure what the files are going to be like that they will be uploading and what if Anu has multiple files all of different responses to interviews and she wants to have an analysis on all the answers.

I suppose it depends more on what the files will be like that she is uploading, we can talk with her about it tomorrow.

[Aidan275]

I'm not saying to remove the ability to analyse multiple files, I just mean we should make the new analysis page like the new dataset page, so instead of selecting one file on the new analysis page, you can select multiple files (or one). And if we do that then there's really no need for a dataset page.

[d3nd0]

Add the ability to re-analyse with different options (if we do this) rather then labeling something a dataset.

[Guycorr]

I agree with Reece if the dataset page is scrapped the functionality of that page should be placed in analysis with all the options

[rogersBen]

Yep that sounds good. We will need to think about having the option to exclude stuff from the analysis anyway I reckon, I tried running the watson analysis on one of Anus papers that she put out and all of the dates and references threw it off.

[rogersBen]

I'm having a go at doing this like this: in newAnalysis controller: download the files to be analyzed concatenate them into a string send that string to the server server sends it to watson for analysis (which needs a new API call) watson sends results back to server, server saves the results on s3 sends the results to the browser

[rogersBen]

Pretty sure this is done now, just need to remove the datasets code