Move data extraction and analysis functionality to the minion

[aneesha09]

Problem description

Visualisation is currently doing data extraction and analysis from an external system, it is best to do this in a minion. We can store the data in an aspect to visualise. Also to make it more scalable, desirable to do it in the minion.

As we have a minion for this, should concentrate related complexity there.

There could be complications in rendering things in browser, e.g.:

• CORS issues and proxy setups
• Data behind passwords etc and authentication
• People's tabs crashing/being killed when it is using too much memory
• Will open up visualisation of data formats that are more challenging to work with inside browser.

When we make client side more plugin based, I can see format specific data parsers in the client side inside plugins.

Be very careful about memory usage when doing this. E.g. having an entire file loaded in the memory can crash the minion Make sure it still runs analysis on the complete data, not a subset as the FE does right now

[aneesha09]

Pull logic from front-end to the minion