search

htm-community / nupic.visualizations

Web application for interactive graphs, anomaly highlighting and online monitoring.
MIT License
17 stars 11 forks source link

WIP: continuous monitoring support #67

Open breznak opened 8 years ago

breznak commented 8 years ago

Fixes: #17

breznak commented 8 years ago

@jefffohl for your preliminary review, if you have time. thx

breznak commented 8 years ago

@jefffohl can you please review this? The monitoring/online updates are working as a draft. I have a problem with the large (2M+) file, re-reading it in PapaParse take some time, but I know the updatade file size, if possible, I'd like to seek/skip to the bytes read and only read the new bytes. I'm somewhat doing it with conditional skipping in the chunk call, but it's still slow

breznak commented 8 years ago

resolved merge conflicts, hope I haven't introduced any bugs during the merge.

breznak commented 8 years ago

@jefffohl how much of this PR are you reusing in your work on #78 ? I was thinking about rebasing a PR off this PR to have only the working functionality, introduction of the Angular's $interval and possibly limited support for monitoring with diff files only. I hope the work on proxy should then be separate and further extend the monitor. capabilities.

jefffohl commented 8 years ago

I am not using any of this code, as it seemed that the approach wouldn't work. I am sorry if I wasn't clear about that. I don't think we should be working on the same feature at the same time.

I am currently working on the server implementation, which is required for online monitoring.

breznak commented 8 years ago

Thanks Jeff. So it would be ok if I go ahead with

introduction of the Angular's $interval and possibly limited support for monitoring with diff files only

from this PR? It would not intervene with any of your work on the server?

jefffohl commented 8 years ago

To be honest, I don't think it is a good idea to keep working on this PR. The code is too divergent between our two branches, and it is going to be tough to merge them. It might be best to wait until I am finished with my server work, and then, if you want to implement a method for monitoring files using a diff method and the fileReader API, you can do that after I have finished the server. At that point, I am hoping you will be able to rebase the server work on to this branch. I don't think it is wise for us to be working concurrently on these features, as there is so much code overlap.

breznak commented 8 years ago

Ok, I'll wait for your server part. Btw, do you have a rough ETA?

jefffohl commented 8 years ago

I am hoping to have it ready before Friday, but there are some things I haven't figured out how to do yet. If I get stuck, I will let you know, and ask for help.

jefffohl commented 8 years ago

@breznak - Sorry, I am still working on this. Making progress though. I hope that this is not holding you back from things that you need to do.

jefffohl commented 8 years ago

@breznak - sorry still working on this. Some things came up this week that I had to prioritize.

jefffohl commented 8 years ago

@breznak - I am working on this issue on this branch: https://github.com/jefffohl/nupic.visualizations/tree/issue78

There are a number of fundamental changes happening here. Specifically:

Remotely hosted files can be parsed, as well as local files. For local files, you must give a full path from the system root to the file. Note that because of this feature, the code as it currently stands should NEVER be hosted on a public server, as that will give the public access to all the files on the web server. We will need to figure out a good way to make sure that this feature is disabled if the app is ever hosted publicly. Right now, the intent is that the app should only be run locally.

Local files can be continuously updated. I added a little bash script: /examples/tests/update.sh that will continuously update the no_timestamp.csv file. You can use this to test the continuous update feature.

If you want to test this branch as it is, first make sure that node and npm are installed. Then run npm install followed by npm start

jefffohl commented 8 years ago

@breznak I have a working prototype on this branch: https://github.com/jefffohl/nupic.visualizations/tree/issue78

This version allows you to "play" a local file. You can enter the path to a local file, and it will load a portion of the file. Clicking the "play" button will start to stream the file from the file system into the web app.

Next steps are:

breznak commented 8 years ago

@jefffohl sorry for the delay, I was off the grid

That sounds great, will you make a PR?

jefffohl commented 8 years ago

I could. I was thinking that I would get some of the other steps completed first, but I can make a PR if you like. I am on vacation right now, but I can do it when I return on Monday.

jefffohl commented 8 years ago

@brezank - I am thinking of making a new PR for what I have now in https://github.com/jefffohl/nupic.visualizations/tree/issue78

Note that this is almost a complete rewrite of the architecture, due to parsing the data now on the server. Some additional things I also changed along the way:

Let me know your thoughts.

breznak commented 8 years ago

@jefffohl thanks Jeff, sorry for the silence, I was somehow missing all notifications from this project :/ Will test over the weekend.. IMHO sounds good.