search

HarleySchool / commonssite

Django-backed website for managing the commons
2 stars 3 forks source link

Graphing app.. #28

Open MB6 opened 9 years ago

MB6 commented 9 years ago

I've made a commonscontrol api graphing app, you can find it at mb6/commonsgraph. Please check it out.

wrongu commented 9 years ago

Very cool! Takes forever to install all of the dependencies, but graphtk.py ran fine once I got everything. (frz.py and commonsgraph.py both break when run from the command line, FYI)

I have to ask, though - what's the plan for using this app? Seems very similar to what the website's analyze page already does.

MB6 commented 9 years ago

Try to run the webapp for over one month of data...

-Max B.

MB6 commented 9 years ago

frz is broken because it is me trying to distribute on windows without installing anything... bbfreeze is unmaintained and doesn't work.

MB6 commented 9 years ago

I could not use ciso8601, but it is the fastest iso 8601 timestamp parser.. i need to make a requirements.txt to speed that up (you just do pip install -r requirements.txt)

wrongu commented 9 years ago

Good point. The long-term solution should be to fix the site.

In terms of downloading big CSV files, using generators to stream rows back would be ideal. The reason that hasn't been done yet is because the split_on_indexes function is a batch operation. The first step would be to streamline that function (for example, we could have it generate the indexes quickly based on only the most recent data points analogous to how we made /data/api/systems faster)

In terms of plotting tons of points with highcharts, this appears relevant.

I'll be working on these things over winter break (when school lets up)

wrongu commented 9 years ago

if you use virtualenv, requirements are easy. pip freeze > requirements.txt

wrongu commented 9 years ago

requirements:

altgraph==0.9
bbfreeze==1.1.3
bbfreeze-loader==1.1.0
ciso8601==1.0.1
matplotlib==1.4.2
mock==1.0.1
nose==1.3.4
numpy==1.9.1
pyparsing==2.0.3
python-dateutil==2.2
pytz==2014.10
requests==2.4.3
six==1.8.0
wsgiref==0.1.2
MB6 commented 9 years ago

BTW, try using that app and make a request for something with indexes for 10 or so months, all that waiting time is the server. I checked.

wrongu commented 9 years ago

Right... Big queries are definitely a bottleneck. Same problem with too much batch processing. There are 2 cards on the ol' Trello that would address this: caching queries and streaming them