Application Development

[alexklibisz]

Hi all, so it's time that we start discussing creating the final application.

Workflow

As we discussed in class, the workflow at a very high level should be:

1. User input 1.1. User enters a search string for features that he/she needs. 1.2. User selects a repos that he/she has used from our list of repos. 1.3. User clicks "search".
2. Search step 2.1. Our system returns a list of repos that satisfy two parameters 2.2. Repos match the search string 2.3. Repos have a relationship value > 0 to the repos that user has used (1.2)

   Platform

This will likely be a python-bsed HTTP server with endpoints defined for the search queries against mongodb and the simserver that @inthesunset wrote implemented as a python module. I'm planning to try django first. The front-end will be an HTML/CSS/JS webapp implemented with Angular.js.

I will begin working on setting up the HTTP server now, and will post back about my progress.

[alexklibisz]

I ended up setting up a skeleton of an http server with the Flask package (http://flask.pocoo.org/).

@inthesunset I set up a skeleton module to implement the feature search: https://github.com/fdac15/OSSFinder/blob/master/webapp/lib/features.py. If we can port your code over to this module, that will help a lot with setting up the application. There are descriptions of each of the funtions in the module file. These can change, but I just wanted to give an example of a potential API for this module.

@audrism I was able to install flask locally on my own machine (Ubuntu 15). However, I wasn't able to get the Flask module installed on da2 (permission error). I tried the command pip install flask as well as pip install -r requirements.txt --upgrade Can you suggest a way to manage this and potentially other module da2?

[audrism]

restarted your container on da2 (audris/ipython-pymongo:v32) with flask installed. Please check if it is workingg.

[alexklibisz]

@audrism Flask is working now, thank you. I've added a localforward via SSH so that I can access the port from my browser. This works fine as well.

We will still need at least one more package: Gensim (http://radimrehurek.com/gensim/simserver.html). We may need some more as we continue. Would it be possible to use virtualenv (http://docs.python-guide.org/en/latest/dev/virtualenvs/) to allow us to install these without having to ask you each time?

Thanks, Alex

[audrism]

Restarted for you with simserver/virtualenvs, if it works will restart for all.

Two reasons why I'd like to update containers: 1) if your container is restarted, all the setup you did via virtualenv goes away unless it is in your home dir 2) you can not share that with others

So once you are happy with the setup you create with virtualenv, let me know what packages are needed and I'll set up the container.

[alexklibisz]

@audrism I see that you installed virtualenv, but I can't seem to get it to activate.

Here is my sequence of steps:

% aklibisz@eeabba3d0a52 ~/OSSFinder/webapp
 % virtualenv venv
Using base prefix '/usr'
New python executable in venv/bin/python3
Also creating executable in venv/bin/python
Installing setuptools, pip, wheel...done.  

% source venv/bin/activate  

% pip install simserver
Collecting simserver
  Using cached simserver-0.1.4.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 20, in <module>
      File "/tmp/pip-build-fmk31a3m/simserver/setup.py", line 19, in <module>
        import ez_setup
      File "/tmp/pip-build-fmk31a3m/simserver/ez_setup.py", line 106
        except pkg_resources.VersionConflict, e:
                                            ^
    SyntaxError: invalid syntax

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-fmk31a3m/simserver

There seem to be two issues:

1. activating the virtualenv doesn't actually activate it. It should show (venv) on the console but doesn't, see here: http://docs.python-guide.org/en/latest/dev/virtualenvs/.
2. pip install seems to be attempting to install to global python packages. But when I import simserver (from simserver import SessionServer) and run my script, it can't find the module:

% python3 server.py
Traceback (most recent call last):
  File "server.py", line 3, in <module>
    from simserver import SessionServer
ImportError: No module named 'simserver'

[audrism]

simserver does not appear to work with python3 You may wan to fix it: rm -rf venv virtualenv venv source venv/bin/activate
virtualenv -p /usr/bin/python2 venv pip2 install simserver

[alexklibisz]

Ok I will try that.

[inthesunset]

I've built the search module in features.py

[alexklibisz]

@inthesunset Do you have the tar file with all of the readme files available somewhere? How large is it? Maybe you could upload it to the shared google drive folder

[alexklibisz]

@inthesunset I see the tar file now. I overlooked it the first time.

[alexklibisz]

I got the search and repo selection input setup in the webapp. We still need to write the code that does the search on the server, but we're getting data back and forth between the server and browser.

@inthesunset I tried to implement the package you wrote, but I'm getting an error about sqlite. I'm not sure what the issue is here, but I figured you have probably seen it at some point:

2015-11-19 06:37:50,854 : INFO : stable index pointer not found or invalid; starting from ./.service/a
2015-11-19 06:37:50,855 : INFO : loading SaveLoad object from ./.service/a/index_fresh
2015-11-19 06:37:50,856 : INFO : loading SaveLoad object from ./.service/a/index_opt
2015-11-19 06:37:50,856 : INFO : loading SimModel object from ./.service/a/model
2015-11-19 06:37:50,856 : INFO : opening Sqlite table 'unnamed' in ./.service/a/payload
2015-11-19 06:37:50,861 : INFO : opening Sqlite table 'unnamed' in /home/aklibisz/OSSFinder/webapp/sqldict92e632
2015-11-19 06:37:50,863 : INFO : loaded SimServer(loc='./.service/a', fresh=None, opt=None, model=None, buffer=SqliteDict(/home/aklibisz/OSSFinder/webapp/sqldict92e632))
Traceback (most recent call last):
  File "./server.py", line 12, in <module>
    Features.service_initialization('./.service', './.readmes')
  File "/home/aklibisz/OSSFinder/webapp/lib/features.py", line 23, in service_initialization
    if 'model'  not in os.listdir(directory_path+'a/') :
OSError: [Errno 2] No such file or directory: './.servicea/'
2015-11-19 06:37:51,877 : INFO : deleting /home/aklibisz/OSSFinder/webapp/sqldict92e632
2015-11-19 06:37:51,877 : ERROR : failed to delete /home/aklibisz/OSSFinder/webapp/sqldict92e632
Traceback (most recent call last):
  File "/home/aklibisz/OSSFinder/webapp/venv/local/lib/python2.7/site-packages/sqlitedict.py", line 283, in terminate
    os.remove(self.filename)
OSError: [Errno 2] No such file or directory: '/home/aklibisz/OSSFinder/webapp/sqldict92e632'

The code that implements this is in webapp/server.py.

See the webapp/README.md file to execute the scripts for setting up python virtualenv and extracting the repo readmes. The directory where the readmes are extracted to is ignored with gitignore. I've also ignored the directories webapp/.service and webapp/.readmes for use with the search model.

See if you can setup and launch the model from server.py. Thanks!

[inthesunset]

@alexklibisz For now, we don't need to care about the sqlite problem(maybe a problem inside this 'simserver' method)

I don't understand the parameter you give in this function: 'Features.service_initialization('./.service', './.readmes'). Instead I use in this way: Features.service_initialization('./model_directory', './readme-files'). model_directory is a folder where to store this model. And load_readme_files in included in this function. No more need to call seperately.

[inthesunset]

And also we need to make this folder(./model_directory) by code if it doesn't exist.

[alexklibisz]

Ok I will try this in about an hour

[alexklibisz]

It looks like it is loading correctly. I'll now check if the search works.

[alexklibisz]

@inthesunset, @MBenkhayal

Here is what's left to do:

• Refining the webapp layout.
• Implementing the match_repos() function: https://github.com/fdac15/OSSFinder/blob/master/webapp/lib/relationships.py
• Implementing the aggregate relationships script: https://github.com/fdac15/OSSFinder/blob/master/relationship-calculations/aggregate-relationships.py
• Paper
• Presentation

[alexklibisz]

I've made some progress on the webapp. A high level description:

• the feature search and the recommendation search endpoints are implemented.
• feature search calls the simserver model and returns the results.
• you can click "check" to quickly see the number of repos that simserver matched to your repo.
• the recommendation search calls match_repos(), but right now it just returns the same repos as simserver.

So the things remaining:

• implement the match_repos() function and make sure that the results get sent back and displayed correctly.

A screenshot:

[alexklibisz]

@inthesunset @MBenkhayal

Whoever does the match_repos() function, can you try to make the returned results in this format:

[{ full_name: 'user/repo', count: 12345 }]

This will make it simple to setup in the browser app. Thanks

[inthesunset]

@MBenkhayal @alexklibisz I've implemented the match_repos() function. But still I have to wait for the completion of collection rel_aggregate.

By the way, you can do the calculation in each document in the process of creating this collection, or you can leave it to match_repos() function. (calculation is implemented in this function)@MBenkhayal

One question is for now, I still can't deploy this server on my computer. Probably I need to tell Dr. Mockus to install something in my docker container. I can't test my function for these two reasons above.

Thanks

[MBenkhayal]

@inthesunset I'll let you know once I've finished running populating rel_aggregate, there are still a few other scripts running for relationships, so right now I'm still waiting on them to finish.

[alexklibisz]

@inthesunset You probably need to have python virtualenv installed.

[alexklibisz]

@audrism I am trying to install the packages necessary to launch the webapp, but it has been hanging on the scipy installation for 30+ min.

Can you maybe restart my docker container? Or can you think of any other reason that it would not work?

I am running this script to setup virtualenv with all of the dependencies: https://github.com/fdac15/OSSFinder/blob/master/webapp/setup.sh

[alexklibisz]

@audrism for clarity, it is getting hung up here. It has been stuck twice now for ~20min each. Never happened before.

Using cached boto-2.38.0-py2.py3-none-any.whl
Collecting bz2file==0.98 (from -r requirements.txt (line 2))
Collecting Flask==0.10.1 (from -r requirements.txt (line 3))
Collecting Flask-Cors==2.1.2 (from -r requirements.txt (line 4))
Collecting gensim==0.12.3 (from -r requirements.txt (line 5))
Collecting httpretty==0.8.6 (from -r requirements.txt (line 6))
  Using cached httpretty-0.8.6-py2-none-any.whl
Collecting itsdangerous==0.24 (from -r requirements.txt (line 7))
Collecting Jinja2==2.8 (from -r requirements.txt (line 8))
  Using cached Jinja2-2.8-py2.py3-none-any.whl
Collecting MarkupSafe==0.23 (from -r requirements.txt (line 9))
Collecting numpy==1.10.1 (from -r requirements.txt (line 10))
Collecting pymongo==3.1 (from -r requirements.txt (line 11))
Collecting Pyro4==4.40 (from -r requirements.txt (line 12))
  Using cached Pyro4-4.40-py2.py3-none-any.whl
Collecting requests==2.8.1 (from -r requirements.txt (line 13))
  Using cached requests-2.8.1-py2.py3-none-any.whl
Collecting scipy==0.16.1 (from -r requirements.txt (line 14))
  Using cached scipy-0.16.1.tar.gz
Collecting serpent==1.11 (from -r requirements.txt (line 15))
  Using cached serpent-1.11-py2.py3-none-any.whl
Collecting simserver==0.1.4 (from -r requirements.txt (line 16))
Collecting six==1.10.0 (from -r requirements.txt (line 17))
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting smart-open==1.3.0 (from -r requirements.txt (line 18))
  Using cached smart_open-1.3.0-py2-none-any.whl
Collecting sqlitedict==1.4.0 (from -r requirements.txt (line 19))
  Using cached sqlitedict-1.4.0-py2-none-any.whl
Collecting Werkzeug==0.11.2 (from -r requirements.txt (line 20))
  Using cached Werkzeug-0.11.2-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): wheel==0.24.0 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 21))
Building wheels for collected packages: scipy
  Running setup.py bdist_wheel for scipy

[alexklibisz]

@audrism It finally ends with this message:

Collecting boto==2.38.0 (from -r requirements.txt (line 1))
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached boto-2.38.0-py2.py3-none-any.whl
Collecting bz2file==0.98 (from -r requirements.txt (line 2))
Collecting Flask==0.10.1 (from -r requirements.txt (line 3))
Collecting Flask-Cors==2.1.2 (from -r requirements.txt (line 4))
Collecting gensim==0.12.3 (from -r requirements.txt (line 5))
Collecting httpretty==0.8.6 (from -r requirements.txt (line 6))
  Using cached httpretty-0.8.6-py2-none-any.whl
Collecting itsdangerous==0.24 (from -r requirements.txt (line 7))
Collecting Jinja2==2.8 (from -r requirements.txt (line 8))
  Using cached Jinja2-2.8-py2.py3-none-any.whl
Collecting MarkupSafe==0.23 (from -r requirements.txt (line 9))
Collecting numpy==1.10.1 (from -r requirements.txt (line 10))
Collecting pymongo==3.1 (from -r requirements.txt (line 11))
Collecting Pyro4==4.40 (from -r requirements.txt (line 12))
  Using cached Pyro4-4.40-py2.py3-none-any.whl
Collecting requests==2.8.1 (from -r requirements.txt (line 13))
  Using cached requests-2.8.1-py2.py3-none-any.whl
Collecting scipy==0.16.1 (from -r requirements.txt (line 14))
  Using cached scipy-0.16.1.tar.gz
Collecting serpent==1.11 (from -r requirements.txt (line 15))
  Using cached serpent-1.11-py2.py3-none-any.whl
Collecting simserver==0.1.4 (from -r requirements.txt (line 16))
Collecting six==1.10.0 (from -r requirements.txt (line 17))
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting smart-open==1.3.0 (from -r requirements.txt (line 18))
  Using cached smart_open-1.3.0-py2-none-any.whl
Collecting sqlitedict==1.4.0 (from -r requirements.txt (line 19))
  Using cached sqlitedict-1.4.0-py2-none-any.whl
Collecting Werkzeug==0.11.2 (from -r requirements.txt (line 20))
  Using cached Werkzeug-0.11.2-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): wheel==0.24.0 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 21))
Building wheels for collected packages: scipy
  Running setup.py bdist_wheel for scipy
  Stored in directory: /home/aklibisz/.cache/pip/wheels/84/d5/73/ad1d93891b39dcc3197feb26d56ae16eb004110379c215184b
Successfully built scipy
Installing collected packages: boto, bz2file, itsdangerous, Werkzeug, MarkupSafe, Jinja2, Flask, six, Flask-Cors, scipy, httpretty, requests, smart-open, numpy, gensim, pymongo, serpent, Pyro4, sqlitedict, simserver
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 223, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 299, in run
    root=options.root_path,
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 646, in install
    **kwargs
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 813, in install
    self.move_wheel_files(self.source_dir, root=root)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 1008, in move_wheel_files
    isolated=self.isolated,
  File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 339, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 310, in clobber
    ensure_dir(destdir)
  File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 70, in ensure_dir
    os.makedirs(path)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/boto-2.38.0.dist-info'

However it looks like it is still missing modules:

./server.py
Traceback (most recent call last):
  File "./server.py", line 2, in <module>
    from flask import Flask, url_for, jsonify, request
ImportError: No module named flask

[alexklibisz]

@audrism I was able to get it to run after uninstalling all of the python2 modules with the pip2 uninstall command. Strange, but it works now.

[audrism]

glad it worked. I have created a docker container with the requirements.tx, if you'd like I can start it for you.

[inthesunset]

@alexklibisz I've done these tests below: These are the test results of five keywords:

<1> First I input a keyword, do a search feature. <2> Second I do a query to rel_aggragate collection, get all entries that are relevant with these feature list. <3> Then do a match_repo between these two lists(features and user_repos) <4> The result returned is the max possible count that can return for the keyword search.(just assume that user has tagged all the relevant repos) 1. responsive [{'full_name': 'scottjehl/picturefill', 'count': 28}, {'full_name': 'zurb/foundation', 'count': 17}, {'full_name': 'viljamis/TinyNav.js', 'count': 15}, {'full_name': 'dhgamache/Skeleton', 'count': 13}, {'full_name': 'robertpiira/ingrid', 'count': 13}, {'full_name': 'rweng/jquery-datatables-rails', 'count': 13}, {'full_name': 'dhg/Skeleton', 'count': 4}, {'full_name': 'johnpolacek/Responsivator', 'count': 2}, {'full_name': 'codrops/Elastislide', 'count': 2}, {'full_name': 'yahoo/pure', 'count': 2}, {'full_name': 'viljamis/responsive-nav.js', 'count': 2}, {'full_name': 'sparkbox/Responsive-Retrofitting', 'count': 2}, {'full_name': 'mattkersley/Responsive-Design-Testing', 'count': 2}, {'full_name': 'codrops/FullscreenLayoutPageTransitions', 'count': 2}, {'full_name': 'wycks/WP-Skeleton-Theme', 'count': 1}, {'full_name': 'filamentgroup/Responsive-Images', 'count': 1}, {'full_name': 'usmanhalalit/charisma', 'count': 1}, {'full_name': 'LeonidasEsteban/curso-responsive-design', 'count': 0}, {'full_name': 'zeljkoprsa/waterlee-boilerplate', 'count': 0}, {'full_name': 'adamdbradley/foresight.js', 'count': 0}, {'full_name': 'aino/galleria', 'count': 0}, {'full_name': 'SaraSoueidan/s-gallery', 'count': 0}, {'full_name': 'joshje/Responsive-Enhance', 'count': 0}, {'full_name': 'thatcoolguy/gridless-boilerplate', 'count': 0}, {'full_name': 'picnicss/picnic', 'count': 0}, {'full_name': 'codrops/SlitSlider', 'count': 0}, {'full_name': 'MattWilcox/Adaptive-Images', 'count': 0}, {'full_name': 'interactiv4/b-responsive', 'count': 0}, {'full_name': 'jasoncostello/slate', 'count': 0}, {'full_name': 'freqdec/slabText', 'count': 0}, {'full_name': 'kumailht/responsive-elements', 'count': 0}, {'full_name': 'scottjehl/Respond', 'count': 0}, {'full_name': 'gfidente/pelican-svbhack', 'count': 0}, {'full_name': 'bradfrost/this-is-responsive', 'count': 0}, {'full_name': 'nikki/liteAccordion', 'count': 0}, {'full_name': 'davatron5000/Foldy960', 'count': 0}, {'full_name': 'mailchimp/Email-Blueprints', 'count': 0}, {'full_name': 'stevenwanderski/bxslider', 'count': 0}, {'full_name': 'matthewhartman/base', 'count': 0}, {'full_name': 'headjs/headjs', 'count': 0}, {'full_name': 'designbyfront/The-Goldilocks-Approach', 'count': 0}, {'full_name': 'codrops/GammaGallery', 'count': 0}, {'full_name': 'Abban/jQuery-Picture', 'count': 0}, {'full_name': 'madoublet/respond', 'count': 0}, {'full_name': 'parsimony/parsimony', 'count': 0}, {'full_name': 'adamdbradley/focal-point', 'count': 0}, {'full_name': 'dfcb/Responsivator', 'count': 0}, {'full_name': 'north/north', 'count': 0}, {'full_name': 'marcj/css-element-queries', 'count': 0}, {'full_name': 'tophermade/sprInvoice', 'count': 0}, {'full_name': 'spsaucier/App-Folders', 'count': 0}, {'full_name': 'jbrewer/Responsive-Measure', 'count': 0}, {'full_name': 'Wilto/Dynamic-Carousel', 'count': 0}] 2. MVC [{'full_name': 'SignalR/SignalR', 'count': 15}, {'full_name': 'elabs/serenade.js', 'count': 14}, {'full_name': 'SpringSource/spring-test-mvc', 'count': 13}, {'full_name': 'TroyGoode/PagedList', 'count': 13}, {'full_name': 'JeremySkinner/FluentValidation', 'count': 2}, {'full_name': 'thinktecture/Thinktecture.IdentityServer.v2', 'count': 1}, {'full_name': 'erichexter/twitter.bootstrap.mvc', 'count': 1}, {'full_name': 'yariv/erlyweb', 'count': 1}, {'full_name': 'n2cms/n2cms', 'count': 0}, {'full_name': 'ninject/ninject', 'count': 0}, {'full_name': 'zendframework/ZendSkeletonModule', 'count': 0}, {'full_name': 'smsohan/MvcMailer', 'count': 0}, {'full_name': 'spring-projects/spring-test-mvc', 'count': 0}, {'full_name': 'jgauffin/griffin.mvccontrib', 'count': 0}, {'full_name': 'martijnboland/MvcPaging', 'count': 0}, {'full_name': 'priyatam/springmvc-bootstrap-showcase', 'count': 0}, {'full_name': 'ninject/ninject.web.mvc', 'count': 0}, {'full_name': 'michealharker/springseed', 'count': 0}, {'full_name': 'CaffeinatedCode/springseed', 'count': 0}, {'full_name': 'fellowshiptech/f1-pco', 'count': 0}, {'full_name': 'andrewdavey/postal', 'count': 0}, {'full_name': 'springseednotes/springseed', 'count': 0}, {'full_name': 'mono/xsp', 'count': 0}, {'full_name': 'byhestia/springseed', 'count': 0}, {'full_name': 'jakubgarfield/Bonobo-Git-Server', 'count': 0}, {'full_name': 'ServiceStackApps/SocialBootstrapApi', 'count': 0}, {'full_name': 'srkirkland/DataAnnotationsExtensions', 'count': 0}, {'full_name': 'donal/mvc', 'count': 0}, {'full_name': 'UnionOfRAD/manual', 'count': 0}, {'full_name': 'mccalltd/AttributeRouting', 'count': 0}, {'full_name': 'DevGrow/jQuery-Mobile-PHP-MVC', 'count': 0}, {'full_name': 'ysc/APDPlat', 'count': 0}, {'full_name': 'priyatam/springmvc-bootstrap', 'count': 0}, {'full_name': 'stoneniqiu/FindLover', 'count': 0}, {'full_name': 'cliftonc/express-mvc-bootstrap', 'count': 0}, {'full_name': '1602/express-on-railway', 'count': 0}, {'full_name': 'spring-projects/spring-mvc-showcase', 'count': 0}, {'full_name': 'SimpleAuthentication/SimpleAuthentication', 'count': 0}, {'full_name': 'SpringSource/spring-mvc-showcase', 'count': 0}, {'full_name': 'AndreyAkinshin/knockout-mvc', 'count': 0}, {'full_name': 'earldouglas/barebones-spring-mvc', 'count': 0}, {'full_name': 'arturadib/agility', 'count': 0}, {'full_name': 'JeremySkinner/git-dot-aspx', 'count': 0}, {'full_name': 'archimag/restas', 'count': 0}, {'full_name': 'FasterXML/jackson-datatype-hibernate', 'count': 0}, {'full_name': 'ChrisFulstow/NBlog', 'count': 0}, {'full_name': 'mdolon/jQuery-Mobile-PHP-MVC', 'count': 0}, {'full_name': 'FrancisShanahan/SenchaTouch2MVCHelloworld', 'count': 0}] 3. mapping [{'full_name': 'openlayers/openlayers', 'count': 13}, {'full_name': 'route-me/route-me', 'count': 13}, {'full_name': 'HPNeo/gmaps', 'count': 2}, {'full_name': 'mbostock/topojson', 'count': 2}, {'full_name': 'mapstraction/mxn', 'count': 2}, {'full_name': 'Atlantic18/DoctrineExtensions', 'count': 2}, {'full_name': 'tschneidereit/SwiftSuspenders', 'count': 0}, {'full_name': 'kartograph/kartograph.js', 'count': 0}, {'full_name': 'sitemesh/sitemesh2', 'count': 0}, {'full_name': 'stamen/maps.stamen.com', 'count': 0}, {'full_name': 'SteveSanderson/knockout.mapping', 'count': 0}, {'full_name': 'vasile/transit-map', 'count': 0}, {'full_name': 'Leaflet/Leaflet.draw', 'count': 0}, {'full_name': 'vigetlabs/jmapping', 'count': 0}, {'full_name': 'amitp/mapgen2', 'count': 0}, {'full_name': 'SteveSanderson/knockout', 'count': 0}, {'full_name': 'flatiron/plates', 'count': 0}, {'full_name': 'Leaflet/Leaflet.markercluster', 'count': 0}, {'full_name': 'knockout/knockout', 'count': 0}, {'full_name': 'jerieljan/ECMI-App', 'count': 0}, {'full_name': 'fridek/gmap', 'count': 0}, {'full_name': 'osmandapp/Osmand', 'count': 0}, {'full_name': 'mapbox/dc-properties-template', 'count': 0}, {'full_name': 'pageman/ECMI-App', 'count': 0}] 4. graphing [{'full_name': 'sensu/sensu-plugin', 'count': 26}, {'full_name': 'codahale/metrics', 'count': 15}, {'full_name': 'square/cubism', 'count': 15}, {'full_name': 'mnutt/hummingbird', 'count': 15}, {'full_name': 'tmm1/graphite', 'count': 14}, {'full_name': 'kytrinyx/etsy', 'count': 14}, {'full_name': 'paperlesspost/graphiti', 'count': 13}, {'full_name': 'joewalnes/smoothie', 'count': 13}, {'full_name': 'randym/axlsx', 'count': 13}, {'full_name': 'talklittle/reddit-is-fun', 'count': 2}, {'full_name': 'sivy/pystatsd', 'count': 2}, {'full_name': 'shutterstock/rickshaw', 'count': 2}, {'full_name': 'nnnick/Chart.js', 'count': 2}, {'full_name': 'gak/pygooglechart', 'count': 2}, {'full_name': 'grafana/grafana', 'count': 1}, {'full_name': 'heavywater/chef-graphite', 'count': 1}, {'full_name': 'monigusto/vagrant-monigusto', 'count': 1}, {'full_name': 'jjoe64/GraphView', 'count': 1}, {'full_name': 'deepakjois/gchartrb', 'count': 0}, {'full_name': 'johnewart/ruby-metrics', 'count': 0}, {'full_name': 'ripienaar/gdash', 'count': 0}, {'full_name': 'luddep/LPKit', 'count': 0}, {'full_name': 'indygreg/collectd-carbon', 'count': 0}, {'full_name': 'novus/nvd3', 'count': 0}, {'full_name': 'leetcoders/LeetCode', 'count': 0}, {'full_name': 'ripienaar/graphite-graph-dsl', 'count': 0}, {'full_name': 'Cue/scales', 'count': 0}, {'full_name': 'HackPlan/AndroidCharts', 'count': 0}, {'full_name': 'etsy/statsd', 'count': 0}, {'full_name': 'Shopify/statsd-instrument', 'count': 0}, {'full_name': 'paulasmuth/fnordmetric', 'count': 0}, {'full_name': 'davidpadbury/node-highcharts', 'count': 0}, {'full_name': 'taitems/jQuery.Gantt', 'count': 0}, {'full_name': 'umaar/Chart.js', 'count': 0}, {'full_name': 'sivy/node-statsd', 'count': 0}, {'full_name': 'michelson/lazy_high_charts', 'count': 0}, {'full_name': 'sivy/py-statsd', 'count': 0}, {'full_name': 'kngsc/jquery-visualize', 'count': 0}, {'full_name': 'topfunky/gruff', 'count': 0}, {'full_name': 'mattetti/googlecharts', 'count': 0}, {'full_name': 'alexyoung/ico', 'count': 0}, {'full_name': 'gnemoug/distribute_crawler', 'count': 0}, {'full_name': 'winston/google_visualr', 'count': 0}, {'full_name': 'briannesbitt/Carbon', 'count': 0}, {'full_name': 'Bantik/seer', 'count': 0}, {'full_name': 'mbielanczuk/jQuery.Gantt', 'count': 0}, {'full_name': 'noahhl/batsd', 'count': 0}, {'full_name': 'RJ/estatsd', 'count': 0}, {'full_name': 'jondot/graphene', 'count': 0}, {'full_name': 'HumbleSoftware/Flotr2', 'count': 0}, {'full_name': 'jmxtrans/jmxtrans', 'count': 0}, {'full_name': 'tim-group/java-statsd-client', 'count': 0}, {'full_name': 'reinh/statsd', 'count': 0}, {'full_name': 'chinmaymk/angular-charts', 'count': 0}, {'full_name': 'highslide-software/highcharts.com', 'count': 0}, {'full_name': 'graphite-project/carbon', 'count': 0}, {'full_name': 'paulrouget/html5dashboard', 'count': 0}, {'full_name': 'etsy/logster', 'count': 0}, {'full_name': 'obfuscurity/tasseo', 'count': 0}] 5. mobile [{'full_name': 'rhomobile/rhodes', 'count': 156}, {'full_name': 'forcedotcom/SalesforceMobileSDK-Android', 'count': 52}, {'full_name': 'commadelimited/jQuery-Mobile-Icon-Pack', 'count': 15}, {'full_name': 'h5bp/movethewebforward', 'count': 15}, {'full_name': 'commadelimited/jQuery-Mobile-Boilerplate', 'count': 13}, {'full_name': 'ucla/mwf', 'count': 13}, {'full_name': 'SpringSource/spring-mobile-samples', 'count': 13}, {'full_name': 'SpringSource/spring-mobile', 'count': 13}, {'full_name': 'taitems/iOS-Inspired-jQuery-Mobile-Theme', 'count': 13}, {'full_name': 'spotify/apps-tutorial', 'count': 2}, {'full_name': 'jeromeetienne/jquery-mobile-960', 'count': 2}, {'full_name': 'webglearth/webglearth', 'count': 2}, {'full_name': 'sebarmeli/JS-Redirection-Mobile-Site', 'count': 2}, {'full_name': 'opitzconsulting/jquery-mobile-angular-adapter', 'count': 2}, {'full_name': 'cheeaun/hackerweb', 'count': 2}, {'full_name': 'jpmobile/jpmobile', 'count': 2}, {'full_name': 'weightshift/The-Personal-Page', 'count': 1}, {'full_name': 'pixedelic/Camera', 'count': 1}, {'full_name': 'wycks/WP-Skeleton-Theme', 'count': 1}, {'full_name': 'tastejs/PropertyCross', 'count': 1}, {'full_name': 'Countly/countly-server', 'count': 1}, {'full_name': 'mindmup/bootstrap-wysiwyg', 'count': 1}, {'full_name': 'ansca/Ghosts-vs.-Monsters', 'count': 0}, {'full_name': 'TapQuo/lungo.thirdparties', 'count': 0}, {'full_name': 'sustainablepace/mobipick', 'count': 0}, {'full_name': 'rapidftr/RapidFTR', 'count': 0}, {'full_name': 'Redth/ZXing.Net.Mobile', 'count': 0}, {'full_name': 'benlangfeld/mobile-fu', 'count': 0}, {'full_name': 'silverstripe/silverstripe-mobile', 'count': 0}, {'full_name': 'fellowshiptech/slablet', 'count': 0}, {'full_name': 'HubSpot/vex', 'count': 0}, {'full_name': 'ccoenraets/backbone-jquerymobile', 'count': 0}, {'full_name': 'JoshBarr/on-media-query', 'count': 0}, {'full_name': 'dmolsen/MIT-Mobile-Web', 'count': 0}, {'full_name': 'blackberry/jQueryMobile-BB10-Theme', 'count': 0}, {'full_name': 'triceam/app-UI', 'count': 0}, {'full_name': 'BoilerplateMVC/Backbone-Require-Boilerplate', 'count': 0}, {'full_name': 'brendanlim/mobile-fu', 'count': 0}, {'full_name': 'ferrante/otwartasiec', 'count': 0}, {'full_name': 'CyanogenMod/android_device_htc_msm7x30-common', 'count': 0}, {'full_name': 'Diablo-D3/DiabloMiner', 'count': 0}, {'full_name': 'libmx3/mx3', 'count': 0}, {'full_name': 'zhaoyan/hello-world', 'count': 0}, {'full_name': 'hcatlin/wikimedia-mobile', 'count': 0}, {'full_name': 'raykrueger/ruby-smpp', 'count': 0}, {'full_name': 'serbanghita/Mobile-Detect', 'count': 0}, {'full_name': 'darashi/jpmobile', 'count': 0}, {'full_name': 'h5bp/mobile-boilerplate', 'count': 0}, {'full_name': 'DrivenSoftware/GitCourseDemo', 'count': 0}, {'full_name': 'spazproject/spazcore', 'count': 0}, {'full_name': 'asyraf9/jquerymobile-splitview', 'count': 0}, {'full_name': 'acidb/mobiscroll', 'count': 0}, {'full_name': 'Azure/azure-mobile-services', 'count': 0}, {'full_name': 'manifestinteractive/socketbug', 'count': 0}, {'full_name': 'operasoftware/Emberwind', 'count': 0}, {'full_name': 'DevGrow/jQuery-Mobile-PHP-MVC', 'count': 0}, {'full_name': 'ncannasse/heaps', 'count': 0}, {'full_name': 'gregmuellegger/django-mobile', 'count': 0}, {'full_name': 'couchbaselabs/Grocery-Sync-iOS', 'count': 0}, {'full_name': 'asyraf9/jquery-mobile', 'count': 0}, {'full_name': 'doat/TouchyBP', 'count': 0}, {'full_name': 'filamentgroup/jqm-pagination', 'count': 0}, {'full_name': 'visiongeist/pull-to-refresh-js', 'count': 0}, {'full_name': 'modolabs/Kurogo-Mobile-Web', 'count': 0}, {'full_name': 'couchbaselabs/iOS-Couchbase', 'count': 0}, {'full_name': 'WindowsAzure/azure-mobile-services', 'count': 0}, {'full_name': 'rogchap/Titanium-Google-Analytics', 'count': 0}, {'full_name': 'brendanlim/sms-fu', 'count': 0}, {'full_name': 'cerebrobr/biblioteca', 'count': 0}, {'full_name': 'rapidftr/RapidFTR-Android', 'count': 0}, {'full_name': 'couchbaselabs/TouchDB-iOS', 'count': 0}, {'full_name': 'dmarkow/jquery-mobile-rails', 'count': 0}, {'full_name': 'appcelerator-titans/tweetanium', 'count': 0}, {'full_name': 'mdolon/jQuery-Mobile-PHP-MVC', 'count': 0}] 6. matrix [{'full_name': 'quil/quil', 'count': 16}, {'full_name': 'CreateJS/EaselJS', 'count': 14}, {'full_name': 'paulasmuth/recommendify', 'count': 14}, {'full_name': 'Graylog2/graylog2-server', 'count': 14}, {'full_name': 'nathanmarz/cascalog', 'count': 14}, {'full_name': 'oberhamsi/gamejs', 'count': 13}, {'full_name': 'eclipse/vert.x', 'count': 2}, {'full_name': 'nathanmarz/storm-deploy', 'count': 2}, {'full_name': 'okfn/annotator', 'count': 1}, {'full_name': 'clojure/clojurescript', 'count': 0}, {'full_name': 'nafg/reactive', 'count': 0}, {'full_name': 'rgbdemo/nestk', 'count': 0}, {'full_name': 'mikera/core.matrix', 'count': 0}, {'full_name': 'clojure/core.logic', 'count': 0}, {'full_name': 'CreateJS/EaselJSRenderers', 'count': 0}, {'full_name': 'openannotation/annotator', 'count': 0}, {'full_name': 'visionmedia/move.js', 'count': 0}, {'full_name': 'GameJs/gamejs', 'count': 0}, {'full_name': 'rosejn/overtone', 'count': 0}, {'full_name': 'joelambert/morf', 'count': 0}, {'full_name': 'openembedded/openembedded', 'count': 0}, {'full_name': 'zachstronaut/jquery-animate-css-rotate-scale', 'count': 0}, {'full_name': 'heygrady/transform', 'count': 0}]

[alexklibisz]

@inthesunset So what would be some concrete query/repos I've used before combinations to use for the demo?

Like if we put "responsive" in the search query, what should we put in the "repos I've used before" in order to have results with points?

[inthesunset]

1: responsive gre/slider.js jscs-dev/node-jscs rgrove/rawgit andrewrk/mineflayer dompdf/dompdf digitalfruit/limejs sorich87/bootstrap-tour scottjehl/iOS-Orientationchange-Fix gma/nesta zachstronaut/jquery-animate-css-rotate-scale svenfuchs/i18n-active_record paulrouget/html5dashboard zurb/orbit jlong/sass-twitter-bootstrap rasmusbergpalm/DeepLearnToolbox benedmunds/CodeIgniter-Ion-Auth viljamis/ResponsiveSlides.js

2: MVC quartznet/quartznet SaltwaterC/aws2js sq/JSIL Cilph/RemoteTech2 restsharp/RestSharp augustl/net-http-cheat-sheet TroyGoode/MembershipStarterKit videlalvaro/gifsockets

3: mapping oyvindkinsey/easyXDM twonds/punjab angular-ui/angular-google-maps etsy/statsd mgood/flask-debugtoolbar morrisjs/morris.js

4:graphing quirkey/resque-status milohuang/reverie qoobaa/s3 oesmith/morris.js graphite-project/graphite-web opscode/bento mbostock/protovis groovydev/twitter-bootstrap-grails-plugin mranney/node_redis flapjack/flapjack shikhar/sshj sitemesh/sitemesh3 opscode-cookbooks/rabbitmq atom/atom schematics/schematics androidquery/androidquery zurb/reveal gilt/rubyXL

5:mobile jquery/jquery-mobile pimcore/pimcore gf3/sandbox taitems/Aristo-jQuery-UI-Theme commadelimited/jQuery-Mobile-Bootstrap-Theme commadelimited/autoComplete.js webcomponents/element-boilerplate rhomobile/rhodes-system-api-samples bshaffer/oauth2-server-php kumarnitin/RedisLive todc/css3-google-buttons wilkerlucio/mongoid_taggable forcedotcom/SalesforceMobileSDK-iOS h5bp/mothereffinganimatedgif SpringSource/spring-android-samples ilektrojohn/creepy yaoli/GSN fbsamples/web-friend-smash-v1 bengardner/uncrustify Benvie/continuum gma/nesta SpringSource/spring-android

6:matrix rodjek/vim-puppet paulasmuth/fnordmetric overtone/emacs-live twitter/scalding ringo/ringojs Graylog2/graylog2-web-interface KKBOX/CompassApp impetus-opensource/Kundera CreateJS/TweenJS

[inthesunset]

@alexklibisz

These are the all relevant repos.( you can select some of them when demoing)