search

shensjw / LSPT-LinkAnalysis

Link Analysis part for search engine project in LSPT course
0 stars 1 forks source link

API Review #3

Open cam626 opened 5 years ago

cam626 commented 5 years ago

I have finished the API for the incoming information and most of the outgoing information but I cannot finish everything until I have the Graph structure to get information from. I have a few questions:

  1. When we get input about links, what format/function do you want me to call to add them to the graph? Currently I have them in my own adjacency list and they are ready to be sent to the graph.
  2. So far I think that it makes the most sense for the API to control the flow of the program since things are supposed to happen when we get input. I think this would work well with the way that each of you have constructed your parts but I would like to know if you feel differently about it.
  3. I need to write and test the portion that deals with the HTTP response from the crawling team but the way that I have my testing set up doesn't send responses (I'm using curl to send POST requests to our server). Could either of you help me set up a test to simulate this? @liushengxuan I was thinking you could help with this.
  4. I have connected to the server space that Goldschmidt got us and cloned our repository there. Once we are finished and merged into master we just need to pull and run it there.
shensjw commented 5 years ago

All graph related functions should be found in Webgraph class addLink() function takes in the url, create a node based on that, and add to the graph. addConnection() takes in two urls, it adds the nodes to the graph if not found. Then it adds the edge into the adj matrix

cam626 commented 5 years ago

I have just pushed another commit on my branch that implements the webgraph into the API. Adding nodes and edges seems to be working fine but I may be using the rank updater wrong because the output to indexing says that all of the ranks are 1. Can one of you take a look? The relevant calls are in listener.cpp.

RioMichael commented 5 years ago

I haven't fully tested the rank updater yet, I will definitely look into it.

RioMichael commented 5 years ago

The updateRank function now should correctly update the pagerank. Previous problems were caused by some incorrect copy and incorrect type, and the problems are fixed. Small scale rank update test was successful.