spark prototype 1

[ohliumliu]

Prototype 1.

get more records from Amazon. use spark to read these records, and perform k-mean. and write the result to a new table.

1. create history table, and populate the table with random data.
2. use spark stream to read history data, and perform classification, and write back to database.
3. frontend javascript, use ajax to send cookie id and url back to RoR.
4. create RoR to handle ajax call.

[ohliumliu]

6cfbd1f set up spark local mode following this

[ohliumliu]

• new model scheme: user_id, session, product, cluster
• predict cluster using existing kmeans model. using spark stream. Constantly updating this model.
• Recommend product based on cluster information.

[ohliumliu]

About using spark streaming to check mysql database https://forums.databricks.com/questions/8736/get-mysqlrdbms-data-using-spark-streaming.html

[ohliumliu]

a tutorial on spark recommendation system with a web app https://www.codementor.io/spark/tutorial/building-a-web-service-with-apache-spark-flask-example-app-part2

[ohliumliu]

Workflow?

• User clicked on a product link
• In the corresponding action, save the link along with user id, redirect to the amazon page
• When the user comes back, run kmeans to find out the user's preference and provide a list of recommendations.

[ohliumliu]

Use spark stream

• user clicks a product link, which has a javascript attached and the javascript is first executed. (this javascript could be relatively easily ported to other sites.)
• this javascript calls a RoR action to add a file to a folder. The files in this folder have user info and product id. If the product is already clustered, don't do anything. In both cases, click_history table is appended.
• spark listens to this folder. When a new file shows up, spark stream uses existing kmeans model to find its cluster and save it.
• based on the most recent 3 clusters the user visited, choose 2 products from each cluster to recommend. Do this in ruby on rails.

[ohliumliu]

See the following examples for spark streaming + machine learning

• http://spark.apache.org/docs/latest/mllib-linear-methods.html#streaming-linear-regression
• https://github.com/apache/spark/blob/master/examples/src/main/python/mllib/streaming_k_means_example.py
• https://databricks.com/blog/2015/07/30/diving-into-apache-spark-streamings-execution-model.html it section: Streaming + MLLib
• mllib is based on RDD, while ml is based on DataFrame. The former has a longer history. The latter supports higher level API such as pipelines. Streaming works with RDD.
• From 2.0, DataFrame also works with streaming, called structured streaming (still alpha). It looks very nice. But machine learning is not officially integrated yet. Maybe 2.1 release (https://issues.apache.org/jira/browse/SPARK-16424?jql=text%20~%20%22structured%20streaming%20ml%22)

[ohliumliu]

Several tables involved

• products: this is the table with all the products. It is periodically updated by calling amazon api. Only a portion of it was used to train the model.
• products_cluster: this contains the result of clustering. When the model was first trained, it populates this table. Later on, more result was added as a result of click_history.
• click_history: it has uid and product id based on user click history. If the product is not clustered, it will trigger spark streaming to do the clustering, and the result is saved to products_cluster table. If the product is already clustered, don't invoke spark streaming.

_In this design, only those clicked by a user will be added to the products_cluster table. Is this what we want? Probably. productscluster also has the original entries used for training the model, and these are probably a large set.

[ohliumliu]

2d3576e For some reason, the ajax call only fires once for each link. This applies to both product link and the hard-coded click_history link on the top right corner. Need to restart server.

After some time, it started to work.

javascript has return false, but href is still followed ?