Implement Bayesian Neural network/ hierarchical Inference using Edward to scale the approach for NPM

[sivaavkd]

User Story

As a Fabric8-analytics IDE user I should be able to get recommendations for the NPM ecosystem within the time frame committed in the SLA.

Acceptance Criteria

Confirmation about whether this approach is scalable and works for the node ecosystem, a working POC of the same.

Description

The current approach uses a Bayesian network(hierarchical Bayesian inference) built on top of pomegranate. The approach however is not scalable as-is and would not scale to support larger ecosystems such as NPM. The idea here is to keep the bayesian inference in place however try to use an approximate Bayesian estimate rather than exact inference while also utilizing the power of Tensorflow and related deep learning libraries(Keras etc.) that are currently SOTA.

Edward is a library for probabilistic modeling built on top of Tensorflow, and we are going to leverage to same to do a quick POC and see whether we are able to scale the Bayesian approach.

Provisional Task List

Using the existing data available for NPM:

• [ ] Try to implement the current hierarchial Bayesian inference model on Edward to check for any performance improvements.
• [ ] Try to use an approximate inference method over the exact inference currently in use and check if the model scales with an acceptable hit in precision.
• [ ] Implement the SOTA hierarchial poisson factorization model using Edward and Tensorflow
• [ ] Think along the lines of a Bayesian neural net to display recommendations

With the availability of "big" data:

• [ ] Collect data for the NPM ecosystem and check if the either of the models are scaling well
• [ ] Come up with a deep probabilistic modelling approach

Each of the tasks related to implementation of the model consists of three phases:

• [ ] Implement the model
• [ ] Check the model for overfitting
• [ ] Calculate the precision of the predictions

[krishnapaparaju]

I am not sure either the title and objective cover the broader goals of this work. Will create a EPIC to cover these objectives..

[rootAvish]

Closing this as it is cloned by #1802 , created a new one so there's no confusion around who to contact in case of any queries around the issue description(since Github currently does not indicate that I'm the one who's written it if the issue has not been created by me). Updated the same in #1738