search

cristiprg / BDAPRO.GlobalStateML

This repository contains my solution to the project "Machine learning algorithms with global state" from the BDAPRO class at TU Berlin. (The repo is based on BDAPRO.WS1617)
Apache License 2.0
0 stars 0 forks source link

Spark Job is blocked if it interacts with Redis #11

Open cristiprg opened 7 years ago

cristiprg commented 7 years ago

@jeyhunkarimov Everything seems to be work alright with one remark, if I submit a Spark job with a map transformation that involves my API, the job does nothing until I interrupt it.

I made a video with a super small example where I multiply the identity matrix by (2 * identity matrix). https://youtu.be/yb09CPiPeZw

Do you have any ideas what is the reason nothing happens until I interrupt?

cristiprg commented 7 years ago

@jeyhunkarimov

jeyhunkarimov commented 7 years ago

@cristiprg maybe redis collects all transactions and does all computation in batches? Is there any configuration regarding this?

cristiprg commented 7 years ago

@jeyhunkarimov No, it was because Spark was misconfigured, I solved it shortly after I posted the issue :D. It had just one thread - the one responsible for the input stream.