snuspl / dolphin

14 stars 2 forks source link

[DOLPHIN-154] Create a Partitioned PS Worker #178

Closed bchocho closed 8 years ago

bchocho commented 8 years ago

Closes #154.

This creates a Partitioned PS Worker.

On local tests, the performance is comparable to ConcurrentParameterWorker.

There is a bug and workaround in this code:

We are observing some lost messages when contexts are immediately closed after the task completes in the example. It appears messages buffered in NCS are not being flushed before context close, but this has to be investigated further. The current workaround in the example is to not close contexts immediately, but this must be resolved before we move to elastically removing contexts.

(This should not be a Worker problem. The ContextStopHandler waits for messages to be flushed from the Worker queue.)

jsjason commented 8 years ago

Checked the code and tests, all look great.