pyspark data manager process

[RuiFilipeCampos]

Simple HTTP server that cycles through a randomized slice of data.

• action: POST /next-big-batch
• returns: 200 text/plain /path/to/folder/with/npy/data

[RuiFilipeCampos]

I'm gonna use a celery worker instead of an http server, results in a simpler setup

[RuiFilipeCampos]

• 1 is the training loop process
• 2 is the celery process that preps data

Flow of process 1

• 1 finishes training current slice
• 1 fetches .npy from redis and deletes it right away from redis
• 1 loads slice into GPU
• 1 frees CPU slice
• 1 drops message in queue with specs for next slice
• repeat

flow of process 2

• 2 finds message in queue
• 2 preps data
• 2 loads it into redis and frees memory

1 will timeout if it doesn't find the file 2 will fail if it sees a file already in there Advantages are, all communication is essentially not handled by me. Redis is trustworthy too. 0 disk memory used

[RuiFilipeCampos]

done