Closed xyz-scorpio closed 3 weeks ago
There are various granularities of parallelism and vectorization within the SPU. It's not clear what your specific requirements are. How much data do you need to process and what granularity of parallelism do you need?
There are various granularities of parallelism and vectorization within the SPU. It's not clear what your specific requirements are. How much data do you need to process and what granularity of parallelism do you need?
Let us take the SPU workflow as an example. My question is: i) What is the upper bound of data scale that SPU could handle? If I write some code to train a model with tensorflow, on very large datasets (e.g. ~TB) from different parties, how does SPU handle that? Will it do data parallel automaticly? ii) How many resources can a SPU VM take advantage of? Say, if I have 4 AWS EC2s, can one SPU VM take advantage of all 4 EC2s, or just one EC2 instance? And what is the parallel model of SPU VM?
There are various granularities of parallelism and vectorization within the SPU. It's not clear what your specific requirements are. How much data do you need to process and what granularity of parallelism do you need?
Let us take the SPU workflow as an example. My question is: i) What is the upper bound of data scale that SPU could handle? If I write some code to train a model with tensorflow, on very large datasets (e.g. ~TB) from different parties, how does SPU handle that? Will it do data parallel automaticly? ii) How many resources can a SPU VM take advantage of? Say, if I have 4 AWS EC2s, can one SPU VM take advantage of all 4 EC2s, or just one EC2 instance? And what is the parallel model of SPU VM?
Tensorflow
to properly handles such large data through batching or some other techniques.SPU
can only take one machine. Right now, SPU
only supports DLP and ILP on one machine.no activity. close
Feature Request Type
Performance
Have you searched existing issues?
Yes
Is your feature request related to a problem?
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe features you want to add to SPU
Just want to know if SPU supports bigdata / parallel computing, and if yes, in what way did it support such a think. TY.
Describe features you want to add to SPU
A clear and concise description of any alternative solutions or features you've considered.