search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

Any success stories using pathos on AWS serverless products? #216

Closed boyanzhu closed 1 year ago

boyanzhu commented 3 years ago

I'm considering using pathos on AWS Lambda or ECS/Fargate I'm hitting snags. For these AWS services there's no support for /dev/shm , etc. I wonder if there's a success example I can stand on.

Thanks

ehoppmann commented 2 years ago

If you use EC2 launch type for ECS instead of Fargate, I can confirm that pathos works. For environments lacking /dev/shm etc my experience is that you can still use multiprocessing, but you have to roll your own with lower level Python interfaces.

mmckerns commented 1 year ago

I'm going to close this. If you have specific questions with regard to the "snags" you may have seen, please either open up a new issue or reopen this issue, whatever makes the most sense.