awslabs / benchmark-ai

Anubis (formerly known as Benchmark AI), measures the goodness of machine learning workloads
Apache License 2.0
16 stars 6 forks source link

Use FSx for data in Anubis #1014

Open haohanchen-aws opened 4 years ago

haohanchen-aws commented 4 years ago

Would it be possible to use the FSx for data fetching/mounting in Anubis? It is doable on EKS, a link for example: https://github.com/aws-samples/mask-rcnn-tensorflow/tree/master/infra/eks

If that is possible, it would highly improve the performance as it takes almost no time to get the data with FSx (currently in anubis for two of our models the data fetching/mounting step would take ~1h).