theogf / ParticleFlow_Exp

3 stars 1 forks source link

AWS Batch integration #2

Closed fela closed 3 years ago

fela commented 3 years ago

Allows to run arbitrary methods on AWS Batch

fela commented 3 years ago

The juliaaws branch contains some preliminary code to run experiments in AWS, under julia/juliaaws. However I encountered a few issues that have to be resolved. Which is why I went on to try the second approach below. The issues were

Note also that the branch has a few changes to the dependencies that were needed to be able to run on my machine, among other things Julia was updated to 1.5

Alternative approach: using SageMaker Notebooks.

I also tried an alternative approach (first approach in julia/juliaaws/README.md) that is actually simpler. In this approach we simply install Julia using conda. One issue is that again there is no Display thus it might be an issue, potentially using CairoMakie might solve the issue. Also, the Julia version used is 1.1.1 thus we would need to change the dependencies if we want to run using this approach.

theogf commented 3 years ago

We can remove Makie/Plots etc as this won't be necessary to do it on the cluster and can be done locally

fela commented 3 years ago

I added instructions for EC2 here.

Some code is still failing but I'm suspecting it might be some issues with the dependencies in the project.