another approach to use spot instance for persistent training in AWS

seung-lab / znn-release

Multi-core CPU implementation of deep learning for 2D and 3D sliding window convolutional networks (ConvNets).

GNU General Public License v3.0

94 stars 33 forks source link

Open xiuliren opened 8 years ago

xiuliren commented 8 years ago

create a spot fleet or persistent request, it is a kind of persistent spot instance managed by AWS. define the training command by user_data of the spot instance.
read all the data and configuration files in S3, save network to S3. This can be implemented using boto3 and remove the complex dependency of starcluster.
whenever there is an update of saved network file, plot the learning curve online using Plotly.

xiuliren commented 8 years ago

created a pull request to implement this feature: https://github.com/seung-lab/znn-release/pull/94 need some more tests.