PPPLDeepLearning / plasma-python

PPPL deep learning disruption prediction package
http://tigress-web.princeton.edu/~alexeys/docs-web/html/
79 stars 43 forks source link

Model builder changes #18

Closed ASvyatkovskiy closed 6 years ago

ASvyatkovskiy commented 6 years ago

The PR implements a set of changes to standardize runner and mpi_runner modules, affects the model builder module.

  1. Compile models in runner and mpi_runner. Do not recompile the model in mpi_runner
  2. Only build NN architecture (layers, activations) in the model builder, add optimizer/loss later (in runners)
  3. Provide more optimizer options, including TFOptimizer

Minor changes:

  1. Do not mask all GPUs via CUDA_VISIBLE_DEVICES for architectures with only 1 GPU per node (OLCF Titan)
  2. Minor changes in the conf: enable learning rate decay, but reduce patience.
  3. Reduce base learning rate as a function of number of workers to restore model convergence