2022/01/10 Tensorflow hydrology

[rburghol]

Hydro modeling in tensorflow

• hydro-tensorflow-Simulating and Predicting.pdf

Running a tensorflow on deq2

cd /opt/model/tf
# do this next command once in a session to get into the environment
source ./bin/activate
# Now, run this test script:
python3 test1.py

Tensorflow basics

• Learn about Tensorflow and tensors
  • https://www.datacamp.com/community/tutorials/tensorflow-tutorial
  • note: this tutorial is good in that it uses very simple data sets, but does include the use of "tf.Session()" to tell tf to start evauating, which is a deprecated construct, and it appears that now tf evaluates when the expressions are parsed.
  • However, while "sessions" may be deprecated, I think we will want to exaluate our models as "Graphs", which will then allow iterative calling (I think)
  • Tensors and Operations: https://www.tensorflow.org/tutorials/customization/basics
  • Graphs: https://blog.tensorflow.org/2021/11/introducing-tensorflow-gnn.html
  • Graphs (2): https://www.tensorflow.org/guide/intro_to_graphs
  • Leaving a python ENV https://stackoverflow.com/questions/990754/how-to-leave-exit-deactivate-a-python-virtualenv

Installation / Platform Info

• Linux install and testing was fairly painless. See below Linux
• Windows testing thus far incomplete because install via the methods advised by RStudio (which installs from inside RStudio) have been met with network timeouts which have prevented install.

Windows

• Install tensorflow in R on Windows, shell stuff in Gitbash (like editing .bash_profile) - use: https://tensorflow.rstudio.com/installation/
  • Had to manually install miniconda: https://docs.conda.io/en/latest/miniconda.html
    • Put it in /usr/local/bin
    • Edit ~/.bash_profile to add:
    • PATH="$PATH:/usr/local/bin/Miniconda3/Scripts"
    • Do similar adding of PATH to Windows account, see Image 1 Search for "Edit environment variables for your account"
  • conda config --set ssl_verify False
  • library(tensorflow)
  • library(reticulate)
  • conda_create("r-reticulate", conda = "c:\\usr\\local\\bin\\Miniconda3")
  • install_tensorflow(method = "conda", conda = "/usr/local/bin/Miniconda3")
    • Tells the install package where my Miniconda3 install is (because previous ones looked in the normal path)

Image 1 Search for "Edit environment variables for your account".

Linux

• Install tensorflow on ubuntu 20.04 (used https://www.linode.com/docs/guides/how-to-install-tensorflow/)

  • sudo apt install python3.8-ven
  • python3 -m venv --system-site-packages ./tf
  • source ./tf/bin/activate
  • this puts you in a virtual environment so pip changes are not global.
  • to leave type deactivate
  • `pipinstall
  • python -c 'import tensorflow as tf; print(tf.__version__)'
  • Test it note: there will be warnings that ...libcudart.so.11.0: cannot open shared object file: No such file or directory
  • This warning is OK since this is a library that is only used for loading GPU support, and we are on CPU only system
  • To suppress warnings do this (1st command enters a python CMD prompt):
    • python
    • import os;
    • os.environ["TF_CPP_MIN_LOG_LEVEL"]="3";
    • print("TensorFlow version:", tf.__version__)

• Command line linux testing

• file-based testing

  • python test1.py
  • see Code 2 (below) for contents of test1.py Code 1: Command line testing

    
    python
    import os;
    os.environ["TF_CPP_MIN_LOG_LEVEL"]="3";
    print("TensorFlow version:", tf.__version__)

**Code 2:** Execute a file.

import tensorflow as tf import os os.environ["TF_CPP_MIN_LOG_LEVEL"]="3"; print("TensorFlow version:", tf.version)

from tensorflow.keras.layers import Dense, Flatten, Conv2D from tensorflow.keras import Model

mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data() x_train, x_test = x_train / 255.0, x_test / 255.0

Add a channels dimension

x_train = x_train[..., tf.newaxis].astype("float32") x_test = x_test[..., tf.newaxis].astype("float32")

train_ds = tf.data.Dataset.from_tensor_slices( (x_train, y_train)).shuffle(10000).batch(32)

test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)

class MyModel(Model): def init(self): super(MyModel, self).init() self.conv1 = Conv2D(32, 3, activation='relu') self.flatten = Flatten() self.d1 = Dense(128, activation='relu') self.d2 = Dense(10)

def call(self, x): x = self.conv1(x) x = self.flatten(x) x = self.d1(x) return self.d2(x)

Create an instance of the model

model = MyModel()

loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

optimizer = tf.keras.optimizers.Adam()

train_loss = tf.keras.metrics.Mean(name='train_loss') train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')

test_loss = tf.keras.metrics.Mean(name='test_loss') test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='test_accuracy')

@tf.function def train_step(images, labels): with tf.GradientTape() as tape:

training=True is only needed if there are layers with different

# behavior during training versus inference (e.g. Dropout).
predictions = model(images, training=True)
loss = loss_object(labels, predictions)

gradients = tape.gradient(loss, model.trainable_variables) optimizer.apply_gradients(zip(gradients, model.trainable_variables))

train_loss(loss) train_accuracy(labels, predictions)

@tf.function def test_step(images, labels):

training=False is only needed if there are layers with different

behavior during training versus inference (e.g. Dropout).

predictions = model(images, training=False) t_loss = loss_object(labels, predictions)

test_loss(t_loss) test_accuracy(labels, predictions)

EPOCHS = 5

for epoch in range(EPOCHS):

Reset the metrics at the start of the next epoch

train_loss.reset_states() train_accuracy.reset_states() test_loss.reset_states() test_accuracy.reset_states()

for images, labels in train_ds: train_step(images, labels)

for test_images, test_labels in test_ds: test_step(test_images, test_labels)

print( f'Epoch {epoch + 1}, ' f'Loss: {train_loss.result()}, ' f'Accuracy: {train_accuracy.result() 100}, ' f'Test Loss: {test_loss.result()}, ' f'Test Accuracy: {test_accuracy.result() 100}' )

[rburghol]

• Error: Error: Error 127 occurred creating conda environment r-reticulate
• fix from: https://github.com/metrumresearchgroup/mrgsolve/issues/160#issuecomment-258655845
• First had to manually install Rtools (from: https://cran.r-project.org/bin/windows/Rtools/rtools40.html)
  • put it in /usr/local/bin/rtools

    
    path <- Sys.getenv("PATH")
    # original:
    # path <- c("C:\\RBuildTools\\3.3\\bin", "C:\\RBuildTools\\3.3\\gcc-4.6.3\\bin", path)
    # mine
    path <- c("C:\\usr\\local\bin\\rtools40", "C:\\usr\\local\bin\\rtools40\\ucrt64\\bin", path)
    path <- paste(path,collapse=";")
    Sys.setenv(PATH=path)

[rburghol]

• Try this from https://stackoverflow.com/questions/65542831/cannot-install-tensorflow-in-r-could-not-find-a-python-environment-for-path/65543257
• path may be C:\usr\local\bin\Miniconda3

library('reticulate')
reticulate::use_condaenv("/c/usr/local/bin/Miniconda3")
reticulate::conda_create(envname = "r-reticulate")
reticulate::use_condaenv("/c/usr/local/bin/Miniconda3/envs/r-reticulate")

[rburghol]

Install the reticulate environment from Git bash shell: https://github.com/rstudio/keras/issues/1056#issuecomment-646970452

conda create -n R36 python=3.6 numpy keras requests Pillow scipy
conda activate R36
conda install -c conda-forge hdf5=1.10.5 pyyaml==3.12
pip install --force-reinstall --ignore-installed https://files.pythonhosted.org/packages/72/b8/2ef7057c956f1062ffab750a90a6bdcd3de127fb696fb64583c2dfe77aab/tensorflow-2.2.0-cp36-cp36m-win_amd64.whl

Open R:

reticulate::use_python("C:/..../Anaconda3/envs/R36/python.exe", required = TRUE)
reticulate::use_condaenv("R36")
reticulate::py_config() 

[rburghol]

• Command line linux testing
• file-based testing
  • python test1.py
  • see Code 2 (below) for contents of test1.py Code 1: Command line testing

    
    python
    import os;
    os.environ["TF_CPP_MIN_LOG_LEVEL"]="3";
    print("TensorFlow version:", tf.__version__)

**Code 2:** Execute a file.

import tensorflow as tf import os os.environ["TF_CPP_MIN_LOG_LEVEL"]="3"; print("TensorFlow version:", tf.version)

from tensorflow.keras.layers import Dense, Flatten, Conv2D from tensorflow.keras import Model

mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data() x_train, x_test = x_train / 255.0, x_test / 255.0

Add a channels dimension

x_train = x_train[..., tf.newaxis].astype("float32") x_test = x_test[..., tf.newaxis].astype("float32")

train_ds = tf.data.Dataset.from_tensor_slices( (x_train, y_train)).shuffle(10000).batch(32)

test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)

class MyModel(Model): def init(self): super(MyModel, self).init() self.conv1 = Conv2D(32, 3, activation='relu') self.flatten = Flatten() self.d1 = Dense(128, activation='relu') self.d2 = Dense(10)

def call(self, x): x = self.conv1(x) x = self.flatten(x) x = self.d1(x) return self.d2(x)

Create an instance of the model

model = MyModel()

loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

optimizer = tf.keras.optimizers.Adam()

train_loss = tf.keras.metrics.Mean(name='train_loss') train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')

test_loss = tf.keras.metrics.Mean(name='test_loss') test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='test_accuracy')

@tf.function def train_step(images, labels): with tf.GradientTape() as tape:

training=True is only needed if there are layers with different

# behavior during training versus inference (e.g. Dropout).
predictions = model(images, training=True)
loss = loss_object(labels, predictions)

gradients = tape.gradient(loss, model.trainable_variables) optimizer.apply_gradients(zip(gradients, model.trainable_variables))

train_loss(loss) train_accuracy(labels, predictions)

@tf.function def test_step(images, labels):

training=False is only needed if there are layers with different

behavior during training versus inference (e.g. Dropout).

predictions = model(images, training=False) t_loss = loss_object(labels, predictions)

test_loss(t_loss) test_accuracy(labels, predictions)

EPOCHS = 5

for epoch in range(EPOCHS):

Reset the metrics at the start of the next epoch

train_loss.reset_states() train_accuracy.reset_states() test_loss.reset_states() test_accuracy.reset_states()

for images, labels in train_ds: train_step(images, labels)

for test_images, test_labels in test_ds: test_step(test_images, test_labels)

print( f'Epoch {epoch + 1}, ' f'Loss: {train_loss.result()}, ' f'Accuracy: {train_accuracy.result() 100}, ' f'Test Loss: {test_loss.result()}, ' f'Test Accuracy: {test_accuracy.result() 100}' )