Roadmap

[AlephNotation]

I've looked it over. Cool project!

Where do would you like to start?

[bmoore20]

Thanks Ty!

Below are some possible paths we can take. I'm thinking that it would be a good idea to start with data augmentation and generating a larger dataset with more variation. But I am open to other starting points too if you have one in particular that you think would be better. Looking forward to touching base on Tuesday to scope out a roadmap!

Pre-processing & Improving Dataset

• Data augmentation
  • increase the amount of samples per class by 1-2 magnitudes
  • chop up images into series of 32x32 pixels
  • linear transformations
  • vertical and horizontal flips
  • change brightness and saturation
  • linear transformations to change the angle
• Limit biases in images (ex. sun glares, shadows, time-stamps)
• Work with drone pictures that aren't as uniform as current dataset
  • current dataset consists of images that are taken by a camera at the same location and height and only contain water
  • handle images that were taken by a drone (ex. bigger area, varying heights)
  • ignore foreign objects (shoreline, docks, trees, etc.)

Machine Learning:

• Replace Keras library with PyTorch library
  • move away from "black box"
  • understand what is going on with the model (math, etc.)
  • experiment with different amounts of layers, wights, and biases to create optimal model

Software Development:

• Re-organize structure of code
  • break independent tasks up into methods
  • create a basic class or library that handles the generic tasks (sorting images into folders, creating accuracy report, etc.)
  • could be used for other image classification problems, would not be specific to HABs project
  • would hide tedious work and would make _HABsClassify.py cleaner and easier to read
• Include functional and unit tests
• Handle various versions of Python and other libraries and dependencies
• Review code and take advantage of Python's shortcuts and libraries
• Fix bug where base image folders have to have at least 3 subfolders at all times, even if some are empty

Environment Set-up:

• Conda
• Pytest
• GPU
• Tox
• Black

[bmoore20]

Documenting important points from Tuesday's meeting (2.23.21)

First steps to take:

1. Create a custom HABs dataset class using PyTorch https://pytorch.org/tutorials/beginner/data_loading_tutorial.html

   class HABDataset(torch.utils.data.Dataset):
            def __init__(self, data_dir: str):
                  self.data_dir = data_dir
   
            def _transform(self, image):
                  image = np.array(image.resize((32, 32))) / 255.0
   
            def __len__(self):
                  # of images in dir
   
            def __getitem__(self, idx):
                 imagePath = pathlib.Path(self.data_dir) + f`path_to_data_{idx}.png`
                 image = Image.open(imagePath)
                 return self._transform(image), target

2. Transition current Keras model to PyTorch
   • Keras model will give us a baseline to compare our new PyTorch model to
3. Reorganize structure of repo
   • look at Papers with Code to see how they organize their repos and to start recognizing common patterns https://paperswithcode.com/ https://github.com/taki0112/UGATIT

      top
           data \
               image_set 1
               image_set 2
           hab \
               dataset.py
               utils
               train
               model \ 
                    model.py
                    blocks \
                         resnet.py

Future & miscellaneous items:

• Remove date & time pixels from bottom of dataset images
• Replace print statements with logs
• Break code up using helper methods
• Delete classify file and merge code into train
• Colab https://colab.research.google.com/notebooks/intro.ipynb
• Docker
• Write README in LaTex
• Think about publishing paper

@AlephNotation feel free to add anything that I may have missed!

[AlephNotation]

@bmoore20 do you want to set up a zoom for next week?

[bmoore20]

@AlephNotation yeah that would be great. Thanks Ty!

I haven't received my new Pratt email yet, so I might have to use my eamoore133@gmail.com email for this time. What days/times work for you?

[bmoore20]

3.8.21 Meeting Notes

• Changes to make to dataset.py:
  • Use torchvision.transforms.compose to create list of Transforms to be passed into dataset
  • Use Path instead of os to handle directory paths
  • Load references to image paths into array and use length of that array for __len__()
  • Move image loading to __getitem__ to save memory
• Make a separate directory for true hold-out test set of images (20% of dataset)
  • Include correct class label in image name

def _get_image(self, idx):
     im_path = self.images[idx]
     image = Image.open(im_path)
     return self._transform(image)

def _make_target(self, idx):
     im_path = Path(self.images[idx])
     _class = im_path.parents[-1]
     if ....

     return target

def __getitem__(self, idx):
     image = self._get_image(idx)
     target = self._make_target(idx)

     return image, target

Thanks for meeting today @AlephNotation !

[bmoore20]

3.23.21 Meeting Notes

• Get rid of large files (i.e. PDF user manuals) -> ~30MB
• New logger will need to be created in every file
• Logger info needs to go after imports, outside of methods
• Use rotating file handlers
• Break data loader in train.py out into a function -> lap (Stephen's term)
• Model needs to be switched to eval() mode before testing -> gets rid of dropouts
• Need to capture seeds in log so we can refer back to good seeds
• Read about Lottery Ticket Hypothesis https://arxiv.org/abs/1803.03635
• Need to do doc strings for my code -> extremely important when you are working with other people on your code (portrays one difference between production code and personal code)
• Should set up the code formatter Black
  • set up pre-commit https://ljvmiranda921.github.io/notebook/2018/06/21/precommits-using-black-and-flake8/
• Should set up PyTest for tests
  • test important things
• Next big step (after addressing things above):
  • Data augmentation
  • Not a good idea to work on data augmentation and improving the model in parallel -> depend on each other and hard to tell what is causing the results
  • Add in more transformations in transformations.py -> random crops, 90 degree flips, etc.
  • This may make us have to make changes to our HABsDataset (ex. __len__ will no longer be able to calculated using the length of the image_paths array because we are making thousands more images out of these images)
  • Should the images in classify mode also be given the same transformations?? -> Both Ty and I will do reading on this and then we will talk about what we found.

[bmoore20]

3.30.21 Meeting Important Points

• Ty and I will both read up on data augmentation and transformation methods
  • collaborate/share sources we find
  • start with: https://arxiv.org/abs/2004.13649 https://discuss.pytorch.org/t/transform-and-image-data-augmentation/71942
• Need to set up testing which will be weird (non-deterministic state problem)
  • will first need to add functionality to save seeds
• Watch lecture on Transformers https://www.youtube.com/watch?v=8BdMObVdr1Y

[bmoore20]

4.6.21 Meeting Important Points

• Links:
  • https://www.lesswrong.com/posts/JZZENevaLzLLeC3zn/predictive-coding-has-been-unified-with-backpropagation
  • https://typer.tiangolo.com/
  • https://fastapi.tiangolo.com/
  • https://stackoverflow.com/questions/51030782/why-do-we-pack-the-sequences-in-pytorch
  • https://www.tensorflow.org/tensorboard/get_started

[bmoore20]

4.13.21 Meeting Important Points

• Links:

  • https://buomsoo-kim.github.io/colab/2020/05/09/Colab-mounting-google-drive.md/
  • https://stackoverflow.com/questions/58785726/google-drive-and-colaboratory-virtual-machine-are-not-syncing-properly

• Logging

  
  # at the top of any file that uses logger
  #### Logging ####
  logging.basicConfig(
  level=logging.INFO, format="[%(asctime)s] PW4k:%(levelname)s - %(name)s - %(message)s", handlers=[logging_utils.ch, logging_utils.fh],
  )

logging.captureWarnings(True) logger = logging.getLogger(name) #################

def name_and_args() -> List[Tuple[str, Any]]: """ Helper function to print args of the function it is called in.

:return: Tuple of arg names and values
"""
caller = inspect.stack()[1][0]
args, _, _, values = inspect.getargvalues(caller)
return [(i, values[i]) for i in args]

- Notes:
  - We both decided that it would be best to make the user always pass in a PyTorch `transforms` object to `HABsDataset`. If `transforms`  is not passed in, then an error is thrown. This is because at the VERY LEAST a Rescale and Crop transform needs to be passed in. Also, ToTensor and Normalize is a good idea too. But we are allowing the user to take full responsibility of what transforms are used, as long as at least one is. 
    - We were originally thinking of doing a conditional in `dataset.py` that performed ToTensor and Normalize transforms if the user passed in _None_ for `transforms`, but we decided that that would be over compensating for the users actions. It would just make it more confusing and unclear on what the program is doing to the user. Trying to "out-smart" the user is not a good idea. 

- To do:
  - Add logging code
    - code that gets the names and arguments -> want to record as much info as possible so we can recreate really good inputs
  - Throw error if there are no transforms
  - Add input parameter for magnitude_increase variable 
  - Merge in code for `feature/capture-seeds` and `feature/typer`
  - Look into how to get my repo into Colab 
     - write script that syncs my code to Google Drive 
     - set up a separate 30 min meeting with Ty for setting up Colab?

[bmoore20]

4.20.21 Meeting Important Points

To do:

• Include device in train.py
• Make the model, optimizer, and # of epochs configurable (arguments)
• Abstract and pull out code for train and test loops
• Load and mount images into Google Drive directory
• Install necessary requirements
• Run the code! (and fix any bugs/errors that pop up)

Notes:

• Use Colab as a place to experiment and rapidly iterate
• When changes are made to my local copy of the repo (PyCharm), push it up to remote GitHub repo ... then re-pull the code into the mounted Google Drive directory (reclone?)
• Always terminate a Colab session when you are done working ... don't leave it running
• touch command creates and names a new blank file

Links:

• https://colab.research.google.com/github/googlecolab/colabtools/blob/master/notebooks/colab-github-demo.ipynb#scrollTo=Rmai0dD30XzL
• https://github.com/drillan/jupyter-black

[bmoore20]

4.27.21 Meeting Notes

• Set up the Black code formatter with the HABs repo
• Discussed how I need to improve the current construction of the training lap
  • break up training into training_lap and validation_lap
  • training_lap -> how you update model weights, optimizer and back propagation
  • validation_lap -> used to see how good the hyperparameters are, no optimizer/back prop
  • calculate the loss for both the training_lap and validation_lap
  • if training loss is down and the validation loss goes up, then that could be a sign of overfitting
  • validation will let us introduce learn rate optimizers

[bmoore20]

@AlephNotation I read a little more on why we need the validation set because it is still fairly new to me. With your explanation you gave me today and what I read, it is starting to make sense. Below is my current understanding of the next steps that I should take. In order to make sure that we are on the same page before I make changes in PR #39, can you please have a quick look to check to see if my logic is correct?

Split the original HABs dataset of images into a total of 3 subsets:

1. training set (60%)
2. validation set (20%)
3. test set (20%)

The training set and validation set will both be inside the epoch loop and passed into either training_lap or validation_lap (which I still have to make). The test set will be passed into my current evaluate method, which run after all of the training is completed and calls the final chosen model. So I am going to keep the current evaluation method I have, but also create two new methods training_lap and validation_lap. The validation_lap does not perform back propagation or have an optimizer, but it does calculate loss. Both methods will return a loss.

Thanks! Also - sorry I was a little discombobulated on the call today

[bmoore20]

5.4.21 Meeting Notes

• In training lap for epoch in range(epoch): log the epoch # and the training loss to a 7-point decimal
• Save the model
• Give batch size to DataLoader object and do for batch in data_loader: in _traininghelper.py
• Configure batch parameter
• Need to fix current model architecture to handle 3 classifications
• Helper function that creates a temp directory and takes a random 20% of the data paths from the test set to make a validation set
• Will eventually need to use randomCrop on images ... maybe use a progressive GAN?
• RUN THE PROGRAM!!

[bmoore20]

5.11.21 Meeting Notes

• Research ways to improve the architecture of my model -> Google the following things ...
  • image classification tasks
  • res nets -> look at architecture and how it works
  • batch norms
  • pooling
  • different types of norms
  • list of layers ... convolutional layers
  • auto-encoder, k-means
• Start messing around with different combinations of various transforms
• Set up Tensor Board https://colab.research.google.com/github/tensorflow/tensorboard/blob/master/docs/tensorboard_in_notebooks.ipynb

  writer.add_scalar("Loss/train", train_loss, epoch)
  writer.add_scalar("Loss/val", val_loss, epoch)

• Add __repr__ to my custom Transformation classes so they can be represented as a string and recorded when "print" is called
• Call __repr__ on Transformation objects and model so we can print and document them for each run in our logger

[bmoore20]

5.18.21 Meeting Notes

• Answered question about using __call__ or forward for transforms
• Helped created new log file each time by adding date and time into the log file name
• Discussed IBM challenge ... https://developer.ibm.com/callforcode/?utm_content=000039JL&utm_term=10008917&p1=PSocial&p2=297608652&p3=142596057&dclid=CKX2lLvixvACFRpLDQod5ToGNw
• Next steps -> research model architecture to use and start trying different combinations of transforms (also increase magnitude of data images by 1000)

[bmoore20]

6.1.21 Meeting Notes

• The problem that I was running into with getting TensorBoard to work in Colab is that I was trying to get it to run on the local host for my laptop. I needed to get TensorBoard to run on the local host of the Colab machine.
• In order to do this I needed to use magic commands (%)
• Magic commands are special commands that help with running code in your notebook
• When you run a command with !, it directly executes a bash command in a subshell
• When you run a command with %, it executes one of the magic commands defined in IPython
• Colab -> built on html and js
• **QUESTION -> How do I know when to use ! (bash) vs % (magic) when doing a command in Colab?
• Make a new notebook that doesn't have a GPU that is running the TensorBoard so you can have it next to the Colab notebook that is actually running the code -> more convenient but not necessary
• This will work because the TensorBoard works from a directory reference ... doesn't have to be in same Colab notebook as code
• Make a tiny function, makeWriter that handles functionality for creating path for writer

  
  from datetime import datetime
  from pathlib import Path

now = datetime.now() now_str = now.strftime("YYYY-mm-dd_HH-MM-SS") RUN_DIR=PATH(TENSORBOARD_DIR) / now_str

https://pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter

- Colab notebook is an instance. When we close out of it, we lose our history. That is why we have to reload in the HABs github directory every time. Therefore, we want to write to the Google Drive dir. 
- ResNets -> skip it
- Replace convolution layers with ResNet layers
- People are moving away from Batch Norms (article from Ty)

[bmoore20]

Looks like someone is already doing a similar program for HABs:

• https://www.noaa.gov/what-is-harmful-algal-bloom (look under tab 4 - Where HABs Happen)
• https://qz.com/755458/algae-bloom-satellite-images-nasa-noaa/

[bmoore20]

6.10.21 Meeting Notes

• Run the program!!
• Try running the program on a different data set ... not my HABs one (ex. ImageNet)
  • Help us know that our model is actually working properly
• It happens all the time that models don't converge
• If model does not work, we will figure it out ... we can do some crazy sh**
• Transfer learning -> Pre-training
  • drastically reduce data
  • get similar data -> train data in that domain -> freeze middle
• Few-shot & One-shot => GOOGLE TERMS
  • how to do training with no data
  • ex) Car crash -> predict rare events

[bmoore20]

No meeting on 6.15.21

[bmoore20]

6.22.21 Meeting Notes

• Talked through error below...

  • a sample: [512, 750, 3] -> [height, width, rgb]
  • a batch of 1: [1, 512, 750, 3]
  • a batch of 8: [8, 512, 750, 3]

• Bigger batch sizes are better -> multiple samples to do back-prop

• Nerds -> pick batches in powers of 2 -> 2, 4, 8, 16, 32...

• ResNet Model and Convolutions

  • each convolution adds a different feature!!!
  • create features from data
  • each block of ResNet enhances each original image
  • each down-stream block added from previous block -> CUMULATIVE!
  • Ex.
  • 1st layer -> edges
  • 2nd layer -> shapes
  • 3rd layer -> dog
  • Downsampling (pooling layer)
  • each level -> pull out more abstract image
  • downsample -> more abstract

• ResNet:

[bmoore20]

6.29.21 Meeting Notes

• Model is not training ... 50% is best (not better than probability)
• Probably due to bad dataset
• If we can use model out of box and avoid bullsh**, then that's best ... doesn't look like that is the case
• Want to make the problem as simple as possible!!
  • Switch to two classes -> reduce classes from 3 to 2
    1. BGA
    2. Non-Algae
  • Will have to slightly change format of HABs Dataset
  • Does our data have enough signal to do it well?
• Also -> Pre-training / Transfer learning
  • trained network does not have enough data -> maybe not at college to get more data or maybe there is no algae in lake
  • take adjacent dataset -> collect bga and pictures of lakes, can you find the difference between the two
  • freeze intermediate layers (except input and outputs)
  • teach model clear & unclear, algae & clear in general -> algae detection model
  • fine tune this algae detection model -> give it very specific data for specific use case
• Train model on clear lake water
  • reconstruct clear water pictures -> Healthy Lake
  • switch problem up -> healthy lake -> no bga
• Semi-supervised image classification -> Papers With Code https://paperswithcode.com/task/semi-supervised-image-classification
  • Mix-Up & Cut-Up
  • BGA Image and Clear Image -> create a new image
  • .9 of orig
  • add 1-.9 of bga image
  • cat/dog example
  • what is new classification? -> pre-training -> make sure model is predicting 50% cat, 50% dog
  • linear combination of image vector
  • pre-train using self-supervised -> take pre-trained model and use for classification task
• **Cut-Mix
• HOW to integrate pre-trained model into model?!
  • in forward
  • pytorch -> freeze weights https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088/2
  • param.requires_grad = False-> call back-prop won't update
• Every class should have 10,000 images!!
• Scientist -> try to keep everything the same as much as possible
• overfit model -> mix-up dataset again -> another full training loop
• CHECKPOINT mode
  • save every 5 epochs incase your program ends in the middle of training, that way you don't loose the progress of the trained weights
  • you can then pick-up where you left off by passing in this model object into training loop
  • load in model weights
  • freeze model -> import model -> different layers
• Might be a good idea for me to get Colab Pro
  • $10 a month
  • faster and bigger GPU
  • can run models for up to 24 hours
• GPU -> vector algebra -> designed todo graphics, happens that nn math is the same as graphics math
• TPU -> made more specifically for ml, f*** graphics
• Training loss is bigger than validation loss because training dataset has WAY more images than validation training set
  • proportional to # of images in dataset
  • if training set is 2x larger, then loss should be 2x larger
  • they should swoop in and get close to each other
• It's okay to use batch norm here instead of dropout
• READ PAPERS that discuss current ways that people are detecting images of HABs using ML!!

Next Steps

• Run model with 2 Classes (7.1.21)
  • put data in two classes (only include obvious HABs images)
• Look into Mix-up & Cut-up
  • Papers with code
  • How to implement
• Look for HAB detection research papers