Meetings - Vedran

[linaskerath]

2023 02 07

---

• [x] Meeting structure
  • Have a special meeting once a month:
    • discuss timeline/milestones
    • go over to do list; send report; guide general direction of project
  • Update, ToDo and Q&A
• [x] Project description/proposal
• [x] Project name
• [x] Timeline/ What to start with etc.
• [x] Discuss problems:
  • regression vs classification
  • bias:
    • cloud
    • dataset match problem (80%)
    • seasonal bias?
    • threshold?

      ---

• Writing:
  • Proposal
    • 1 page
    • more or less the same but adding more precise info about features, regression etc.
    • feature importance should be in research question
    • maybe ask SAP about if title is still changeable afterwards?
  • Thesis: length should be 30< 40-50 < 80
• Model:
  • focus on regression
  • maybe add binary classification output as feature later on (one vs. rest classification?)
  • types of regression:
    • linear: ridge and lasso
    • Random forest
    • XG boost
    • Gaussian process model
  • for linear: try regularization or non linearities
  • fine-tune train-test splits according to temporal information
  • two models on different train-test splits (high and low temporal granularity) and evaluate against each
• Data:
  • mask out inner land (maybe ask Baptiste about how to filter?)
  • other convolutions to data?
  • use non-linearity as feature
  • ozone layer concentration
  • history of one pixel as feature
  • maybe consider average temperature increase of a year for comparison to "norm"
  • Research:
  • look for features from other studies

Focus on:

1. Writing proposal
2. Cleaning up pipeline
3. Research on possible features
4. Research on additional datasets

[linaskerath]

2023 02 14

---

Questions:

• [x] Temporal information:

  • [ ] how should we incorporate the history of a pixel - maybe as average?
  • [ ] history of which data set ? mw or optical ?

• [ ] Focus on melt areas, how?

  • [ ] How do area specific RMSE?

What we did:

• write project proposal
• clean up repo
• requested weather data
• started on features (convolution, ..)

What we're doing next:

• continue on features and weather data
• outlier testing
• normalization
• implement training pipeline with proper 5-fold cv

---

• Data:
  • temporality:
    • temporal granularity of pixel history should depend on how stable weather in Greenland is in general ( for temporal weather data)
    • unclear if we will do temporal melt data
    • possible ways to include temporal info: delta (current and previous day), avg of several days, weighted avg, incluse several days before as features.
  • figure out how to properly normalize data (do not randomize folds)
  • maybe filter data by cropping out areas/pixels that do not melt once throughout the whole time (remove all areas where melt was never recorded)
• Model:
  • figure out how to properly split data into train and test
• Features:
  • For convolution features: max difference (between highest and lowest value in convolution window)
  • maybe forwards and backward feature selection
    • only for linear models
    • we could do plot of Accuracy % vs. (number of) features

Focus on:

1. Feature Selection
2. Data Normalization/ standardization
3. Model Selection
4. Train-Test Splits

[ninazahn]

2023 02 28

---

From meeting with Baptiste:

• [x] Inform him about the slope feature and the seasonal info.
• [x] Discuss outlier removal.
• [x] Inform that external data is not very valuable + out of scope.
• [x] Discuss weather station data:
  • [ ] Introduce V to the problems with weather station data: ...
  • [ ] Whether to use as predictor or evaluation
  • [ ] Mention that B said to focus on only a few weather stations (if use for evaluation).
  • [ ] Discuss features to add (if use as predictors).
  • [ ] Separate model for ice..?
  • [ ] ..

---

[ninazahn]

2023 03 06

---

Model training:

• [ ] should we mix dates in training data or is it fine if we feed it consecutive days?
• [ ] cv for hyperparameter tuning?
• [ ] if yes, use the same data to do cv for tuning and deciding on model?
• [ ] how to deal with over-fitting: print train scores in cv as well?
• [ ] spatial cv
• [ ] permutation testing vs feature selection

---

• nested cv for hyperparameter tuning and model selection https://weina.me/nested-cross-validation https://arxiv.org/pdf/2108.02497.pdf

• spatial cv - remove no-melt data and then split into several static regions https://www.nature.com/articles/s41467-020-18321-y

• overfitting - some models like lasso and ridge regression reduce overfitting by regularization

• feature selection:

  • we could also try training on several regions separately and compare features to see if most important features change.
  • but probably stick with other feature selection methods, not permutation testing?
  • some models have 'build-in' feature selection.

• 3 distribution for outlier detection: snow, ice, melt

[linaskerath]

2023 04 04

---

• [ ] filter more on melting area to force towards focusing on melt? (melt area focus)
• [ ] area specific rmse?
• [ ] no melt area for cv? rmse check? - nah
• [ ] Grain size problem - how to deal with (postprocessing?) - maybe address this later
• [ ] Data Normalization
• [ ] feature selection (convolutions, temporal)
• [ ] ask if mention research project in thesis
• [ ] hpc

---

• If the hpc no good, he can provide us with an aws instance
• melt area focus:
  • analysis on zoomed area
  • melt (bucket) vs error (prediction)
• Normalization:
  • try z score and log and plot on normal and loglog scale
• Feature selection: continue as did. This paper talks about automated feature creation: https://dlab.epfl.ch/teaching/spring2019/cs718/papers/blumenstock2015predicting.pdf (pg 12 Deterministic finite automaton (DFA))

[ninazahn]

2023 04 18

---

• Normalization:
  • plot with logbins (equal sized bins with log log scale)
  • do log transform on target variable (opt_values) - remember to transform back when looking at error (also substracting constant -1??)
• Feature Importance/Selection:
  • ideally we would do foreward or backward feature selection, but not feasible for scope and computationally.
  • group features, e.g. location vs. everything else (maybe static vs. dynamic?)
  • plot model accuracy with loc features, with everything else, with both
  • x-axis = feature groups, y-axis = accuracy in %
  • possibly, if you want a feature to have less power -> reduce its scale (we already do that by normalization)
  • it Might be possible to restrict feature importance in RF by reducing selection probability in bagging (for x and y), but need to investigate this.

• Ideas for Plotting of Predictions:
  • plot data of one validation day
  • plot data of all models in one plot with different std deviations (??)
  • plot 2D heatmap of y (actual) vs. y_pred
  • plot with x-axis actual y and y-axis error R2
  • plot with x-axis being variables/features and y-axis being weights
  • plot Map with true vs. predicted values
  • plot Map with True - Predicted values
• Models:
  • save 5 models from outer CV and predict on 5 (easy for LR since only a few float values)
  • save trees from random forest for prediction
  • use only one Model for SHAP feature importance

[linaskerath]

2023 05 09

---

• [ ] ask why forest not running
• [ ] why no coef_ in lasso and elastic
• [ ] why ridge all positive coef
• [ ] feature importance -> almost no diff wo x and y (in LinReg)
• [ ] When can we give him the report to read through?

---