Meeting Notes

[cwickham]

A place to keep a record of what we talk about.

2020-12-09

Updates

First three tasks from #1 complete.

Questions

How should Ericka send Charlotte questions/updates before meeting?

Use these issues.

Was McNemar's test appropriate here?

Charlotte doesn't know. Let's keep this in the back of our minds:

• What/How exactly did they do/use McNemar's?
• Is that appropriate? Is there something more appropriate?

Action Items

From todo.md:

• [x] Find resources on random forests
• [x] Dive into data - are there some resampling steps that need to occur?
• [x] Set up week 1 meeting

[cwickham]

2021-01-07

Updates

• Resources on random forests located here
• Data information located here

Questions

• I'm confused because it seems that they used both test and training cities from the competition (Tuia 2017 and Yokoya 2017). This brings in the issue of getting LCZ reference data/"truth" for the test cities, since it's not publicly published. I could email and try to get it, or we could just focus on using the training cities since those have the "truth" data we can use to verify?

  Hong Kong is training city so still a good candidate for a scaled down replication.

• Since they're input data is just pulled from USGS Earth Explorer, is it an adequate reproduction to just download those files based on the dates they've given? Would it make more sense to use the reference data provided by the competition instead? I'm unclear on why they used their own input data except maybe that it's more recent.

  Charlotte suggests using competition data to avoid extra data pre-processing steps. Yoo et al. might have had reasons (e.g. better prediction performance?) for a different down/up-sampling scheme, but we won't worry about that for now.

• How does bilinear resampling work? I see how you actually do it, they're are just options in QGIS to do it so that you can match resolutions together when combining data, but don't have a great understanding of what it actually does.

  Not as relevant if we don't have to pre-process data. Some reading:

  • https://en.wikipedia.org/wiki/Bilinear_interpolation
  • https://desktop.arcgis.com/en/arcmap/latest/extensions/spatial-analyst/performing-analysis/cell-size-and-resampling-in-analysis.htm

• Why doesn't the whole area have a LCZ classification?

  Probably something to do with how they got the classification from human experts. If you want a better answer, it probably means digging in to how the LCZ "truth" was generated.

• How do we go from the competition data files to a data frame/tibble that you can feed into a random forest function?

  Each row in that data frame is a pixel (Charlotte is pretty sure). One column will contain the response: the LCZ class. The other columns are the explanatory variables, the value for that pixel in each the LandSat bands. Also keep track of polygon IDs.

  In practice, I think the steps will roughly be (this reference from your list looks useful)):

  • Import all tiffs
  • Extract from Landsat images/bands the values from pixel where you have a LCZ

  • Make sure you add/keep columns for LCZ class, and polygon ID.

    Probably the raster package will have the tools for that.

  You'll have to split the competition training data into your own training/test split, so you can honestly evaluate your models (since you only know LCZ classifications for the training polygons). Re-reading this part in Yoo et al. - how did they split?

Action Items

• [x] Set up weekly meetings - Thursday at 1pm. Ericka to set up zoom/invites.
• [x] Attempt to get from Tiffs to a data frame appropriate for feeding into random forest function. To get started:
  • Identify a pixel in an LCZ polygon
  • Get the value from a LandSat image band, for that pixel
• [x] Schedule Oral Exam
• [x] Go through 2+ random forest resources

[ericka-howard]

2021-01-14

Updates

• Resources worked through: World Urban Database and Access Portal Tools Video, Classification Vignette - Chris Holden, Victor Zhou - Intro to Random Forests
• Data successfully loaded into R. Difficulty convincing myself that what it looks like makes sense.
• Oral exam scheduling waiting on a response

Questions

• I went through the example from Chris Holden and it made me wonder why our LCZ data is in raster files instead of polygon shapefiles, and if that's advantageous for some reason. It seems like the best way to use randomForest is to convert all the data into a dataframe, which we talked about. But I'm not sure which method makes the most sense (or if it really matters):

  • Option A:
    1. Import tiffs using brick()
    2. Convert each separately into a dataframe
    3. Combine dataframes
  • Option B:
    1. Import tiffs using brick()
    2. Combine bricks into a rasterstack
    3. Convert rasterstack to dataframe
  • Option C:
    1. Import tiffs using brick()
    2. Convert LCZ to shapefile (can switch order with 1)
    3. Combine brick and shapefile into a dataframe using extract() (like in example)

  Start with Option B. Option A is the back up plan if Option B seems too memory intensive.

• Looking at the data my understanding is that:

  • there are four different dates with landsat 8 imagery and that within the folder for each of these dates there is a separate tiff file for each band 1-11
  • there are two different lcz tiff files, one is colored (_col) and the other is black and white and I think geometry and topology (_GT) .

  • SO why are there two LCZ tiff files and why does it seem like they don't say much (see ___ for histograms)? What do those numbers even mean (i.e. what lcz classes do they correspond to)?

    Charlotte suspects the four bands are: Red, Green, Blue and alpha. Combined these give the LCZ classes.

    Ericka says looks like _GT file has one band, and value is the class. Make histogram of this band in the _GT and try to confirm it looks like the LCZ classes.

• Does it make sense that I would get rid of the zeroes while running the random forest since they don't have a value for lcz?

  Yes, you won't need to the pixels with LCZ of zero (missing LCZ) to build the models, but you will need them later for prediction. Build the data frame with all LCZ pixels, then subset after.

• What to do with the four LandSat days? Combine? Pick one?

  • Include all bands from all days - 4 days x 9 bands = 36 predictor variables
  • Summarizing the four days, you decide on reasonable way to get more predictive power by combing the days in some way (feature engineering)
  • Pick the most predictive days (variable selection)

Action Items

• [x] Finish data preprocessing
  • [x] Combine bands and get rid of ones not used in paper
  • [x] Figure out how to/how they randomly and evenly divide the polygons.
• [x] Start references
• [x] Go through 2+ random forest resources
• [x] What are the things/arguments that I control when I run that randomForest function?
• [x] Start writing background info/introduction

[ericka-howard]

2021-01-21

Updates

• Loaded in data for a specific day (/"scene") and ran a default random forest on the LCZ data, then used it to predict the rest of Hong Kong
• References added to Zotero At some point, look into bibliographies in R markdown: https://rmarkdown.rstudio.com/authoring_bibliographies_and_citations.html
• Outline and writing has finally started. Outline looks great!
• Resources on random forests:
  • Leo Breiman (2001)
  • Parsnip
  • R Documentation
• Final exam is scheduled, program of study is fully done.

Questions

• The data isn't exactly clipped to Hong Kong, it includes some of Shenzhen. There's one polygon that overlaps.

  • I found a shapefile I could clip it with but I'm not sure if it makes sense to do so?
  • I'm pretty sure that they did because there's one LCZ polygon that goes into Shenzhen, but I quit pursuing it for now.

  Let's not worry about clipping, you might need to talk about "the Hong Kong region".

• There's a discrepancy in the numbers of pixels in [Table 3]() compared to the numbers of pixels I get when I do it.

  • My original thought was that it was because I only looked at one scene, but my table had more pixels in multiple categories.
  • After removing all NAs in the thermal bands (bands 10 & 11), numbers were closer, but there were still differences in 14 (low plants) and 17 (water).
  • Should I care about this?

  Yes, this is important place to verify your replication of their work. You should talk about it in the paper. But, make a decision on how you'll do it, document that, and move on.

• Does the "class imbalance" here matter?

  A useful baseline overall accuracy, is to think predicting every pixel to be the most represented class.

  How does this influence the model fit? Can you tell if this is a problem from the output? E.g. how does overall accuracy break down across the LCZ classes?

• I don't think they really explained how they divided the polygons up. It seems straightforward except that I only have the data in pixels rather than in polygons, and they mention in the paper that using pixels from the same polygon for both training and test artificially inflates accuracy assessments.

  Is there a way to identify which polygon a pixel is in, like a polygon ID? If not, you need to assign it. I don't know how to do that off the top of my head: shift from raster to shapefile, do point in polygon type operation; figure how to draw a boundary around pixels that are adjacent and of the same LCZ class.

• Not feeling great about starting methods because I'm not sure how much detail to go into about random forest(s?) specifically vs. what I actually did with the data. The examples I've seen of peoples' final reports seem to go either way.

• What does the output mean?

  Err on the side of depth of explanation, we can trim later for the report.

Action Items

• [x] randomly and evenly divide the polygons.
• [x] Go through 2+ random forest resources
  • [x] https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm
  • [x] StatQuest Videos
• [x] Start writing methods
• [x] Fit Different Classification Schemes

[ericka-howard]

2021-01-28

Updates

• Writing! 😄 (https://docs.google.com/document/d/1cW2cLbzescFHPJHLsVdA93uS9pZTvmCnZKI90fSgoIs/edit?usp=sharing) has finally started.
• Polygon IDs added to LCZ reference data. (Probably will have divided into training and test by the time of meeting but not currently done).
• Plugging along through resources. StatQuest Videos were pretty helpful. Did not make it through massive Berkeley/Breiman website.

Questions

• Everything I've read says that Out-Of-Bag estimates are just as accurate as cross-validation, so I don't understand why they used cross-validation.

  • My initial thought was that they wanted to be able to use the OA metrics used in the competition for consistency, but I don't think they were actually in the competition.
  • It says you can still upload your predictions to get a score from the competition website, which they may have wanted to do since they used some of the test cities (so full LCZ classification isn't provided anywhere).
  • I still think it's weird they didn't mention OOB Error somewhere in the paper.

  I think this is a good question, and good observation. It might be interesting to compare test data accuracy to the Out-Of-Bag accuracy estimates.

  One thought is that the OOB error is going to be based on random samples of pixels, but pixels within the same polygon as probably similar, and this may over estimate the accuracy for new pixels in new polygons - the test/train split of polygons doesn't have that problem.

  Another thought is the OOB error is used for tuning. If you use the data to fit many forests with different parameters and pick the one with the lowest OOB error, that OOB error is no longer an unbiased estimate of the true error, and you need another unseen set of data.

• What about the 90/10 split inside the training data?

  We are assuming this split in on pixels. Why not use OOB error to choose parameters here? Can you get their OA metric on the OOB samples directly from randomforest output? If you can, I'd just use the OOB estimates for parameter tuning...

• Once a variable has a node can it be reused on a different branch? Feels like it has to be? But never on the same branch? I'm having a hard time conceptualizing what all is in the pile of variables that are randomly selected at each node.

• I don't think my preprocessing actually matches up with theirs... (but does it even matter because it'd be a rabbit hole?)

  • my outline, what they did
  • in their paper, what they did
  • my outline, what competition data (& therefore my data) is
  • competition paper, what they did (page 3, A. Landsat Images)

Replicate "as close as practical", point differences where you think they exist.

• I think my description of random forests needs to be more technical for the type of project this is, do you agree?

Conceptual, is fine for now, I'll point out if you need to firm it up when I review.

Action Items

• [x] Randomly and Evenly Divide the Polygons.
• [x] Fit Different Classification Schemes
• [x] At least set up accuracy metrics (add calculation formats into R somewhere)
• [x] Organize repo, especially docs section
• [x] Cite some of claims in introduction

[ericka-howard]

2021-02-04

Updates

• Polygons divided. I went with randomly and evenly by polygon (but not pixel). It turned out relatively even most times running it that way. Looks good.
• Default classification scheme fit, between 70-78% accuracy on test pixels. Turns out this was due to not changing a set of variable names, TBD how well the other classification scheme works.
• OA accuracy function set up, need to make F-1 function What exactly is F-1 function? Read article they reference about it.
• Read through comments on writing, fixed some but haven't fixed the bigger two yet (will do this week). Talked about how to firm up random forest paragraph. See comments on Google doc for full but - Talk about decision trees first, need to clarify some of the terms, then talk about how random forests are different. Introduce depth, nodesize, maxnodes
• Repo looking better.

Questions

• The best way I could think of for accomplishing even polygons, even pixels, and randomization was to pair polygons by size and randomly assign each half of a pair to each group. There's the issue of polygon pairs not being able to be in the same group though. This felt like more of a problem than doing it randomly by polygon and having more uneven numbers of pixels, which is why I went with the latter. Does that seem reasonable? Here's my plot Seems fine
• Brainstorm on what sorts of plots/tables/figures make sense to have? See Google doc. Table/Plot that shows the OA (possibly F1) over the tuning parameters tried, variable importance plot, for one model, a complete map of predictions.
• What should the end product look like? Should I try to have one "best" model and then just list the others and why they aren't as good? Should it be the map of predictions in some way? Covered in last question.

Action Items

• [ ] Finish Methods Section
  • [x] Description of data (actual volume of data involved + how this data relates to the model)
  • [x] Explanation of how a typical decision tree is built
  • [x] Explain Gini Impurity & Information Gain. What "best" means
• [ ] Start preparing results figures
  • [ ] Table/Plot that shows the OA over the tuning parameters tried
  • [ ] Variable importance plot
  • [ ] One complete map of predictions
• [ ] Which models look the best? Why?
• [x] Create F1 metric function

[ericka-howard]

2021-02-15

Updates

• Finished most methods section, located here in google doc. Still working on volume of data description (need to do table)
• Also started creating the actual report, located here in repo as .Rmd and here as .pdf. These don't actually say anything different, but the formulas are legible and I just wanted to see it all together.

Questions

• 3

• Should explanation of what parameters I tried varying go in methods or results? Methods
• Summarise mostly just works.
• {{ }} do quasiquotation better
• What to do about getting all 1s from OOB accuracy metrics calculations? Charlotte says:
  • Try replicating OOB Error with my function first.
  • Currently predicting on entire random forest when it needs to be just on the specific tree who's OOB sample you're looking at.
  • Backup plan if it doesn't work is to choose my model parameters based on OOB error rates and then calculate my metrics on my actual models
  • Make sure to estimate how long it will take to run before starting things up.

Action Items

First

• [x] Replicate OOB Error with my Function
• [x] Update function to predict based on one tree, not entire random forest
• [x] See how long it will take to run with params and run
• [ ] Add information about what parameters are going to be varied and how in methods section

If time:

• [ ] Work on results figures
  • [x] Table/Plot that shows the OA over the tuning parameters tried
  • [x] Variable importance plot
  • [ ] One complete map of predictions
• [x] Write Results
• [x] Write discussion
• [ ] Final Evaluation of where things are at and setting what "finished" means by Thursday Meeting (18)

Charlotte:

• [x] review methods section, think on commenting options (maybe use pull request infrastructure?)

[ericka-howard]

2021-02-18

Updates

• Got OOB accuracy calculation replicated. It's exactly 1-OA
• Created function that takes in a parameter argument (like mtry=51 or 1ntree=700) and returns OA, OAurb, OAnat, and F-1
• Still working out how to consolidate those metrics when they are returned
• Fixed description of some of the parameters in methods section

Questions

• Unsure if the way I have that return() part of the function set up in the way that makes the most sense.
  • Try returning as one tibble with F1 scores rotated to be columns (mtry, OA, ..., F1_1, F1_2, ...) and then use map_drf()
  • Also consider using the ... to get the name of the parameter varied into the tibble
• Does it make sense to vary one at a time or multiple at a time?
  • With time in mind do one, but note in report. Interesting to do multiple because there could be interactions but stick with separate until everything is ready

Action Items

• [x] Do the actual thing! by Monday
  • [x] Get a parameter varied
  • [x] Plotted. Stay simple for now. Plots we've talked about are: OA over turning parameters, variable importance plot, full map
  • [x] Talked about (methods, results, and (?)conclusion)
• [x] Go through pull request on methods section

[ericka-howard]

2021-02-25

Updates

• Parameter varied, plotted, talked about
• Map of full prediction completed, needs legend
• I found the information about the Landsat scenes!

Questions

• Really would like to have the tables and references not be double spaced but haven't been able to figure out how (or if it's doable), any ideas? Am I missing something obvious?
• Same but for the bibliography (though I feel like I still might be able to find the answer somewhere, I haven't dug into this one as much)
• I added the scene information to that final barplot, since I have it now, but I'm not sure if it all should go on there or if there's too much unnecessary info

Action Items

• [x] Send to Lisa and James (by end of day Tuesday 2nd)
• [x] Figure out what can be cut to make it less than 15 pages
• [ ] Presentation:
  • [x] Make slide outline
  • [ ] make slides

[ericka-howard]

2021-03-04

Updates

• Paper sent out!
• Presentation still in progress

Questions

• Not a question but I think going through talking points for my slides might be helpful today
• For using importance measures to build a model with better predictive power - is it just that you want to choose the predictors with higher mean decrease Gini?
  • Unsure how I feel about the idea of keeping band 5 in scene 3, for example , but not band 5 in scene 4. (I've made the band and scene numbers up). Because there's not a good way to transfer that to a model for a different city. Unless there's some sort of scene effect being unaccounted for.
  • But then how is using the model for a different city not extrapolating regardless?

Action Items

• [ ] Finish slides
• [ ] Finish talking points
• [ ] Make loose script
• [ ] Practice
  • [ ] 1 time timed without seeing clock
  • [ ] x times recorded and timed