WSP_Regional_Summaries.Rmd: Mapping & Table Generation RMD

glenncampagna commented 1 year ago

This issue will track status and progress of RMD file made to accomplish goals laid out in https://github.com/HARPgroup/HARParchive/issues/853#issue-1720232156 New filename: WSP_Regional_Summaries.Rmd New package as of 7/29: pandoc for viewing our 2nd round of tables

Current file: https://github.com/HARPgroup/HARParchive/blob/master/HARP-2023-Summer/WSP_Regional_Summaries.Rmd Current params w/ examples:

rivseg = "JL6_7430_7320"
locality = "Montgomery"
region = "Shenandoah_2" 
type = "facility" 
model_version = "vahydro-1.0" 
runid_list_facilities = c("runid_11","runid_13")
runid_list_riversegs = c("runid_11","runid_13")
metric_mod = "wd_mgd" 
metric_feat = "wsp2020_2040_mgy"
rivseg_metric = "l90_Qout"
map_type = "region"
map_style = "custom"
map_by = "runid_11_wd_mgd"
limit = "basins"
table_col = c("runid_11_wd_mgd","runid_13_wd_mgd","fiveyr_avg_mgy")
bbox_type = "auto"

Param Info: rivseg: river segment ID of basin to be mapped, for map type basin locality: locality/county of interest, for map type locality region: region of interest, must match a region name in https://github.com/HARPgroup/HARParchive/blob/master/HARP-2023-Summer/Regions_ProposedReg_053122.csv type: either facility or source map_style: styling category for map aesthetics, either custom, default, or colorblind model_version: used in pulling facility metrics from vahydro runid_list_facilities: list of runids for facils/sources maps & tables runid_list_riversegs: list of runids for riverseg drought maps & tables
metric_mod: the modeled metric name to be mapped & used in table. Does not need to be set to fiver_avg_mgy metric_feat: non-modeled metric name rivseg_metric = drought metric for riverseg maps map_type: either basin , locality , or region , to control the extent of the map & table map_by: metric(s) to map and scale points/bubbles by. Must include both the runid and metric name for modeled metrics formatted like runid_11_wd_mgd limit: either basins, for including all facils/sources from intersecting basins in the map/table, or boundary to only include those within the locality or regional boundary table_col: specific metrics to include in the flextable. Must include both the runid and metric name for modeled metrics (formatted like map_by var) bbox_type: either auto or vahydro

Example Render:

rmarkdown::render("~/Desktop/GitHub/HARParchive/HARP-2023-Summer/mapping_codeReview.Rmd", 
                  output_file = "~/Desktop/HARPteam23/mappingRMD_knit", 
                  output_format = "word_document",
                  params = list(
                    rivseg = "JL7_7070_0001", 
                    locality = "Montgomery", 
                    region = "MiddleJames_2", 
                    type = "facility", 
                    model_version = "vahydro-1.0", 
                    runid_list_facilities = c("runid_11","runid_13"), 
                    runid_list_riversegs = c("runid_11","runid_13"),
                    metric_mod = "wd_mgd",
                    metric_feat = "wsp2020_2040_mgy",
                    rivseg_metric = "7q10",
                    map_type = "region",
                    map_style = "custom",
                    map_by = "runid_13_wd_mgd",
                    limit = "basins",
                    table_col = c("runid_11_wd_mgd","runid_13_wd_mgd","fiveyr_avg_mgy","wsp2020_2040_mgy"),
                    bbox_type = "auto"))

Note: output_format determines the format of the rendered and saved file with the options: pdf_document, word_document, html_document

Functions & Config Files:

Functions: fn_mapgen: contains the mapping process used to create all map types (facils/sources only and rivserseg maps) fn_filter_map: contains process of filtering flowlines, labels etc. to be mapped based on size of boundary box, only called within mapping function fn_tablegen: creates a flextable from data frame provided (and soon the columns specified within that data) fn_nhd_labs: contains nhd-specific actions for labeling (flowlines and waterbodies) fn_road_labs: contains roads-specific actions for labeling fn_labelprep: general actions for label processing, run on nhd labels and road labels after each of their respective functions fn_download_read: used for downloading & unpacking less-straightforward filetypes like .zip files fn_process_geom: for creating spatial objects by specifying which data column contains geometry information fn_centroid_coords: for adding centroid coordinates to a data frame based on the geometry contained, used mostly for labeling purposes
Config: mapstyle_config: contains aesthetic customizations for mapping (all map types), including unique styling for each label type (rivers/streams, counties, cities etc.) rivsegmaps_config: contains customizations specific for rivseg drought maps, including values and colors for % difference ranges

rburghol commented 1 year ago

Hey all -- happy to report that I used the runid_11/runid_13 arguments in my test rendering (see #988 ) and the map and table rendered well. Couple things (will update in the body of this issue):

The table should be rendered as a table, not as an image, as users will invariable wish to copy items from these.
Stream order and/or threshold for displaying should be part of the customizable styling parameters -- When I do the whole James, it gets too busy (see below).
It's time to start retrieving custom items from vahydro.
Currently the output file is html, and I think we want to move towards doing pdf or word/docx? (tho I would note that pdf, while rendering nicely will undermine the ability to copy and paste tables, as in, by "undermine" I mean it will make it impossible).

rmarkdown::render(
    'C:/usr/local/home/git/HARParchive/HARP-2023-Summer/mapping_codeReview.Rmd',
    output_file ='C:/Workspace/tmp/james_rva_wsp.html',
    params = list(
        rivseg = "JL7_7070_0001", locality = "NA", region = "Shenandoah_2",
        type = "facility", model_version = "vahydro-1.0", runid_list = c("runid_11", "runid_13"), 
        metric= "wd_mgd",
        map_type = "basin", bbox_type = "auto"
    )
)

glenncampagna commented 1 year ago

Script & param updates:

New params: map_by and table_col to customize which metric(s) are mapped and shown in the table. Both usable for multiple metrics, and multiple maps will be returned if requested Modified: table_limit was changed to just limit, since it applies to both the map and table metric no longer needs to be set to fiver_avg_mgy when that metric is wanted for mapping or table. Just make sure the metric is called where you'd like it, in map_by and/or table_col

[ ] Add limiting of nhd streamlines and labels to customizable elements, the main thing cluttering Rob's large James map. Part of this could be adding more specific classes to the waterbody labels so they can be filtered how the streamlines are
[ ] Pivot to rendering as word doc or pdf (word might be preferred for tables). Note: final product will be in pdf format
Size of waterbodies could be better represented by volume than surface area, which are both included in nhd data. Update: not all waterbodies have a volume value, while all have a surface area

glenncampagna commented 1 year ago

Update:

Side-by-side mapping and the extra package grid have been disabled for better detail and now testing can be done without any new packages with params outlined in the top of this issue (@gmahadwar) Also: the map_by and table_colparams must correspond to the metric data pulled with the metric and runid_list, for example runid_13_wd_mgd cannot be mapped or added to table unless the metric is set to wd_mgd and runid_13 is included in runid_list Recent: a problem with basin maps was fixed and merged @ 12:45 pm 7/17

EllaF21 commented 1 year ago

README File

A README file has been created for the mapping process and can be viewed at this link: https://github.com/HARPgroup/HARParchive/blob/master/HARP-2023-Summer/README.md This is just a first draft and as we think of new things we can easily add them in. Feel free to recommend any fixes or additions you notice necessary to the base outline now.

rburghol commented 1 year ago

RE-do zoomed in on James Planning Region. @glenncampagna I am getting a LateX error wen trying to render. It appears to complete successfully, although there are some data timeouts that may be causing issues? @glenncampagna can you try to run this and see if it will render on your machine? I am wondering if my knitr is out of date.

Error: Error: LaTeX failed to compile C:/Workspace/tmp/james_rva_wsp.tex. See https://yihui.org/tinytex/r/#debugging for debugging tips.

rmarkdown::render(
    'C:/usr/local/home/git/HARParchive/HARP-2023-Summer/mapping_codeReview.Rmd',
    output_file ='C:/Workspace/tmp/james_rva_wsp', output_format = "html_document",
    params = list(
        rivseg = "JL7_7070_0001", locality = "NA", region = "MiddleJames_3",
        type = "facility", model_version = "vahydro-1.0", runid_list = c("runid_11", "runid_13"), 
        metric= "wd_mgd",
        map_type = "region", bbox_type = "auto"
    )
)

glenncampagna commented 1 year ago

@rburghol Sorry, there's a new package required for knitting by PDF: tinytex Use this: tinytex::install_tinytex()

rburghol commented 1 year ago

Thanks @glenncampagna -- see above, map renders, had to export as HTML instead of pdf. @jdkleiner pointed out that in order to export as html (or any other format) we can override in the calling statement by including output_format = "html_document"

jdkleiner commented 1 year ago

@rburghol @gmahadwar @glenncampagna @megpritch @EllaF21

I'm successful generating as a word doc as well! (Using the output_format Rob noted above). Looks pretty good too! Does a nice job out of the box generating the map and table within the word doc (see snapshot below).

rmarkdown::render(
  input = "C:/Users/nrf46657/Desktop/GitHub/HARParchive/HARP-2023-Summer/mapping_codeReview.Rmd",
  output_file = "C:/Users/nrf46657/Desktop/GitHub/plots/james_rva_wsp",
  output_format = "word_document",
  params = list(
    rivseg = "JL7_7070_0001", locality = "NA", region = "MiddleJames_3",
    type = "facility", model_version = "vahydro-1.0", runid_list = c("runid_11", "runid_13"), 
    metric= "wd_mgd",
    map_type = "region", bbox_type = "auto"
  )
)

glenncampagna commented 1 year ago

Fixed an error related to textsize variable as of 7/20 @ 12:30 pm so a new version of the script should be pulled for testing

EllaF21 commented 1 year ago

Rendering All Regions

All region maps were rendered with the following render command successfully, only adjusting the region parameter. Some were also rendered with word_document to ensure this also functioned correctly. rmarkdown::render(paste0(getwd(),"/mapping_codeReview.Rmd"), output_file = paste0(export_path,"mappingRMD_knit25"), output_format = "pdf_document", params = list( rivseg = "JL7_7070_0001", locality = "Montgomery", region = "NorthernVirginia", type = "facility", model_version = "vahydro-1.0", runid_list = c("runid_11","runid_13"), metric_mod = "wd_mgd", metric_feat = "wsp2020_2040_mgy", map_type = "region", map_by = "runid_13_wd_mgd", limit = "boundary", table_col = c("runid_11_wd_mgd","runid_13_wd_mgd","fiveyr_avg_mgy","wsp2020_2040_mgy"), bbox_type = "auto"))

In viewing all of the maps together we found some potential areas for adjustment. Included are examples of good and crowded maps for comparison and what we think needs adjustment.

These maps are good examples of maps that are not too crowded and have good label placement:

These maps have more crowding and the potential for us to make a few fixes:

What we want to fix:

[ ] Make sure the scalebar is more adjustable so it does not add to crowding
[ ] Get road labels to not be as frequent
[ ] Make sure our set extents for labeling frequency are at the level we want (less nhd lines/labels in more crowded extents)
[ ] Decrease city labeling in some extents (especially Northern Virginia) -- plan to use max.overlaps on separate city labs layer
[x] Do we want the boundary limit to have gray space/cut labels outside the boundary or keep the counties whole (the basin limit grays out outside of the counties)
[ ] Make sure our zooms for extents are set as we want -- does zoom have any effect on the basemaps without labels?

Another note: When mapping the region MiddleJames_2 in the basin limit as a pdf, we received a latex error, but not when mapping as boundary or when rendering as a word document.

glenncampagna commented 1 year ago

First version of Rivseg drought metric maps

Let us know if anything stands out to you @gmahadwar that needs to be changed (besides the legend). I tried to use a color scheme similar to that of the state plan maps Accompanying table coming soon

EllaF21 commented 1 year ago

Rivseg Map/Table Updates

We now have the capability to map and create tables for multiple metrics in one rmd:

Highlighted is any value below -10% difference, this can be adjusted in the config file.
Tidal segment differentiation is also now in place.
The metric for the specific map is specified in the legend and adding it to the title is being worked on.
At the moment each metric has its own table that are just put under eachother in the rmd. The metric is specified in the Metric column.
There is also capability to specify which columns are desired in the tables.

Let us know if there is anything that stands out that needs fixing.

gmahadwar commented 1 year ago

@EllaF21 @glenncampagna @megpritch Additions and appearance in the latest map look good! Pausing on having appearance tweaks be tasks at this stage of the summer. A couple comments I have for when it makes sense to pick this up during the year

the river seg boundary color is really similar to the fill color in the map two comments up, that's not necessarily a problem though
- [x] in the table one comment up, if instead of a highlighter mark a background fill color is an option for a row below -10%, then that would be better, but again this does accomplish the goal

Edits:

[x] If you haven't already the table column names should also be user understandable, (ie. no runid_11 or runid_13)
[x] This is true for the legend (as you've added), the table, the title, basically anywhere that the end user will be reading it
[x] Also for organizational purposes, adding keyword differentiators in the title for whether it's a locality/region map, and the metric / map type (again as you've mentioned) would be helpful
- [x] Ex. Stafford Locality, Lowest 30-day flow

rburghol commented 1 year ago

Random testing. Pretty damn solid y'all.

	Facility	HydroID	Locality	River Segment Name	River Segment ID	Permitted Capacity	runid 400 wd mgd	runid 600 wd mgd	5-yr Avg Use (mgy)	wsp2020 2040 mgy
1	Lake Kilby Water Treatment Facility	68367	Suffolk	Lake Mead Dam	JB1_8090_0001	11680	30.95	18.54	6116.93	6614.16
2	Perry Minnow Farm Incorporated	68367	Isle of Wight	Lake Mead Dam	JB1_8090_0001	No Permit	NA	NA	29.80	0.00
3	Suffolk Golf Course	68367	Suffolk	Lake Mead Dam	JB1_8090_0001	No Permit	NA	NA	0.36	0.00
4	IDLEWOOD SERVICE AREA	68367	Suffolk	Lake Mead Dam	JB1_8090_0001	No Permit	NA	NA	NA	0.00

rmarkdown::render("C:/usr/local/home/git/HARParchive/HARP-2023-Summer/WSP_Regional_Summaries.Rmd", 
                  output_file = paste0(export_path,"lake_kilby"), 
                  output_format = "word_document",
                  params = list(
                      rivseg = "JB1_8090_0001", 
                      locality = "Montgomery", 
                      region = "MiddleJames_2", 
                      type = "facility", 
                      model_version = "vahydro-1.0", 
                      runid_list_facilities = c("runid_400","runid_600"), 
                      runid_list_riversegs = c("runid_400","runid_600"),
                      metric_mod = "wd_mgd",
                      metric_feat = "wsp2020_2040_mgy",
                      rivseg_metric = "7q10",
                      map_type = "basin",
                      map_style = "custom",
                      map_by = "runid_600_wd_mgd",
                      limit = "basins",
                      table_col = c("runid_400_wd_mgd","runid_600_wd_mgd","fiveyr_avg_mgy","wsp2020_2040_mgy"),
                      bbox_type = "auto"))

HARPgroup / HARParchive