Add Instructions to the User Guide for FAF inputs to CT

[bettinardi]

Instructions need to be added to the user guide so that ODOT knows how to update the CT inputs that use FAF data for times when FAF has new releases.

[bettinardi]

FAF is currently on 4.5.1- https://faf.ornl.gov/fafweb/ One way to tackle and close this issue might be to have RSG complete this FAF-SWIM update and write the instructions as they complete this.

[bettinardi]

I'm capturing this request (in an email from Dec 2019) to hope this gets addressed during this issue:

We have noted that we need to reduce the truck volume on I-84. As I read through the model build documentation, I see that this file was created with a google API. And all shipments from a given FAF zone were assigned to a given SWIM external station. The text above from the wiki seems to indicate that a given FAF zone can be split across external stations. So if I wanted to assign 50% of Texas to I-84 and 50% to I-5 the input file might let me do that. But the input file doesn’t show an example with multiple external stations with different weights. Could I ask that the wiki be updated to discuss how this input file gets updated to add weighted external station information. Or maybe just an example input file with Texas split out could be provided to TPAU to work from. I think this one little addition could go a long way in helping TPAU clean up some of the issues we have seen with Truck assignment (with I-84 being way too high). I would like to play around with sending some of I-84 traffic (which is the bulk of where all the FAF data is destine) to some of the other external stations (thinking that some of those shipments might stop by some of the other big destinations on the West Coast, not headed straight to Oregon).

[bstabler]

do #129 at the same time

[bettinardi]

Summary for WOC 7 - To date we have not been able to develop the instruction set ODOT is looking for to provide ODOT written steps of how to pull a recent FAF product (like 4.5.1 currently) and update SWIM-CT input files to use the latest product. The intent of this issue / task is to have RSG pull the latest FAF product, and working with Rick update the CT inputs to use the latest FAF. As this update is being completed, RSG will write a detailed and sequenced process that can be followed by ODOT staff in the future to update CT as new FAF products are released.

The key parts of this issue / task are to:

• primarily provide ODOT with an instruction set for how to update CT to the latest FAF in the future,
• the byproduct of having a new version of SWIM-CT with the latest FAF,
• to build a wider knowledge base for how CT is setup and designed,
• as RSG updates this, it might require tweaks to CT setup and code, so a new CT code release might be another by-product, and
• issues #129 and #78 should be address and closed as this issue is worked on.

Specifically issue #129 wants the wiki to document what dollars CT is in (is it the SWIM 2009 dollar base, or is it based on commodity flow data from FAF). As RSG is documenting the process to update to FAF, part of this needs to include how dollars are or are not converted and make that clear to the user (bolded on the wiki - with it's own special CT dollars section header).

Issue #78 can then be resolved after this issue. As RSG builds knowledge with this task, it will be better positioned to help review and finalize documentation around issue #78.

[bstabler]

start with "primarily provide ODOT with an instruction set for how to update CT to the latest FAF in the future"

[goreaditya]

@bettinardi There's a script in the the package swimctr called preprocess_faf4_database which I am planning to use as a template to process raw data. According to the description in the script the function uses a set of FAF region pairs (called outer regions) to process flows that are likely to pass through model region. I found another file oregon-outer-regions.csv that lists all the FAF region pairs that are likely to pass through model region. When I compared it with the process FAF (v.4.2) dataset I noticed that most of these pairs are dropped out. May I know what logic/criteria was used to filter out most of the FAF region pairs from the oregon-outer-regions.csv in the final processed FAF v4.2 data. I have included a snapshot of FAF regions pairs that are included in and excluded from processed FAF data.

Included

Seattle - {AZ, AR, CO, LA, MS, Las Vegas, NM, OK, TX, UT, WY}

dms_orig	dms_dest	description	state	entry	exit	o_directions
531	321	Las Vegas-Henderson	NV	I82N	I84E	through

Spokane - {AZ, AR, CO, LA, MS, Las Vegas, NM, OK, TX, UT, WY}

dms_orig	dms_dest	description	state	entry	exit	o_directions
539	321	Las Vegas-Henderson	NV	I82N	I84E	through

Seattle - CA

dms_orig	dms_dest	description	state	entry	exit	o_directions
531	61	Los Angeles-Long Beach	CA	I5N	I5S	through
531	62	Sacramento-Roseville	CA	I5N	I5S	through
531	63	San Diego-Carlsbad-San Marcos	CA	I5N	I5S	through
531	64	San Jose-San Francisco-Oakland	CA	I5N	I5S	through
531	65	Fresno-Madera	CA	I5N	I5S	through
531	69	Remainder of California	CA	I5N	I5S	through

Spokane - Reno

dms_orig	dms_dest	description	state	entry	exit	o_directions
539	329	Remainder of Nevada	NV	I82N	US95S	through

Spokane - CA

dms_orig	dms_dest	description	state	entry	exit	o_directions
539	61	Los Angeles-Long Beach	CA	I82N	US97S	through
539	62	Sacramento-Roseville	CA	I82N	US97S	through
539	63	San Diego-Carlsbad-San Marcos	CA	I82N	US97S	through
539	64	San Jose-San Francisco-Oakland	CA	I82N	US97S	through
539	65	Fresno-Madera	CA	I82N	US97S	through
539	69	Remainder of California	CA	I82N	US97S	through

Seattle - Reno

dms_orig	dms_dest	description	state	entry	exit	o_directions
531	329	Remainder of Nevada	NV	I5N	OR39S	through

Excluded

AK - {CA, Reno}

dms_orig	dms_dest	description	state	entry	exit	o_directions
20	61	Los Angeles-Long Beach	CA	I5N	I5S	through
20	62	Sacramento-Roseville	CA	I5N	I5S	through
20	63	San Diego-Carlsbad-San Marcos	CA	I5N	I5S	through
20	64	San Jose-San Francisco-Oakland	CA	I5N	I5S	through
20	65	Fresno-Madera	CA	I5N	I5S	through
20	69	Remainder of California	CA	I5N	I5S	through
20	321	Las Vegas-Henderson	NV	I5N	I5S	through
20	329	Remainder of Nevada	NV	I5N	I5S	through

Seattle - {AZ, AR, CO, LA, MS, Las Vegas, NM, OK, TX, UT, WY}

dms_orig	dms_dest	description	state	entry	exit	o_directions
531	41	Phoenix-Mesa-Glendale	AZ	I82N	I84E	through
531	42	Tucson-Nogales	AZ	I82N	I84E	through
531	49	Remainder of Arizona	AZ	I82N	I84E	through
531	50	Arkansas	AR	I82N	I84E	through
531	81	Denver-Aurora	CO	I82N	I84E	through
531	89	Remainder of Colorado	CO	I82N	I84E	through
531	221	Baton Rouge	LA	I82N	I84E	through
531	222	Lake Charles	LA	I82N	I84E	through
531	223	New Orleans-Metairie-Hammond	LA	I82N	I84E	through
531	229	Remainder of Louisiana	LA	I82N	I84E	through
531	350	New Mexico	NM	I82N	I84E	through
531	401	Oklahoma City-Shawnee	OK	I82N	I84E	through
531	402	Tulsa-Muskogee-Bartlesville	OK	I82N	I84E	through
531	409	Remainder of Oklahoma	OK	I82N	I84E	through
531	481	Austin-Round Rock	TX	I82N	I84E	through
531	482	Beaumont-Port Arthur	TX	I82N	I84E	through
531	483	Corpus Christi-Kingsville-Alice	TX	I82N	I84E	through
531	484	Dallas-Fort Worth	TX	I82N	I84E	through
531	485	El Paso	TX	I82N	I84E	through
531	486	Houston-The Woodlands	TX	I82N	I84E	through
531	487	Laredo	TX	I82N	I84E	through
531	488	San Antonio-New Braunfels	TX	I82N	I84E	through
531	489	Remainder of Texas	TX	I82N	I84E	through
531	491	Salt Lake City-Provo-Orem	UT	I82N	I84E	through
531	499	Remainder of Utah	UT	I82N	I84E	through
531	560	Wyoming	WY	I82N	I84E	through

Vancouver - {AZ, AR, CO, LA, MS, Las Vegas, NM, OK, TX, UT, WY}

dms_orig	dms_dest	description	state	entry	exit	o_directions
532	41	Phoenix-Mesa-Glendale	AZ	I5N	US95S	through
532	42	Tucson-Nogales	AZ	I5N	US95S	through
532	49	Remainder of Arizona	AZ	I5N	US95S	through
532	50	Arkansas	AR	I5N	I84E	through
532	81	Denver-Aurora	CO	I5N	I84E	through
532	89	Remainder of Colorado	CO	I5N	I84E	through
532	221	Baton Rouge	LA	I5N	I84E	through
532	222	Lake Charles	LA	I5N	I84E	through
532	223	New Orleans-Metairie-Hammond	LA	I5N	I84E	through
532	229	Remainder of Louisiana	LA	I5N	I84E	through
532	321	Las Vegas-Henderson	NV	I5N	US95S	through
532	350	New Mexico	NM	I5N	I84E	through
532	401	Oklahoma City-Shawnee	OK	I5N	I84E	through
532	402	Tulsa-Muskogee-Bartlesville	OK	I5N	I84E	through
532	409	Remainder of Oklahoma	OK	I5N	I84E	through
532	481	Austin-Round Rock	TX	I5N	I84E	through
532	482	Beaumont-Port Arthur	TX	I5N	I84E	through
532	483	Corpus Christi-Kingsville-Alice	TX	I5N	I84E	through
532	484	Dallas-Fort Worth	TX	I5N	I84E	through
532	485	El Paso	TX	I5N	I84E	through
532	486	Houston-The Woodlands	TX	I5N	I84E	through
532	487	Laredo	TX	I5N	I84E	through
532	488	San Antonio-New Braunfels	TX	I5N	I84E	through
532	489	Remainder of Texas	TX	I5N	I84E	through
532	491	Salt Lake City-Provo-Orem	UT	I5N	I84E	through
532	499	Remainder of Utah	UT	I5N	I84E	through
532	560	Wyoming	WY	I5N	I84E	through

Spokane - {AZ, AR, CO, LA, MS, Las Vegas, NM, OK, TX, UT, WY}

dms_orig	dms_dest	description	state	entry	exit	o_directions
539	41	Phoenix-Mesa-Glendale	AZ	I82N	I84E	through
539	42	Tucson-Nogales	AZ	I82N	I84E	through
539	49	Remainder of Arizona	AZ	I82N	I84E	through
539	50	Arkansas	AR	I82N	I84E	through
539	81	Denver-Aurora	CO	I82N	I84E	through
539	89	Remainder of Colorado	CO	I82N	I84E	through
539	221	Baton Rouge	LA	I82N	I84E	through
539	222	Lake Charles	LA	I82N	I84E	through
539	223	New Orleans-Metairie-Hammond	LA	I82N	I84E	through
539	229	Remainder of Louisiana	LA	I82N	I84E	through
539	350	New Mexico	NM	I82N	I84E	through
539	401	Oklahoma City-Shawnee	OK	I82N	I84E	through
539	402	Tulsa-Muskogee-Bartlesville	OK	I82N	I84E	through
539	409	Remainder of Oklahoma	OK	I82N	I84E	through
539	481	Austin-Round Rock	TX	I82N	I84E	through
539	482	Beaumont-Port Arthur	TX	I82N	I84E	through
539	483	Corpus Christi-Kingsville-Alice	TX	I82N	I84E	through
539	484	Dallas-Fort Worth	TX	I82N	I84E	through
539	485	El Paso	TX	I82N	I84E	through
539	486	Houston-The Woodlands	TX	I82N	I84E	through
539	487	Laredo	TX	I82N	I84E	through
539	488	San Antonio-New Braunfels	TX	I82N	I84E	through
539	489	Remainder of Texas	TX	I82N	I84E	through
539	491	Salt Lake City-Provo-Orem	UT	I82N	I84E	through
539	499	Remainder of Utah	UT	I82N	I84E	through
539	560	Wyoming	WY	I82N	I84E	through

Vancouver - CA

dms_orig	dms_dest	description	state	entry	exit	o_directions
532	61	Los Angeles-Long Beach	CA	I5N	I5S	through
532	62	Sacramento-Roseville	CA	I5N	I5S	through
532	63	San Diego-Carlsbad-San Marcos	CA	I5N	I5S	through
532	64	San Jose-San Francisco-Oakland	CA	I5N	I5S	through
532	65	Fresno-Madera	CA	I5N	I5S	through
532	69	Remainder of California	CA	I5N	I5S	through

Vancouver - Reno

dms_orig	dms_dest	description	state	entry	exit	o_directions
532	329	Remainder of Nevada	NV	I5N	OR39S	through

[bettinardi]

This is a great question / find and could be the solution to my comment in this thread above from June 25th - where we are getting flows that are too high on I-84 and too low on I-5. I do not know the answer to this questions. @bstabler, hopefully we can get Rick to respond to this once we have access to him.

[nsdhakar]

Response from Rick Donnelly (10/26/2020):

The process I developed tries to get around the fact that SWIM does not have an external network (i.e., it is a windowed network). Thus, when FAF flows enter or leave the SWIM cordon I need to figure where they do so. Fortunately Oregon is on the edge of the continent, so coming up with logic for this was easy. Anything that flows between Washington and (California, Arizona, Nevada) or vice-versa are assumed to flow through Oregon. So 100 percent of the truck flows between FAF flows in those regions are included. Ditto for imports and exports through Oregon. Likewise, flows going to and from Oregon from anywhere else in the country are easy to include.

Where it gets fuzzy is when flows between nearby states might flow through Oregon. Flows from Spokane, which we might infer as the epicenter of “rest of Washington” - FAF region 539) to Las Vegas (FAF region 321) come down I-82 through Umatilla, then go east on I-84 through Oregon to Boise and on to Twin Falls, where they turn south on US-93. Granted, the flows between those pairs are very small, but they’re not zero. According to Google Maps that route takes 15.75 hours versus 16.1 hours if traveling east on I-90 from Spokane and then I-15 south to Phoenix instead. Thus, truckers might view those two routes are equivalent. And if you add intermediate stops – something FAF doesn’t track – that might spell the difference between them. Thus, I might apportion 50 percent of the truck flows between FAF regions 539 and 321 to I-84/US-93 (through Oregon) and drop the other half (using I-90/I-15).

The more difficult part comes in allocating these flows to entry and exit points on the SWIM cordon. Interchanges between Seattle/Tacoma (FAF region 531), Vancouver WA (FAF region 532), and Portland (FAF region 411) are easily allocated to I-5. Flows between them and most FAF regions east of the Pacific Northwest can likewise easily be allocated to Interstate highways. Where it gets fuzzy again is dealing with FAF regions 419 (remainder of Oregon) and 539 (remainder of Washington). In those cases we must assume one or more centroids to determine most likely routings between them and other FAF regions. In Oregon it is relatively easy, for most of Oregon’s population and employment outside of Portland is in the I-5 corridor. Thus, I used Eugene as the centroid for FAF region 419 when looking at routings. Washington is more ambiguous, although the next largest area is Spokane. Thus, I use it as the centroid for FAF region 539. But it pretty much makes no difference where the centroid is if located in Eastern Washington, for flows to and from Oregon all still flow through I-82 in Umatilla. It would appear that the only possibility of measurable error might be Yakima-Portland flows, which would likely cross into Oregon and onto I-84 at Biggs or The Dalles rather than Umatilla. Without going through an arbitrary (as in data-free) allocation of FAF region 539 origins and destinations to counties in Washington it is hard to envision how we could capture that other than perhaps putting “attraction weights” or other arbitrary factors on each bridge. I cannot see the value in doing that.

I used Google Maps to generate the routings between the Washington and Oregon FAF region centroids and the rest of the country to assign them to an entry or exit node on the SWIM cordon, which is what you see coded in oregon-outer-regions.csv. I saw too few instances where crossing points were ambiguous (as in the FAF regions 539-321 interchange cited above) to worry about guessing what proportions to assume for different routes.

Absent major new US highway or Interstate routes in the Pacific Northwest I’m guessing these travel patterns wouldn’t change much over time.

Greg Macfarlane and I tried the different approach of allocating FAF interregional flows to counties using different weights on population and employment, and then routing them between counties on the ORNL network in MATSim. We checked the interzonal travel times of a subset of county interchanges in the Western USA and found that they compared decently close to Google auto travel times. We then looked at flows as SWIM cordon points and found that they didn’t vary much from the non-network allocation described above. Thus, we concluded that adding an external network to SWIM would add more overhead and complexity to SWIM than the small benefit derived.

I’m happy jumping on weekly chat with you folks to chat further about this if you’d like, but hopefully that gives enough insight into what and why I did it that way. I’m wide open to better ideas.

[nsdhakar]

Rick's email on Dec 17, 2020

Over the past two weeks I’ve dug out the 2006 and 2012 Commercial Vehicle Survey (CVS) from Ontario and have sliced and diced it a few different ways. You can see here what I was wanting to show you last night. I think it’s pretty self-explanatory but we can set up a teleconference tomorrow or next week if you want to chat about it before our next call. The gist from my end:

• The vehicle type utilizations by distance look reasonable to me, although I’m not sure whether I believe the FAF or CVS better represent Oregon conditions. Thus, I will run simulations with both and we’ll compare them.

• The empty trucks are a puzzler, for we don’t have very good data on their incidence by industry. I propose sticking with the FAF empty truck factors for now.

• The cargo weight distributions by distance look encouraging and a lot more useful than the FAF data. Sampling from these distributions will be a lot more straight-forward than the convoluted process used in the FAF to translate tons to truck equivalents by trailer type, and then from that back into vehicle types. Using these distributions will definitely result in more robust vehicle weights.

I’m testing code to implement sampling from these data right now, which I hope to have finished testing by this weekend. I will send a summary of truckload equivalents using this approach versus the FAF approach we have been using early next week.

[bettinardi]

Not sure what issue to put this error on, but deciding to put it here. To deal with this issue, CT has been getting updated to the latest FAF, along with several other revisions. CT is being tested in more current years, but as I am attempting the current repo copy, it is failing in 2010 in the full model stream with the following issue. This is just a note to capture this and correct in the repo before attempting a full SWIM run again:

# Process the raw FAF data and transform into format that can be used by the functions below
faf.flow.data = swimctr::preprocess_faf4_database(RTP[["faf.flow.data"]], 
                                            2015,#as.numeric(RTP[["t.year"]]) +  as.numeric(RTP[["base.year"]]),
                                          FALSE,
                                            RTP[["ct.oregon.regions"]],
                                            RTP[["ct.oregon.outer.regions"]])
Processing FAF flow data from spec_tbl_df contents in RTP[["faf.flow.data"]]
FAF data from 2015 is closest to target year 2015
Error in strsplit(internal_regions, split = ",") : non-character argument
Calls: <Anonymous> -> unlist -> strsplit
Execution halted

[goreaditya]

@bettinardi - I noticed that the test script, that I was using to test changes to swimctr package, got pushed to the repository. This script was manually passing the year 2015 to the FAF preprocessing function instead of reading it from the "ct.properties" file. This issue can be fixed by simply deleting 2015,# from 2015,#as.numeric(RTP[["t.year"]]) + as.numeric(RTP[["base.year"]]), line. The error produced is unrelated to this change in the script. The commit https://github.com/tlumip/tlumip_dependencies/tree/2a9f7a725ad6bb83cc72e2a4c5c096fd9118db79 introduced some functionality in "swimctr" package that was later pulled out in the latest commit https://github.com/tlumip/tlumip_dependencies/tree/53148a66592163315c21864e908826e81ac55c1e. If you pull the latest "tlumip_dependenices - master" then it should work with the latest "tlumip - working-ver-26" branch.

[bettinardi]

Adding some notes and thoughts here so that I don't lose them - Part of the on going confusion around using CT is - what dollars are CT in. Related (although it might not seem like it at first) is that SWIM has historically had too many trucks assigned and now with the latest tests with FAF 5.2 the issue has gotten worse. This relates to "what dollars are SWIM in" because if we don't deflate FAF dollars to the SWIM operating year, newer FAF will likely have higher values flowing due to inflation and if FAF data gets put into CT without adjusting back to the year CT is assumed to operate in - then it might look like more value/goods is/are flowing, when in actuality, it's just that the same amount of goods are costing more (have more value) due to inflation.

Another note here is that I have never felt confident that I have been able to understand if CT / FAF is in real or nominal dollars. I think i have heard in the past that they are real, but that has never made sense, and in my latest CT analysis I have numbers that seem to prove to me that the value output coming out of CT / FAF are nominal (they don't have built in inflation assumptions). Assuming that I am correct in this, that's more evidence that we need to deflate the latest FAF values back to the year that CT was built on or assumes (and would help correct the over assignment of trucks).

What I have found recently, is that if I sum all of the FAF value flowing in SWIM in CT in 2040 and divide by 2019 value I get 1.45, or 45% growth over 21 years. When I do the same with FAF tons I get 1.40 (40% growth in tons). If CT were in real dollars that would imply that no additional goods were flowing, because inflation would have accounted for the 45% growth in the value of goods. Additionally, since Tons don't inflate like dollars, if FAF was in real dollars then the value would have to have grown much faster than tons.

Anyway, it seems clear that the dollars in FAF are in some year - so when we update FAF inputs, a critical role piece is to document the year the dollars/values are assumed to be in, and then deflate them back to what SWIM is built in, which is 2010 dollars.

[bettinardi]

I believe this section of the wiki resolves this issue - https://github.com/tlumip/tlumip/wiki/CT#updating-faf-inputs-to-ct-model

The remaining issues discussed in this thread are either being addressed in different ways or are now obsolete