Following is a summary of changes in this pull request:
==================
VISIONEVAL PACKAGE
Fixed capabilities for loading a datastore and removed datastore referencing
The capability for loading a datastore (initializeModel function) was incomplete and could not be used during a model run. This was corrected so that an existing datastore can be loaded into a new model run. The datastore to be loaded can be one that exists in the model run directory or that is in a different directory. Here is an example of how it can be used in managing scenarios:
1) A base year model scenario and be calibrated and validated.
2) Several future land use scenarios (e.g. synthesize households and assign to Bzones that have various land use characteristics) can be developed/run. Each land use model run loads the base year model datastore.
3) Several transportation policy scenarios are developed/run for each land use scenario. Each policy scenario run loads the land use scenario datastore that it applies to.
Enabling this functionality required modifying several visioneval functions:
The initializeModel function was modified to copy the datastore to be loaded and the corresponding model state file to enable combining with the model data that are to be added. Consistency checks between the loaded datastore and model run are made. Loading is done in a way that if errors are found, an existing datastore and model state file is not altered.
The initDatastoreGeography function was modified so that is works when loading an existing datastore.
The initDatastoreRD and initDatastoreH5 functions were modified to enable appending Groups to an existing datastore.
The simDataTransactions function was modified so that it works with loaded datastore.
The capability to use datastore referencing was removed. This did not work.
Fix handling of error and warning messages returned by modules
Modify the runModule function so that when a module returns a warning and/or error message, a message to that effect is always printed to the console and that the error/warning message(s) returned by the module are properly written to the model run log. Correct addWarningMsg function documentation.
Loading/fetching module data
Modify loadPackageDataset function so that a DefaultPackage name can be specified. If the DatasetsByPackage_df is not in the ModelState_ls as when a module is under development, the function will use the named DefaultPackage to load the dataset from. This simplified the process of module development.
To simplify module debugging during model runtime, the fetchModuleData function was added. This function can be called at the command line when a model run fails to retrieve the data that the failed module used (i.e. the L data list). Then module code can be stepped through to identify where the error occurred and the state of variables used in the computation that errors. (What I do is use the 'edit()' function to retrieve the module code and paste into a temporary file that I step through at the command line.)
Correct bug in ordering of input data
Corrected a bug in processModuleInputs function which caused input values to be put in datastore in wrong geographic order when not all input files have data in same order. Also changed function so that it would no longer throw an error when input file contains data for years that are not model run year. This makes it easier for users to set up their data for all years they may run but only run for some of those years. The function now simply ignores the non-model-run years when pushing data into the datastore.
Correct error with using HDF5 datastore
After coming up with the Rdata (RD) datastore format, testing with HDF5 (H5) datastores fell by the wayside. Testing datastore loading (see above) with H5 datastores revealed errors. These were corrected so that H5 datastores will work properly.
Remove datastore query capability from package
Removed the reporting.R script which defined the readDatastoreTables function. This functionality and more is now included in the VEReports package.
=================
VEREPORTS PACKAGE
The QueryDatastore.R script was added to the VEReports package. This script defines several functions that are used to retrieve datasets and compute performance measures from datasets in one or more datastore tables. The summarizeDatasets function enables users to specify computations using R expression syntax and can pull datasets from multiple tables and can also join those datasets using specified key.
=======================
VESIMHOUSEHOLDS PACKAGE
In CreateHouseholds, correct assignment for Prsn_AzAp and Prsn_AzAg
In CreateHouseholds module, add check whether household population inputs by age group and group quarters population inputs by age group are non-integer values. If so, round to nearest whole number and write out warning messages to user identifying this condition.
Modify PredictWorkers module function so that it properly addresses employment adjustment factors and checks for the possibility that adjusted number of workers exceeds the number of people for an Azone and age category. Add output of error messages.
=================
VELANDUSE PACKAGE
Enable users to specify the degree that parking pricing is applied to non-work trips
Modified the AssignParkingRestrictions module in the VELandUse package to add the capability for users to specify the proportion on non-work trips to each Bzone which pay for parking. Revised 'bzone_parking.csv' test file to include PropNonWrkTripPay field needed for the revised AssignParkingRestrictions module which allows users to specify the proportion of non-work trips (e.g. shopping) that have to pay for parking.
Correct bug in CalculateUrbanMixMeasure module which causes error for very high population density Bzones
Modified the matchMixTarget function in CalculateUrbanMixMeasure function to replace any infinite odds with 1e6 so that the probabilities are 1 rather than NA.
Modified the Calculate4DMeasures module, to add a check whether any Bzone population densities (D1B) are greater than 100 persons per acre and output a warning message (addWarningMsg) identifying the year and Bzones with high population density.
Clean up files/scripts causing warnings during package checks
Move the HhAllocationModelCoeff.rda file and the HhAllocationModelCoeff.rda file which are inputs to model creation from the data directory to the data-raw directory because they are not documented (causes a warning).
====================
VESIMLANDUSE PACKAGE
Enable users to specify the degree that parking pricing is applied to non-work trips
Modified the AssignParkingRestrictions module to enable users to specify the proportion of non-work trips that pay for parking by area type.
Correct bug in AssignCarSvcAvailability module
Correct error in identifying high car service Bzones by ensuring that sapply which is argument to do.call always returns a list rather than simplifying to vector.
Modify SimulateEmployment module to recognize town workers may work outside of the Azone than where they live
Modify SimulateEmployment module to recognize town workers may work outside of the Azone than where they live. The module uses a new data field in the 'azone_wkr_loc_type_prop.csv' called TowJobWkrRatio which is used to allocate employment to the town location type in each Azone. This is necessary to calculate the correct town activity density and jobs/housing mix given that there can be differences between the numbers of resident workers and jobs in town location type. Modify calcNumJobsByLocType to adjust Azone workers assigned to town location type to calculate Azone town employment.
Add input file consistency checks
Add several input file consistency checks to the Initialize module so that location type proportions add up, proportions are consistent with the Azone being associated with an Marea or not, land allocation is consistent with activity allocation, and densities are at levels the enable SimBzone density distributions to be calculated. Add in the loading of TownJobWkrRatio values. Add getting population and employment data used to calculate densities. Add function definitions for splitIntegers, calcNumHhByLocType, and calcNumJobsByLocType which were moved from the CreateSimBzones module.
Change procedures/algorithms for computing Bzone activity density distributions
Modify the allocateActivityToSimBzones function to change the process for shifting the density distribution so that a viable activity density distribution can be computed if the density for an urbanized area is higher than the highest observed urbanized area density in the Smart Location Database. Correct the call to cut() in the calcDensityDistribution function to set include.lowest = TRUE. Not doing this caused error when density was at the lowest value.
Modify the CreateVehicleTable.R script to add Bzone to Set spec. This was an oversight in developing the Vehicle table. Having it makes it simpler to query the data and join it with other datasets.
AssignDrivers module
Refactor AssignDrivers module code to add error check and message when driver adjustment factor would create more drivers than population. Also change the method for adjusting number of drivers to be more efficient.
Modify the DivertSovTravel module to write out the calculated values of household SOV DVMT proportions of household DVMT. This will be useful in the future for calculating the effects of shared use of car services.
RPAT code changes
Document undocumented PhevModelData_ls object in CalculateTravelDemand module that was causing a package check warning.
Modify the Initialize module to add caveat to the warning to indicate that it only applies to VE-RPAT models, not to VE-RSPM or VE-State models
Correct bug causing NA values for MPG when no hybrid powertrain
NA values were being output for MPG and related energy and emissions measures were calculated if no hybrid powertrains were present (affected car services, commercial services, trucks, and public transit). Fuel consumption is calculated from consumption by ICEV and HEV. If HEV is NA, then result was NA. This problem was eliminated so that if there are no HEVs, then they are ignored in the calculation.
Correct Set specifications in CalculatePtranEnergyAndEmissions and CalculateComEnergyAndEmissions as rates
Change Set specifications in CalculatePtranEnergyAndEmissions module for BusGGE, RailGGE, VanGGE, BusKWH, RailKWH, and VanKWH from TYPE = "energy" to TYPE = "compound" to correctly identify energy consumption as daily rates. Likewise change specifications of BusCO2e, RailCO2e, and VanCO2e from TYPE = "mass" to TYPE = "compound" to correctly identify CO2e production as daily rates.
Change the Set specifications in the CalculateComEnergyAndEmissions module for ComSvcUrbanGGE, ComSvcNonUrbanGGE, HvyTrkUrbanGGE, ComSvcUrbanKWH, ComSvcNonUrbanKWH, HvyTrkUrbanKWH, HvyTrkNonUrbanGGE, HvyTrkUrbanGGE, HvyTrkNonUrbanKWH, and HvyTrkUrbanKWH from TYPE = "energy" to TYPE = "compound" to correctly identify energy consumption as daily rates. Likewise change specifications for ComSvcUrbanCO2e, ComSvcNonUrbanCO2e, HvyTrkUrbanCO2e, HvyTrkNonUrbanCO2e, and HvyTrkUrbanCO2e from TYPE = "mass" to TYPE = "compound" to correctly identify CO2e production as daily rates.
Correct commercial service vehicle powertrain characteristics
Correct calculation of average commercial service vehicle powertrain characteristics to handle case where average vehicle age results in earliest model year that is older than the oldest model year in the vehicle characteristics table.
Correct delay output specifications
Correct the UNITS and DESCRIPTION Set specs for LdvTotDelay, HvyTrkTotDelay, and BusTotDelay
======================================================
RESOLVE SPECIFICATIONS INCONSISTENCIES BETWEEN MODULES
Implementing and testing the datastore loading functionality and other changes to modules resulted in identifying a number of specifications inconsistencies between modules. These were corrected.
Following is a summary of changes in this pull request:
================== VISIONEVAL PACKAGE
Fixed capabilities for loading a datastore and removed datastore referencing
The capability for loading a datastore (initializeModel function) was incomplete and could not be used during a model run. This was corrected so that an existing datastore can be loaded into a new model run. The datastore to be loaded can be one that exists in the model run directory or that is in a different directory. Here is an example of how it can be used in managing scenarios: 1) A base year model scenario and be calibrated and validated. 2) Several future land use scenarios (e.g. synthesize households and assign to Bzones that have various land use characteristics) can be developed/run. Each land use model run loads the base year model datastore. 3) Several transportation policy scenarios are developed/run for each land use scenario. Each policy scenario run loads the land use scenario datastore that it applies to.
Enabling this functionality required modifying several visioneval functions:
The capability to use datastore referencing was removed. This did not work.
Fix handling of error and warning messages returned by modules
Modify the runModule function so that when a module returns a warning and/or error message, a message to that effect is always printed to the console and that the error/warning message(s) returned by the module are properly written to the model run log. Correct addWarningMsg function documentation.
Loading/fetching module data
Modify loadPackageDataset function so that a DefaultPackage name can be specified. If the DatasetsByPackage_df is not in the ModelState_ls as when a module is under development, the function will use the named DefaultPackage to load the dataset from. This simplified the process of module development.
To simplify module debugging during model runtime, the fetchModuleData function was added. This function can be called at the command line when a model run fails to retrieve the data that the failed module used (i.e. the L data list). Then module code can be stepped through to identify where the error occurred and the state of variables used in the computation that errors. (What I do is use the 'edit()' function to retrieve the module code and paste into a temporary file that I step through at the command line.)
Correct bug in ordering of input data
Corrected a bug in processModuleInputs function which caused input values to be put in datastore in wrong geographic order when not all input files have data in same order. Also changed function so that it would no longer throw an error when input file contains data for years that are not model run year. This makes it easier for users to set up their data for all years they may run but only run for some of those years. The function now simply ignores the non-model-run years when pushing data into the datastore.
Correct error with using HDF5 datastore
After coming up with the Rdata (RD) datastore format, testing with HDF5 (H5) datastores fell by the wayside. Testing datastore loading (see above) with H5 datastores revealed errors. These were corrected so that H5 datastores will work properly.
Remove datastore query capability from package
Removed the reporting.R script which defined the readDatastoreTables function. This functionality and more is now included in the VEReports package.
================= VEREPORTS PACKAGE
The QueryDatastore.R script was added to the VEReports package. This script defines several functions that are used to retrieve datasets and compute performance measures from datasets in one or more datastore tables. The summarizeDatasets function enables users to specify computations using R expression syntax and can pull datasets from multiple tables and can also join those datasets using specified key.
======================= VESIMHOUSEHOLDS PACKAGE
================= VELANDUSE PACKAGE
Enable users to specify the degree that parking pricing is applied to non-work trips
Modified the AssignParkingRestrictions module in the VELandUse package to add the capability for users to specify the proportion on non-work trips to each Bzone which pay for parking. Revised 'bzone_parking.csv' test file to include PropNonWrkTripPay field needed for the revised AssignParkingRestrictions module which allows users to specify the proportion of non-work trips (e.g. shopping) that have to pay for parking.
Correct bug in CalculateUrbanMixMeasure module which causes error for very high population density Bzones
Modified the matchMixTarget function in CalculateUrbanMixMeasure function to replace any infinite odds with 1e6 so that the probabilities are 1 rather than NA. Modified the Calculate4DMeasures module, to add a check whether any Bzone population densities (D1B) are greater than 100 persons per acre and output a warning message (addWarningMsg) identifying the year and Bzones with high population density.
Clean up files/scripts causing warnings during package checks
Move the HhAllocationModelCoeff.rda file and the HhAllocationModelCoeff.rda file which are inputs to model creation from the data directory to the data-raw directory because they are not documented (causes a warning).
==================== VESIMLANDUSE PACKAGE
Enable users to specify the degree that parking pricing is applied to non-work trips
Modified the AssignParkingRestrictions module to enable users to specify the proportion of non-work trips that pay for parking by area type.
Correct bug in AssignCarSvcAvailability module
Correct error in identifying high car service Bzones by ensuring that sapply which is argument to do.call always returns a list rather than simplifying to vector.
Modify SimulateEmployment module to recognize town workers may work outside of the Azone than where they live
Modify SimulateEmployment module to recognize town workers may work outside of the Azone than where they live. The module uses a new data field in the 'azone_wkr_loc_type_prop.csv' called TowJobWkrRatio which is used to allocate employment to the town location type in each Azone. This is necessary to calculate the correct town activity density and jobs/housing mix given that there can be differences between the numbers of resident workers and jobs in town location type. Modify calcNumJobsByLocType to adjust Azone workers assigned to town location type to calculate Azone town employment.
Add input file consistency checks
Add several input file consistency checks to the Initialize module so that location type proportions add up, proportions are consistent with the Azone being associated with an Marea or not, land allocation is consistent with activity allocation, and densities are at levels the enable SimBzone density distributions to be calculated. Add in the loading of TownJobWkrRatio values. Add getting population and employment data used to calculate densities. Add function definitions for splitIntegers, calcNumHhByLocType, and calcNumJobsByLocType which were moved from the CreateSimBzones module.
Change procedures/algorithms for computing Bzone activity density distributions
Modify the allocateActivityToSimBzones function to change the process for shifting the density distribution so that a viable activity density distribution can be computed if the density for an urbanized area is higher than the highest observed urbanized area density in the Smart Location Database. Correct the call to cut() in the calcDensityDistribution function to set include.lowest = TRUE. Not doing this caused error when density was at the lowest value.
=========================== VEHOUSEHOLDVEHICLES PACKAGE
CreateVehicleTable module
Modify the CreateVehicleTable.R script to add Bzone to Set spec. This was an oversight in developing the Vehicle table. Having it makes it simpler to query the data and join it with other datasets.
AssignDrivers module
Refactor AssignDrivers module code to add error check and message when driver adjustment factor would create more drivers than population. Also change the method for adjusting number of drivers to be more efficient.
========================= VEHouseholdTravel PACKAGE
Small enhancement to DivertSovTravel module
Modify the DivertSovTravel module to write out the calculated values of household SOV DVMT proportions of household DVMT. This will be useful in the future for calculating the effects of shared use of car services.
RPAT code changes
Document undocumented PhevModelData_ls object in CalculateTravelDemand module that was causing a package check warning.
Modify the Initialize module to add caveat to the warning to indicate that it only applies to VE-RPAT models, not to VE-RSPM or VE-State models
============================= VEPOWERTRAINSANDFUELS PACKAGE
Correct the check for default value files having data for mandatory years in the LoadDefaultValues module of the VEPowertrainsAndFuels package.
=========================== VETRAVELPERFORMANCE PACKAGE
Correct bug causing NA values for MPG when no hybrid powertrain
NA values were being output for MPG and related energy and emissions measures were calculated if no hybrid powertrains were present (affected car services, commercial services, trucks, and public transit). Fuel consumption is calculated from consumption by ICEV and HEV. If HEV is NA, then result was NA. This problem was eliminated so that if there are no HEVs, then they are ignored in the calculation.
Correct Set specifications in CalculatePtranEnergyAndEmissions and CalculateComEnergyAndEmissions as rates
Change Set specifications in CalculatePtranEnergyAndEmissions module for BusGGE, RailGGE, VanGGE, BusKWH, RailKWH, and VanKWH from TYPE = "energy" to TYPE = "compound" to correctly identify energy consumption as daily rates. Likewise change specifications of BusCO2e, RailCO2e, and VanCO2e from TYPE = "mass" to TYPE = "compound" to correctly identify CO2e production as daily rates. Change the Set specifications in the CalculateComEnergyAndEmissions module for ComSvcUrbanGGE, ComSvcNonUrbanGGE, HvyTrkUrbanGGE, ComSvcUrbanKWH, ComSvcNonUrbanKWH, HvyTrkUrbanKWH, HvyTrkNonUrbanGGE, HvyTrkUrbanGGE, HvyTrkNonUrbanKWH, and HvyTrkUrbanKWH from TYPE = "energy" to TYPE = "compound" to correctly identify energy consumption as daily rates. Likewise change specifications for ComSvcUrbanCO2e, ComSvcNonUrbanCO2e, HvyTrkUrbanCO2e, HvyTrkNonUrbanCO2e, and HvyTrkUrbanCO2e from TYPE = "mass" to TYPE = "compound" to correctly identify CO2e production as daily rates.
Correct commercial service vehicle powertrain characteristics
Correct calculation of average commercial service vehicle powertrain characteristics to handle case where average vehicle age results in earliest model year that is older than the oldest model year in the vehicle characteristics table.
Correct delay output specifications
Correct the UNITS and DESCRIPTION Set specs for LdvTotDelay, HvyTrkTotDelay, and BusTotDelay
====================================================== RESOLVE SPECIFICATIONS INCONSISTENCIES BETWEEN MODULES
Implementing and testing the datastore loading functionality and other changes to modules resulted in identifying a number of specifications inconsistencies between modules. These were corrected.