Remaining bugs to fix

[calmc]

Bugs and other functional code things remaining to address. Might be helpful to break off into separate issues, depending on how things go.

• [x] 'noncombustion.py' is incomplete for downloading and formatting data.

• [ ] Paths

  • [x] Paths to local directory (46 instances) for yamls, creating output data, etc. Find and fix
  • [x] Path to Household.csv, data for the residential sector. Seems to also be an problem for the commercial sector (see BadZipFile error below).

• [ ] `divide_multilevel'

  • On L869, to_drop drops the levels from the numerator that aren't shared by the denominator. Doesn't currently work for all sectors (i.e. not consistently dropping the correct levels from the multiindex columns in the numerator dataframe for all sectors).
  • In applying weights, when ratio is more than one column and columns don't match the weights, how to apply weights? This case is handled by the else statement.
  • When the weights are calculated, the first if statement should check the column names and levels of the weights and ratio (the numerator and denominator dataframes), not just the dimensions of w and ratio.
  • decomposition_results method then calculates the indexed results, which can then be passed to higher levels of aggregation.
  • If the ratio and w don't align, and if the ratio is a single column, then divide by the base year to create an index. It's not necessary to apply weights if ratio only has a single column.

• [ ] Renaming columns in a multiindex results dataframe isn't working here. Example-- commercial sector runs through, but columns aren't renamed. Might be able to f.columns = [t], where t is a string. Also look into built-in Pandas method, .droplevel, for making the columns a single level.

• [ ] transportation.py

  • [x] has an empty dataframe for air energy use. Unsure where this is. data_dict appears to return all activity and energy data for passenger and freight air. Now consider this fixed
  • [x] "TypeErrror: 'NoneType' object is not iterable" now being thrown when running decomposition.
  • [x] Results: huge spike in activity data for mid-2000s (see results graphs). Probably an error somewhere in input data
  • [x] Strange TypeError now in LMDI.py when running transportation.py
  • [ ] Error in pipeline aggregation (zero values in series). See aggregate_level_data and df_utils.create_total_column.

• [ ] subscript variable is currently set as i. Instead, should be the subscript from the yaml that has names and units. Suggestion is to create a new variable in the yaml, something like:

  to_weight: 
  - total.All_Transportation

• [x] commercial.py BadZipFile: File is not a zip file.

• [ ] Something fishy is going on with additive LMDI results (at least for transportation). Doesn't appear that activity and structure effects are calculated correctly.

• [ ] Residential activity data for 2015 has an error.

[calmc]

Fixed the BadZipFile and path to household.csv problems with 0c4fc24.