[API Change] Long and Wide Pandas Conversions

[BenTaylor-TfN]

Current methods for converting pandas DataFrame between wide and long format do not use the index columns correctly. Instead, columns that would be the index are treated as any other column. It could be beneficial to convert these methods to use Index columns instead.

This change should result in less arguments needed for function calls, and slightly faster operation as more can be inferred (never mind the internal pandas speed gains). The following functions would be impacted:

• caf.toolkit.pandas_utils.df_handling.
  • long_to_wide_infill()
  • Less arguments needed, index / col will be inferred
  • long_df_to_wide_ndarray()
  • Less arguments needed, index / col will be inferred
  • long_product_infill()
  • Will return the df without resetting the index, leaving the columns in the index
  • wide_to_long_infill()
  • Will return the df without resetting the index, leaving the columns in the index

If going ahead with this update we need to be careful how the API change is fed back into users. Internal changes should easily be flagged by test failures, but other CAF packages may need updating to take these changes into account.

[isaac-tfn]

On the third point, is this supposed to make a multiindex with every possible combination of index values from different levels? I've done the other three will push up now and link to this issue.

[BenTaylor-TfN]

On the third point, is this supposed to make a multiindex with every possible combination of index values from different levels?

That's right. There's some added functionality to make some assumptions where a defined list isn't given too.

[isaac-tfn]

On the third point, is this supposed to make a multiindex with every possible combination of index values from different levels?

That's right. There's some added functionality to make some assumptions where a defined list isn't given too.

OK I'll rewrite it now, having some trouble with black but hopefully have stuff pushed up by the end of the day