Add functions to set format and storage options of spark dataframes when calling spark namematching save. Example usage: nm_obj.write().format('parquet').options(**options_dict).save(path)
Get rid of np.array_split on pd series future deprecation warning
Remove any extra whitespace from processed names for n-char indexing.
Pass thru correct-match column if present in SparkAggregator; useful for accuracy testing.
mbaak
commented
4 months ago
nm_obj.write().format('parquet').options(**options_dict).save(path)