Mixed methods approach and interactive dashboard to analyse research impact through Impact Case Studies submitted to the UK's Research Excellence Framework (REF) 2021.
Added decorators to all data wrangling functions to monitor number of rows post function
Added clean_dep_level function as this was missing in current milestone and required to make cleaN_ref_dep_data.xlsx which is used in load_dept_vars
Fixed incorrect merging in load_dept_vars which was merging on Unit of assessment number and inst_id rather than uoa_id and inst_id (and commented out code on lines 568:571 that made uoa_id identical to Unit of assessment number. @crahal double check whether the incorrect merge was used in previous iterations, leading to some duplicate rows (correct number of rows is 6371)
took section_columns argument used in the text-mining functions out of each individual function and as a parameter
included sys.argv approach such that different datasets can be generated (-bq collects dimensions data and makes the data paper level, -top estimates the topic model (to be finished yet in current script), -tm adds text-mining data to the dataset.
Added sys.argv approach to output path of the final file. If none is provided, file simply outputs at the original location final_path / 'enhanced_ref_data.csv' unless the file is already there, in which case it only overwrites if -f is provided to the call. If a path is provided, the output file is written there.
Streamlined data pre-processing script.
Main features:
clean_dep_level
function as this was missing in current milestone and required to makecleaN_ref_dep_data.xlsx
which is used inload_dept_vars
load_dept_vars
which was merging onUnit of assessment number
andinst_id
rather thanuoa_id
andinst_id
(and commented out code on lines 568:571 that madeuoa_id
identical toUnit of assessment number
. @crahal double check whether the incorrect merge was used in previous iterations, leading to some duplicate rows (correct number of rows is 6371)section_columns
argument used in the text-mining functions out of each individual function and as a parametersys.argv
approach such that different datasets can be generated (-bq
collects dimensions data and makes the data paper level,-top
estimates the topic model (to be finished yet in current script),-tm
adds text-mining data to the dataset.sys.argv
approach to output path of the final file. If none is provided, file simply outputs at the original locationfinal_path / 'enhanced_ref_data.csv'
unless the file is already there, in which case it only overwrites if-f
is provided to the call. If a path is provided, the output file is written there.Proposed approach:
python3 make_enhanced_data.py [path] [-bq] [-tm] [-top] [-f]