Mixed methods approach and interactive dashboard to analyse research impact through Impact Case Studies submitted to the UK's Research Excellence Framework (REF) 2021.
Added decorators to all data wrangling functions to monitor number of rows post function
Added clean_dep_level function as this was missing in current milestone and required to make cleaN_ref_dep_data.xlsx which is used in load_dept_vars
Fixed incorrect merging in load_dept_vars which was merging on Unit of assessment number and inst_id rather than uoa_id and inst_id (and commented out code on lines 568:571 that made uoa_id identical to Unit of assessment number. @crahal double check whether the incorrect merge was used in previous iterations, leading to some duplicate rows (correct number of rows is 6371)
took section_columns argument used in the text-mining functions out of each individual function and as a parameter
included sys.argv approach such that different datasets can be generated (-bq collects dimensions data and makes the data paper level, -top estimates the topic model (to be finished yet in current script), -tm adds text-mining data to the dataset.
Added sys.argv approach to output path of the final file. If none is provided, file simply outputs at the original location final_path / 'enhanced_ref_data.csv' unless the file is already there, in which case it only overwrites if -f is provided to the call. If a path is provided, the output file is written there.
Streamlined data pre-processing script.
Main features:
clean_dep_levelfunction as this was missing in current milestone and required to makecleaN_ref_dep_data.xlsxwhich is used inload_dept_varsload_dept_varswhich was merging onUnit of assessment numberandinst_idrather thanuoa_idandinst_id(and commented out code on lines 568:571 that madeuoa_ididentical toUnit of assessment number. @crahal double check whether the incorrect merge was used in previous iterations, leading to some duplicate rows (correct number of rows is 6371)section_columnsargument used in the text-mining functions out of each individual function and as a parametersys.argvapproach such that different datasets can be generated (-bqcollects dimensions data and makes the data paper level,-topestimates the topic model (to be finished yet in current script),-tmadds text-mining data to the dataset.sys.argvapproach to output path of the final file. If none is provided, file simply outputs at the original locationfinal_path / 'enhanced_ref_data.csv'unless the file is already there, in which case it only overwrites if-fis provided to the call. If a path is provided, the output file is written there.Proposed approach:
python3 make_enhanced_data.py [path] [-bq] [-tm] [-top] [-f]