search

IMCR-Hackathon / Hackathon-Central-2018

Command center for IMCR Hackathon participants to share ideas, coordinate teams, develop projects and access all logistics information
3 stars 0 forks source link

Develop Best Practices Guidelines for Data Cleaning #7

Open reblake opened 6 years ago

reblake commented 6 years ago

Using existing data for re-analysis or new synthesis research is challenging because of the large amounts of time and effort needed to clean the data. Having best practices guidelines would streamline this process for data users and reduce the time spent cleaning data before analysis can proceed. These guidelines would ideally include a script that addresses the most common data cleaning tasks/problems/issues, as well as a document (or maybe another script?) that gives more details of specific problems (ex: scraping data from the web, using NETCDF files, extracting data from Excel workbook). This would all be done in R.

Common issues include: Capitalization, misspelling, white space, dots, missing values not represented by NA, abbreviated text, different names for the same sites/species, metadata in the data table, etc.

jhp7e commented 6 years ago

https://academic.oup.com/bioscience/article/63/7/574/289294 may be helpful in identifying issues.