clean dataset - Githubissues

[x] drop na from "ConvertedCompYearly" column
[x] only use "Employed full-time"
[x] convert "Less than 1 year" and "More than 50 years" to 0.5 and 51 for "YearsCode" and "YearsCodePro" columns
[x] remove countries that have below a certain threshold (500?)
[x] combine US-country and UK-state?
- Michigan, USA -> United States of America:Michigan
- Just USA, no state -> United States of America:
- this may split US into too many different features, most without enough data. might not be able to do this. could potentially leave US states instead. or just use country.
[x] countries (one hot encoding? or too many countries?)
[x] simplify education to: No college, Associates, Bachelors, Masters, PhD and convert to 0,1,2,3,4 (manual, or via LabelEncoder from sklearn.preprocessing)
- [x] change to ordinal encoding? example https://github.com/rachittoshniwal/machineLearning/blob/master/ordinal%20encoder.ipynb
[x] exclude salaries above/below certain range. may need to probe to find the amounts ~~- [ ] handle DevType (simplify, one hot encode)~~
[x] handle LanguageHaveWorkedWith (simplify, one hot encode)
[x] handle Age. currently strings representing rangs ('25-34 years old'). '25-34 years old' -> 29.5 (midpoint). make "prefer not to say" the mean age.
[ ] check for multicollinearity problems with one hot encoded features https://www.analyticsvidhya.com/blog/2020/03/one-hot-encoding-vs-label-encoding-using-scikit-learn/
[ ] move encoding out of initial data loading (do independently on train/test) https://datascience.stackexchange.com/questions/107714/encoding-before-vs-after-train-test-split
- all? just one hot encoded? this seems to suggest that it wouldn't be a problem for ordinal encoding https://datascience.stackexchange.com/questions/78146/does-label-encoding-an-entire-dataset-cause-data-leakage

BlairCurrey / swe-salary-predictor

clean dataset #1