Find External Dataset - Githubissues

jinchen1036 / DataScience_Project

1 stars 0 forks source link

Find External Dataset #8

Closed jinchen1036 closed 5 years ago

jinchen1036 commented 5 years ago

Requirement

Find the external dataset that can be used for improve the model performance
Look into different data source
Can match fields
- Zip Code
- BIN
- BBL

Acceptance Criteria

Should find the external datasets
Test the matching ratio of the external dataset with the existed train and test datasets.

xzhu0706 commented 5 years ago

This dataset contains the estimated rental income, expense, and market value of condo & coop in NYC. We can match this dataset with bbl: https://data.cityofnewyork.us/City-Government/DOF-Cooperative-Comparable-Rental-Income-Citywide-/myei-c3fa

jinchen1036 commented 5 years ago

@shinzhao any dataset you found? If not, we will just use the income dataset.

jinchen1036 commented 5 years ago

@xzhu0706 Please see test result of the external dataset below, many rows don't match the based on the bbl. Analysis Result:

9413 unique bbl of the rent dataset (all train, test1, test2, test3 with total 18000)
16187 unique bbl from the external dataset (originally with 118266 bbl)
Only 2082 bbl match out of 9413.

Advice

Please look into the dataset with both bbl and bin information for better match as bbl can be duplicated
Also look into dataset match with zip code.

xzhu0706 commented 5 years ago

https://www.kaggle.com/jakerohrer/zip-codes-and-stats

jinchen1036 commented 5 years ago

This dataset is very outdate compare to some of data in 2017. https://www.incomebyzipcode.com/newyork Please check if we can extract the income information from above link.

jinchen1036 commented 5 years ago

Finish extract the zip code income from https://www.incomebyzipcode.com/newyork. Reference to the ExternalDF Folder

Result Analysis

Total 154 unique zip code (base on train, test1 and test 3)
One not find zip code: 11249
Result in 15787 not null out of 16000