search

sfbrigade / datasci-firerisk

This project attempts to model and acquire data from SF OpenData - and other sources - to predict the relative risk of fire in San Francisco’s buildings and public spaces.
http://codeforsanfrancisco.org/projects/SF-Fire-Risk-Project
10 stars 9 forks source link

Model Runs #13

Open sewardlee337 opened 7 years ago

sewardlee337 commented 7 years ago

This is a temporary place to record results of model runs.

Please record the following:

yamariva2000 commented 7 years ago

Model run binary classifier 9/20/2017 based on commit 3bf815e. Test/Train recall values good. However, cross validation values are rather volatile. Also included is feature importances list in descending order. Size, value, and number of units, as well as tenderloin location seem to be important. Data set used was based on file from 9/06.

/home/kel/anaconda3/bin/python /home/kel/fire2/multiclass.py
training accuracy 0.97
testing accuracy 0.92
labels ['0 No incident', '1 Incident']
               precision    recall  f1-score   support

0 No incident       0.95      0.96      0.96     56325
   1 Incident       0.71      0.62      0.67      8117

  avg / total       0.92      0.92      0.92     64442

['Property_Area', 'Assessed_Improvement_Val', 'Land_Value', 'age', 'Tot_Rooms', 'Num_Rooms', 'Property_Code_Des_D - DWELLING', 'Num_Units', 'Num_Stories', 'Num_Bathrooms', 'Num_Bedrooms', 'Property_Code_Des_F - FLATS & DUPLEX', 'Neighborhood_TENDERLOIN', 'Property_Code_Des_C - COMMERCIAL STORES', 'Neighborhood_SUNSET/PARKSIDE', 'Neighborhood_WEST OF TWIN PEAKS', 'Neighborhood_OUTER RICHMOND', 'Neighborhood_SOUTH OF MARKET', 'Neighborhood_MISSION', 'Property_Code_Des_O - OFFICE', 'Neighborhood_HAYES VALLEY', 'Property_Code_Des_F2 - FLAT & STORE', 'Property_Code_Des_I - INDUSTRIAL', 'Neighborhood_CASTRO/UPPER MARKET', 'Neighborhood_INNER RICHMOND', 'Neighborhood_MARINA', 'Neighborhood_PACIFIC HEIGHTS', 'Neighborhood_NOE VALLEY', 'Neighborhood_BERNAL HEIGHTS', 'Neighborhood_INNER SUNSET', 'Neighborhood_HAIGHT ASHBURY', 'Neighborhood_RUSSIAN HILL', 'Neighborhood_POTRERO HILL', 'Property_Code_Des_Z - CONDOMINIUM', 'Neighborhood_EXCELSIOR', 'Neighborhood_NOB HILL', 'Neighborhood_LONE MOUNTAIN/USF', 'Neighborhood_PRESIDIO HEIGHTS', 'Neighborhood_PORTOLA', 'Neighborhood_NORTH BEACH', 'Neighborhood_CHINATOWN', 'Neighborhood_FINANCIAL DISTRICT/SOUTH BEACH', 'Neighborhood_OUTER MISSION', 'Neighborhood_GLEN PARK', 'Neighborhood_OCEANVIEW/MERCED/INGLESIDE', 'Neighborhood_WESTERN ADDITION', 'Neighborhood_VISITACION VALLEY', 'Property_Code_Des_H2 - HOTELS - OTHER', 'Property_Code_Des_AC - APARTMNT & COMMERCIAL STORE', 'Neighborhood_TWIN PEAKS', 'Property_Code_Des_W - CHURCHES,CONVENTS,RECTORIES', 'Property_Code_Des_RH - RESIDENTIAL HOTEL & SRO', 'Property_Code_Des_TIA - TI APARTMENT', 'Property_Code_Des_DA - DWELLINGS - APARTMENTS', 'Neighborhood_SEACLIFF', 'Property_Code_Des_ZBM - CONDOMINIUM BMR', 'Neighborhood_JAPANTOWN', 'Property_Code_Des_TIF - TI FLATS & DUPLEX', 'Neighborhood_MISSION BAY', 'Property_Code_Des_G - GARAGES (COMMERCIAL)', 'Property_Code_Des_IW - INDUSTIAL WAREHOUSE', 'Property_Code_Des_X - MISC', 'Property_Code_Des_LZ - LIVE/WORK CONDOMINIUM', 'Neighborhood_MCLAREN PARK', 'Property_Code_Des_CZ - COMMERCIAL STORE CONDO', 'Property_Code_Des_B - BANK', 'Neighborhood_LAKESHORE', 'Property_Code_Des_F2 - FLATS & DUPLEX', 'Property_Code_Des_Z - FLATS & DUPLEX', 'Property_Code_Des_CO - COOP UNITS UNSEGREGATED', 'Property_Code_Des_X - THEATRES', 'Property_Code_Des_N2 - CONVALESCENT/NURSING HOMES', 'Property_Code_Des_S - GAS STATION', 'Property_Code_Des_U - CLUBS,LODGES,FRATERNAL ORGANIZATIONS', 'Property_Code_Des_HC - HOTEL COMMERCIAL (H2W/COM)', 'Property_Code_Des_FA - 1 FLAT & 1 APT BLDG-1 PARCEL', 'Property_Code_Des_Z - TI APARTMENT', 'Property_Code_Des_TH - TOWN HOUSE', 'Neighborhood_LINCOLN PARK', 'Property_Code_Des_DD - 2 DWELLINGS ON ONE PARCEL', 'Property_Code_Des_E - SCHOOLS', 'Property_Code_Des_V - VACANT LOT', 'Property_Code_Des_M - MOTELS', 'Property_Code_Des_T - THEATRES', 'Property_Code_Des_COS - COOP UNITS UNSEGREGATED', 'Property_Code_Des_VR - VACANT LOT - RESTRICTIONS', 'Neighborhood_PRESIDIO', 'Property_Code_Des_Z - LIVE/WORK CONDOMINIUM', 'Property_Code_Des_DBM - DWELLING BMR', 'Property_Code_Des_DD - DWELLING', 'Property_Code_Des_CM - COMMERCIAL/MIXED USE', 'Property_Code_Des_RH1 - RETAIL & HOTEL', 'Property_Code_Des_OC - OFFICE WITH MAJOR RETAIL UNITS', 'Property_Code_Des_Z - FLAT & STORE', 'Property_Code_Des_CZ - COMMERCIAL STORES', 'Property_Code_Des_OZ - OFFICE - CONDOMINIUM', 'Property_Code_Des_X - SCHOOLS', 'Property_Code_Des_PD - PUD (PLANNED UNIT DEVELOPMENT)', 'Property_Code_Des_IZ - INDUSTRIAL CONDOMINIUM', 'Property_Code_Des_C1 - SHOPPING CENTER', 'Property_Code_Des_DF - 1 FLAT & DWELLING-1 PARCEL', 'Property_Code_Des_IDC - INDUSTRIAL DATA CENTER', 'Property_Code_Des_ZBM - OFFICE - CONDOMINIUM', 'Property_Code_Des_PZ - PARKING STALL CONDOMINIUM', 'Property_Code_Des_Z - TI FLATS & DUPLEX', 'Property_Code_Des_H1 - HOTEL', 'Property_Code_Des_OCL - OFFICE - LOW CLASS C', 'Property_Code_Des_Z - INDUSTRIAL CONDOMINIUM', 'Property_Code_Des_N2 - DWELLING', 'Property_Code_Des_OA - OFFICE AND APARTMENTS', 'Property_Code_Des_Z - OFFICE - CONDOMINIUM', 'Property_Code_Des_OMD - MEDICAL- DENTAL OFFICE BUILDING', 'Property_Code_Des_N1 - HOSPITALS', 'Property_Code_Des_MIX - MIXED USE', 'Property_Code_Des_TSU - TIME SHARE UNSEGREGATED', 'Property_Code_Des_Z - TOWN HOUSE', 'Property_Code_Des_N2 - FLATS & DUPLEX', 'Property_Code_Des_N2 - HOTELS - OTHER', 'Property_Code_Des_OCH - OFFICE - HIGH CLASS C', 'Property_Code_Des_Z - DWELLING', 'Property_Code_Des_W - INDUSTIAL WAREHOUSE', 'Property_Code_Des_OBH - OFFICE - HIGH CLASS B', 'Property_Code_Des_COS - COOP UNITS SEGREGATED', 'Property_Code_Des_W - VACANT LOT - RESTRICTIONS', 'Property_Code_Des_O - SCHOOLS', 'Property_Code_Des_OAH - OFFICE - HIGH CLASS A', 'Property_Code_Des_OCM - OFFICE - MIDDLE CLASS C', 'Property_Code_Des_Z - PARKING LOT', 'Property_Code_Des_W - DWELLING', 'Property_Code_Des_X - OFFICE', 'Property_Code_Des_DF - DWELLING', 'Property_Code_Des_CD - COMMERCIAL STORES', 'Property_Code_Des_W - INDUSTRIAL MIXED/OTHER USE', 'Property_Code_Des_HC - HOTELS - OTHER', 'Property_Code_Des_IX - INDUSTRIAL MIXED/OTHER USE', 'Property_Code_Des_DD - APARTMENT', 'Property_Code_Des_U - HOTELS - OTHER', 'Property_Code_Des_Z - INDUSTRIAL', 'Property_Code_Des_OMD - OFFICE', 'Property_Code_Des_Z - PARKING STALL CONDOMINIUM', 'Property_Code_Des_FA - FLATS & DUPLEX', 'Property_Code_Des_FA - APARTMENT', 'Property_Code_Des_W - COMMERCIAL STORES', 'Property_Code_Des_ZBM - GARAGE CONDOMINIUM', 'Property_Code_Des_X - VACANT LOT', 'Property_Code_Des_GZ - GARAGE CONDOMINIUM', "Property_Code_Des_XV - SINGLE STRUCT ON MULTI LOT(D & F'S ONLY)", 'Property_Code_Des_W - SCHOOLS', 'Property_Code_Des_ZBM - GARAGES (COMMERCIAL)', 'Property_Code_Des_N2 - HOTEL', 'Property_Code_Des_W - CONVALESCENT/NURSING HOMES', 'Property_Code_Des_U - COMMERCIAL STORES', 'Property_Code_Des_IW - SHOPPING CENTER', 'Property_Code_Des_H - HOTEL', 'Property_Code_Des_W - RESIDENTIAL HOTEL & SRO', 'Property_Code_Des_Z - GARAGES (COMMERCIAL)', 'Property_Code_Des_U - LIVE/WORK CONDOMINIUM', 'Property_Code_Des_Z - COOP UNITS UNSEGREGATED', 'Property_Code_Des_P - PUBLIC BUILDINGS (GOVT)', 'Property_Code_Des_GC - GOLF COURSE', 'Property_Code_Des_PL - PARKING LOT', 'Property_Code_Des_W - SHOPPING CENTER', 'Property_Code_Des_W - COMMERCIAL/MIXED USE', 'Property_Code_Des_FA - DWELLING', 'Property_Code_Des_ZBM - MISC', 'Property_Code_Des_ZBM - INDUSTRIAL', 'Property_Code_Des_ZBM - LIVE/WORK CONDOMINIUM', 'Property_Code_Des_U - FLAT & STORE', 'Property_Code_Des_ZBM - DWELLING BMR', 'Property_Code_Des_DD - DWELLINGS - APARTMENTS', 'Property_Code_Des_CP - CITY PROPERTY', 'Property_Code_Des_OBM - OFFICE - MIDDLE CLASS B', 'Property_Code_Des_Z - VACANT LOT - RESTRICTIONS', 'Property_Code_Des_EC - ENTERTAINMENT COMPLEX', 'Property_Code_Des_W - FLAT & STORE', 'Property_Code_Des_Z - TIME SHARE UNSEGREGATED', 'Property_Code_Des_F - SCHOOLS', 'Property_Code_Des_X - PARKING STALL CONDOMINIUM', 'Property_Code_Des_MB - MISSION BAY', 'Property_Code_Des_I - SCHOOLS', 'Property_Code_Des_Z - MISC', 'Property_Code_Des_Z - VACANT LOT', 'Property_Code_Des_W - OFFICE', 'Property_Code_Des_ZBM - DWELLING', 'Property_Code_Des_GC - SCHOOLS', 'Property_Code_Des_X - VACANT LOT - RESTRICTIONS']
cross validation [ 0.75165139  0.74605694  0.8912331   0.85871569  0.7986993   0.69187833
  0.86500743  0.81789317  0.77564398  0.79336304]
yamariva2000 commented 7 years ago

Created Branch of new model 9_22. https://github.com/sfbrigade/datasci-firerisk/tree/9_22

Incorporates updated 9/20 data table and additional features. Also generates visualization csv table.

/home/kel/anaconda3/bin/python /home/kel/data_sci_fire/datasci-firerisk/multiclass.py

training accuracy 0.97
testing accuracy 0.93
labels ['0 No incident', '1 Incident']
model run time 2017-09-22 17:42:37.408801
               precision    recall  f1-score   support

0 No incident       0.94      0.98      0.96     56334
   1 Incident       0.77      0.59      0.67      8118

  avg / total       0.92      0.93      0.92     64452

['Property_Area', 'Assessed_Improvement_Val', 'Land_Value', 'age', 'Tot_Rooms', 'Num_Rooms', 'Num_Units', 'Perc_Ownership', 'count all complaints', 'Num_Stories', 'count potential fire control', 'Num_Bathrooms', 'Neighborhood_TENDERLOIN', 'Num_Bedrooms', 'Building_Cat_OTHER', 'Building_Cat_COMMERCIAL USE', 'Neighborhood_MISSION', 'count fire emergency safety', 'Neighborhood_FINANCIAL DISTRICT/SOUTH BEACH', 'Building_Cat_DWELLING', 'Neighborhood_SOUTH OF MARKET', 'Building_Cat_FLATS AND DUPLEX', 'Neighborhood_CASTRO/UPPER MARKET', 'Building_Cat_OFFICE', 'Neighborhood_NOB HILL', 'Neighborhood_HAYES VALLEY', 'Neighborhood_MARINA', 'Neighborhood_CHINATOWN', 'Building_Cat_INDUSTRIAL USE', 'Neighborhood_SUNSET/PARKSIDE', 'Neighborhood_HAIGHT ASHBURY', 'Neighborhood_WESTERN ADDITION', 'Neighborhood_RUSSIAN HILL', 'Neighborhood_OUTER RICHMOND', 'Neighborhood_PACIFIC HEIGHTS', 'Neighborhood_POTRERO HILL', 'Neighborhood_BERNAL HEIGHTS', 'Neighborhood_NORTH BEACH', 'Neighborhood_PORTOLA', 'Neighborhood_INNER RICHMOND', 'Neighborhood_LONE MOUNTAIN/USF', 'Neighborhood_NOE VALLEY', 'Neighborhood_OCEANVIEW/MERCED/INGLESIDE', 'Neighborhood_PRESIDIO HEIGHTS', 'Neighborhood_EXCELSIOR', 'Neighborhood_INNER SUNSET', 'count potential fire cause', 'Neighborhood_WEST OF TWIN PEAKS', 'count all complaints not corrected', 'Neighborhood_VISITACION VALLEY', 'Building_Cat_CONDO, ETC.', 'Neighborhood_OUTER MISSION', 'Neighborhood_GLEN PARK', 'Neighborhood_MISSION BAY', 'Neighborhood_TWIN PEAKS', 'Neighborhood_JAPANTOWN', 'count potential fire control not corrected', 'count fire emergency safety not corrected', 'count potential fire cause not corrected', 'Neighborhood_SEACLIFF', 'Neighborhood_LAKESHORE', 'Neighborhood_MCLAREN PARK', 'Neighborhood_PRESIDIO', 'Neighborhood_LINCOLN PARK']

cross validation [ 0.7340194   0.81060878  0.72202453  0.72670951  0.81073193