search

quanted / hem_app

django app for the human exposure model
2 stars 0 forks source link

Add 'aftershave' dose data #148

Closed kdionisio closed 7 years ago

kdionisio commented 7 years ago

Load to development database, after #147 is complete

kdionisio commented 7 years ago 
LOAD DATA LOCAL INFILE '/home/kdionisio/HEMDBLoad/S2D_daily_all_aftershave.csv' INTO TABLE hem_app_dose FIELDS TERMINATED BY ',' ENCLOSED BY '"' ignore 1 lines (person_id, @chemical, day, dir_derm_exp, dir_derm_max, dir_derm_abs, dir_inhal_exp, dir_inhal_mass, dir_inhal_max, dir_inhal_abs, dir_ingest_exp, dir_ingest_abs, hem_app_dose.release, ind_derm_exp, ind_derm_max, ind_derm_abs, ind_inhal_exp, ind_inhal_max, ind_inhal_mass, ind_inhal_abs, ind_ingest_exp, ind_ingest_abs, out_sur, out_air, drain, waste) SET created_at = NOW(), updated_at = NOW(), chemical_id = (SELECT id FROM hem_app_chemical WHERE dtxsid = @chemical), runparams_id=2;
kdionisio commented 7 years ago

Data load successful, also loaded the 'aftershave' data into the 'all chemical/all PUC' run as temporary data (until the all chemical run is complete), so that the by chemical searches/plots would be populated.

kdionisio commented 7 years ago

currently includes 1 chemical

kdionisio commented 7 years ago

Dose results not valid; must re-run with updated code and re-loaded into DB

kdionisio commented 7 years ago

Command used for loading data output from 8/3 S2D version

LOAD DATA LOCAL INFILE '/home/kdionisio/HEM/output/S2D/aftershave_OPP/Daily/S2D_daily_all_aftershave_OPP.csv' INTO TABLE hem_app_dose FIELDS TERMINATED BY ',' ENCLOSED BY '"' ignore 1 lines (person_id, @chemical, day, dir_derm_exp, dir_derm_max, dir_derm_abs, dir_inhal_exp, dir_inhal_mass, dir_inhal_max, dir_inhal_abs, dir_ingest_exp, dir_ingest_abs, hem_app_dose.release, ind_derm_exp, ind_derm_max, ind_derm_abs, ind_inhal_exp, ind_inhal_max, ind_inhal_mass, ind_inhal_abs, ind_ingest_exp, ind_ingest_abs, out_sur, out_air, drain, waste) SET created_at = NOW(), updated_at = NOW(), chemical_id = (SELECT id FROM hem_app_chemical WHERE dtxsid = @chemical), runparams_id=2;
kdionisio commented 7 years ago

Loaded into prod_hem database

kdionisio commented 7 years ago

Test for speed- do matching of dtxsid to chemical_id outside of SQL and direct load file using command below.

LOAD DATA LOCAL INFILE '/home/kdionisio/HEM/output/S2D/aftershave_OPP/Daily/S2D_daily_all_aftershave_OPP.csv' INTO TABLE hem_app_dose FIELDS TERMINATED BY ',' ENCLOSED BY '"' ignore 1 lines (person_id, chemical_id, day, dir_derm_exp, dir_derm_max, dir_derm_abs, dir_inhal_exp, dir_inhal_mass, dir_inhal_max, dir_inhal_abs, dir_ingest_exp, dir_ingest_abs, hem_app_dose.release, ind_derm_exp, ind_derm_max, ind_derm_abs, ind_inhal_exp, ind_inhal_max, ind_inhal_mass, ind_inhal_abs, ind_ingest_exp, ind_ingest_abs, out_sur, out_air, drain, waste) SET created_at = NOW(), updated_at = NOW(), runparams_id=2;

loaded in 25 seconds, compared to matching within SQL (during load) where file loaded in 8 hours.

kdionisio commented 7 years ago

dose data deleted from DB due to speed issues; need to reload dose data when DB transition occurs