smallAreaHealthStatisticsUnit / rapidInquiryFacility

The Rapid Inquiry Facility (RIF) helps epidemiologists and public health researchers in environmental health activities.
GNU Lesser General Public License v3.0
14 stars 5 forks source link

Implementing linearity test with exposure #116

Closed bparkes closed 5 years ago

bparkes commented 5 years ago

Implementing linearity test with exposure

peterhambly commented 5 years ago

SQL Server is OK, Postgres fails Test 1005 with a SQL error

JDBC ERROR FETCHING!  execute JDBC update query failed in dbSend Update (ERROR: column "nan" does not exist at position: 158) 
Querying by JDBC: select * FROM rif40.t_rif40_homogeneity WHERE inv_id =533 AND study_id=561 and adjusted = 0 and genders = 2

(attached: TEST 1005 LUNG CANCER 95 96 Risk Analyisis 01 points.json.txt )

    Trying JDBC connection using driver class org.postgresql.Driver, lib dir C:\Program Files\Apache Software Foundation\Tomcat 8.5\webapps\rifServices\WEB-INF\lib, URL jdbc:postgresql://localhost:5432/sahsuland, user peter
... JDBC connection established 
Querying by JDBC: SELECT rif40_sql_pkg.rif40_startup() 
Connected to DB
connectToDb exitValue: 0
About to fetch extract table
JDBC EXTRACT TABLE NAME: rif_studies.s561_extract
Querying by JDBC: select * from rif_studies.s561_extract 
JDBC Saving extract frame to: scratchSpace/d501-600/s561/data/tmp_s561_extract.csv
JDBC rif_studies.s561_extract numberOfRows=39160==
About to calculate band data
Covariates: none
About to run homogeneity tests
callRiskAnal exitValue: 0
callRiskAnal() OK:  0 
About to save to table
In saveDataFrameToDatabaseTableSaving data frame to: scratchSpace/d501-600/s561/data/tmp_s561_map.csv
Creating temporary table: peter.tmp_s561_map
Replace INF will NA for temporary table: peter.tmp_s561_map
Replace NAN will NA for temporary table: peter.tmp_s561_map
Replace "" will NA for temporary table: peter.tmp_s561_map
About to write temporary table: peter.tmp_s561_map; first 10 rows
  area_id gid gid_rowindex username study_id inv_id band_id exposure genders
1      NA  NA           NA       NA      561    533       1        0       1
2      NA  NA           NA       NA      561    533       1        0       2
3      NA  NA           NA       NA      561    533       1        0       3
4      NA  NA           NA       NA      561    533       2        0       1
5      NA  NA           NA       NA      561    533       2        0       2
6      NA  NA           NA       NA      561    533       2        0       3
  direct_standardisation adjusted observed  expected   lower95   upper95
1                      0        0      388  418.6505 0.8390080 1.0237504
2                      0        0      198  225.8527 0.7626874 1.0077048
3                      0        0      586  646.8006 0.8355332 0.9824052
4                      0        0     2346 2431.7723 0.9264690 1.0045680
5                      0        0     1298 1290.9190 0.9522457 1.0617015
6                      0        0     3644 3715.5221 0.9494180 1.0131169
  relative_risk smoothed_relative_risk posterior_probability
1     0.9267874                     NA                    NA
2     0.8766777                     NA                    NA
3     0.9059979                     NA                    NA
4     0.9647285                     NA                    NA
5     1.0054853                     NA                    NA
6     0.9807505                     NA                    NA
  posterior_probability_upper95 posterior_probability_lower95
1                            NA                            NA
2                            NA                            NA
3                            NA                            NA
4                            NA                            NA
5                            NA                            NA
6                            NA                            NA
  residual_relative_risk residual_rr_lower95 residual_rr_upper95 smoothed_smr
1                     NA                  NA                  NA           NA
2                     NA                  NA                  NA           NA
3                     NA                  NA                  NA           NA
4                     NA                  NA                  NA           NA
5                     NA                  NA                  NA           NA
6                     NA                  NA                  NA           NA
  smoothed_smr_lower95 smoothed_smr_upper95
1                   NA                   NA
2                   NA                   NA
3                   NA                   NA
4                   NA                   NA
5                   NA                   NA
6                   NA                   NA
Creating study_id index on temporary table
SQL> CREATE INDEX s561_ind_study_id ON peter.tmp_s561_map(study_id)Creating area_id index on temporary table
SQL> CREATE INDEX s561_ind_area_id ON peter.tmp_s561_map(area_id)Creating genders index on temporary table
SQL> CREATE INDEX s561_ind_genders ON peter.tmp_s561_map(genders)Created indices on temporary table
saveDataFrameToDatabaseTable finishing
.Primitive("return")
Trying JDBC connection using driver class org.postgresql.Driver, lib dir C:\Program Files\Apache Software Foundation\Tomcat 8.5\webapps\rifServices\WEB-INF\lib, URL jdbc:postgresql://localhost:5432/sahsuland, user peter
... JDBC connection established 
Querying by JDBC: SELECT rif40_sql_pkg.rif40_startup() 
Connected to DB
connectToDb exitValue: 0
SQL> UPDATE  rif_studies.s561_map  a
 SET direct_standardisation=b.direct_standardisation,adjusted=b.adjusted,observed=b.observed,expected=b.expected,lower95=b.lower95,upper95=b.upper95,relative_risk=b.relative_risk,smoothed_relative_risk=CASE WHEN b.smoothed_relative_risk  = "NAN" THEN NULL ELSE b.smoothed_relative_risk END, posterior_probability=CASE WHEN b.posterior_probability  = "NAN" THEN NULL ELSE b.posterior_probability END, posterior_probability_upper95=CASE WHEN b.posterior_probability_upper95  = "NAN" THEN NULL ELSE b.posterior_probability_upper95 END, posterior_probability_lower95=CASE WHEN b.posterior_probability_lower95  = "NAN" THEN NULL ELSE b.posterior_probability_lower95 END, residual_relative_risk=CASE WHEN b.residual_relative_risk  = "NAN" THEN NULL ELSE b.residual_relative_risk END, residual_rr_lower95=CASE WHEN b.residual_rr_lower95  = "NAN" THEN NULL ELSE b.residual_rr_lower95 END, residual_rr_upper95=CASE WHEN b.residual_rr_upper95  = "NAN" THEN NULL ELSE b.residual_rr_upper95 END, smoothed_smr=CASE WHEN b.smoothed_smr  = "NAN" THEN NULL ELSE b.smoothed_smr END, smoothed_smr_lower95=CASE WHEN b.smoothed_smr_lower95  = "NAN" THEN NULL ELSE b.smoothed_smr_lower95 END, smoothed_smr_upper95=CASE WHEN b.smoothed_smr_upper95  = "NAN" THEN NULL ELSE b.smoothed_smr_upper95 END
 FROM peter.tmp_s561_map b WHERE a.study_id=b.study_id AND a.band_id=b.band_id AND a.inv_id=b.inv_id AND a.genders=b.genders 

Trying JDBC connection using driver class org.postgresql.Driver, lib dir C:\Program Files\Apache Software Foundation\Tomcat 8.5\webapps\rifServices\WEB-INF\lib, URL jdbc:postgresql://localhost:5432/sahsuland, user peter
... JDBC connection established 
Querying by JDBC: SELECT rif40_sql_pkg.rif40_startup() 
Connected to DB
connectToDb exitValue: 0
Saving data frame to: scratchSpace/d501-600/s561/data/tmp_s561_Homog.csv
Querying by JDBC: select * FROM rif40.t_rif40_homogeneity WHERE inv_id =533 AND study_id=561 and adjusted = 0 and genders = 1 
Stack tracer >>>

 .handleSimpleError(function (obj) 
{
    calls = sys.calls()
    calls = ca stop("execute JDBC update query failed in dbSendUpdate (", .jcall(x, "S", " .local(conn, statement, ...) dbSendUpdate(connection, updateStmt) JdbcHandler.R#530: dbSendUpdate(connection, updateStmt) Statistics_Common.R#178: withVisible(expr) Statistics_Common.R#178: withCallingHandlers(withVisible(expr), error = err JdbcHandler.R#530: withErrorTracing({
    dbSendUpdate(connection, updateSt doTryCatch(return(expr), name, parentenv, handler) tryCatchOne(expr, names, parentenv, handlers[[1]]) tryCatchList(expr, names[-nh], parentenv, handlers[-nh]) doTryCatch(return(expr), name, parentenv, handler) tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]), names tryCatchList(expr, classes, parentenv, handlers) JdbcHandler.R#529: tryCatch({
    withErrorTracing({
        dbSendUpdate(c insertHomogeneityResults(resultHomog) tryCatch({
    withErrorTracing({
        cat(paste0("About to fetch extrac eval(expr, pf) eval(expr, pf) withVisible(eval(expr, pf)) evalVis(expr) Statistics_JRI.R#280: capture.output({
    tryCatch({
        withErrorTrac runRRiskAnalFunctions() 
<<< End of stack tracer.
JDBC ERROR FETCHING!  execute JDBC update query failed in dbSendUpdate (ERROR: column "nan" does not exist
  Position: 158) 
Querying by JDBC: select * FROM rif40.t_rif40_homogeneity WHERE inv_id =533 AND study_id=561 and adjusted = 0 and genders = 2 
Stack tracer >>>

 .handleSimpleError(function (obj) 
{
    calls = sys.calls()
    calls = ca stop("execute JDBC update query failed in dbSendUpdate (", .jcall(x, "S", " .local(conn, statement, ...) dbSendUpdate(connection, updateStmt) JdbcHandler.R#530: dbSendUpdate(connection, updateStmt) Statistics_Common.R#178: withVisible(expr) Statistics_Common.R#178: withCallingHandlers(withVisible(expr), error = err JdbcHandler.R#530: withErrorTracing({
    dbSendUpdate(connection, updateSt doTryCatch(return(expr), name, parentenv, handler) tryCatchOne(expr, names, parentenv, handlers[[1]]) tryCatchList(expr, names[-nh], parentenv, handlers[-nh]) doTryCatch(return(expr), name, parentenv, handler) tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]), names tryCatchList(expr, classes, parentenv, handlers) JdbcHandler.R#529: tryCatch({
    withErrorTracing({
        dbSendUpdate(c insertHomogeneityResults(resultHomog) tryCatch({
    withErrorTracing({
        cat(paste0("About to fetch extrac eval(expr, pf) eval(expr, pf) withVisible(eval(expr, pf)) evalVis(expr) Statistics_JRI.R#280: capture.output({
    tryCatch({
        withErrorTrac runRRiskAnalFunctions() 
<<< End of stack tracer.
JDBC ERROR FETCHING!  execute JDBC update query failed in dbSendUpdate (ERROR: column "nan" does not exist
  Position: 159) 
Querying by JDBC: select * FROM rif40.t_rif40_homogeneity WHERE inv_id =533 AND study_id=561 and adjusted = 0 and genders = 3 
Stack tracer >>>

 .handleSimpleError(function (obj) 
{
    calls = sys.calls()
    calls = ca stop("execute JDBC update query failed in dbSendUpdate (", .jcall(x, "S", " .local(conn, statement, ...) dbSendUpdate(connection, updateStmt) JdbcHandler.R#530: dbSendUpdate(connection, updateStmt) Statistics_Common.R#178: withVisible(expr) Statistics_Common.R#178: withCallingHandlers(withVisible(expr), error = err JdbcHandler.R#530: withErrorTracing({
    dbSendUpdate(connection, updateSt doTryCatch(return(expr), name, parentenv, handler) tryCatchOne(expr, names, parentenv, handlers[[1]]) tryCatchList(expr, names[-nh], parentenv, handlers[-nh]) doTryCatch(return(expr), name, parentenv, handler) tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]), names tryCatchList(expr, classes, parentenv, handlers) JdbcHandler.R#529: tryCatch({
    withErrorTracing({
        dbSendUpdate(c insertHomogeneityResults(resultHomog) tryCatch({
    withErrorTracing({
        cat(paste0("About to fetch extrac eval(expr, pf) eval(expr, pf) withVisible(eval(expr, pf)) evalVis(expr) Statistics_JRI.R#280: capture.output({
    tryCatch({
        withErrorTrac runRRiskAnalFunctions() 
<<< End of stack tracer.
JDBC ERROR FETCHING!  execute JDBC update query failed in dbSendUpdate (ERROR: column "nan" does not exist
  Position: 159) 
Trying JDBC connection using driver class org.postgresql.Driver, lib dir C:\Program Files\Apache Software Foundation\Tomcat 8.5\webapps\rifServices\WEB-INF\lib, URL jdbc:postgresql://localhost:5432/sahsuland, user peter
... JDBC connection established 
Querying by JDBC: SELECT rif40_sql_pkg.rif40_startup() 
Connected to DB
connectToDb exitValue: 0
.Primitive("return")

----------------- End of Statistics failure details -------------------
devilgate commented 5 years ago

Hard to track down because of misleading logging statements, but I've fixed the "nan" problem. @bparkes, you might want to take a look and make sure it still does everything OK. I don't think I've disrupted anything.