search

PecanProject / pecan

The Predictive Ecosystem Analyzer (PEcAn) is an integrated ecological bioinformatics toolbox.
www.pecanproject.org
Other
202 stars 234 forks source link

Fluxnet met error on db insert step #1756

Open serbinsh opened 6 years ago

serbinsh commented 6 years ago 
6 2005-12-31 23:59:59 FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3
> fp <- file('/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_1-5052//pecan-47951cce983109fa00e9fb1c8f23c2', 'w')
> ign <- serialize(remoteout, fp)
> close(fp)
>
[1] 0
2017-11-03 16:59:59 INFO   [PEcAn.utils::convert.input] :
   RESULTS: Convert.Input
2017-11-03 16:59:59 INFO   [PEcAn.utils::convert.input] :
   c("",
   "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_1-5052//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2001.nc",
   "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_1-5052//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2002.nc",
   "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_1-5052//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2003.nc",
   "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_1-5052//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2004.nc",
   "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_1-5052//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2005.nc")
   c("", "modex.bnl.gov", "modex.bnl.gov", "modex.bnl.gov",
   "modex.bnl.gov", "modex.bnl.gov") c("", "application/x-netcdf",
   "application/x-netcdf", "application/x-netcdf", "application/x-netcdf",
   "application/x-netcdf") c("", "CF (gapfilled)", "CF (gapfilled)", "CF
   (gapfilled)", "CF (gapfilled)", "CF (gapfilled)") c("", "2001-01-01
   00:00:00", "2002-01-01 00:00:00", "2003-01-01 00:00:00", "2004-01-01
   00:00:00", "2005-01-01 00:00:00") c("", "2001-12-31 23:59:59",
   "2002-12-31 23:59:59", "2003-12-31 23:59:59", "2004-12-31 23:59:59",
   "2005-12-31 23:59:59")
   c("FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3",
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3",
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3",
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3",
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3",
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3")
2017-11-03 16:59:59 INFO   [PEcAn.utils::convert.input] :
   file host mimetype formatname startdate enddate dbfile.name
2017-11-03 16:59:59 ERROR  [dbfile.insert] :
   path to dbfiles: is not a valid full path
Error in postgresqlExecStatement(conn, statement, ...) :
  RS-DBI driver: (could not Retrieve the result : ERROR:  new row for relation "dbfiles" violates check constraint "file_path_sanity_check"
DETAIL:  Failing row contains (2000019071, FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3, , null, null, null, 2000000001, 2017-11-03 16:59:59, 2017-11-03 16:59:59, Input, 2000000165).
)
Warning in postgresqlQuickSQL(conn, statement, ...) :
  Could not create execute: INSERT INTO dbfiles (container_type, container_id, file_name, file_path, machine_id, created_at, updated_at) VALUES ('Input', 2000000165, 'FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3', '', 2000000001, '2017-11-03 16:59:59', '2017-11-03 16:59:59')
2017-11-03 16:59:59 SEVERE [db.query] :
   Error executing db query 'INSERT INTO dbfiles (container_type,
   container_id, file_name, file_path, machine_id, created_at, updated_at)
   VALUES ('Input', 2000000165,
   'FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3', '', 2000000001,
   '2017-11-03 16:59:59', '2017-11-03 16:59:59')' errorcode=0
   message='ERROR: new row for relation "dbfiles" violates check constraint
   "file_path_sanity_check" DETAIL: Failing row contains (2000019071,
   FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3, , null, null, null,
   2000000001, 2017-11-03 16:59:59, 2017-11-03 16:59:59, Input,
   2000000165). '
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Error in PEcAn.logger::logger.severe(paste("Error executing db query '",  :
  Error executing db query 'INSERT INTO dbfiles (container_type, container_id, file_name, file_path, machine_id, created_at, updated_at) VALUES ('Input', 2000000165, 'FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3', '', 2000000001, '2017-11-03 16:59:59', '2017-11-03 16:59:59')' errorcode=0 message='ERROR:  new row for relation "dbfiles" violates check constraint "file_path_sanity_check"
DETAIL:  Failing row contains (2000019071, FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3, , null, null, null, 2000000001, 2017-11-03 16:59:59, 2017-11-03 16:59:59, Input, 2000000165).
'
2017-11-03 16:59:59 WARN   [db.close] :
   Connection created outside of PEcAn.DB package
serbinsh commented 6 years ago 
> print(remoteout)
                                                                                                                                file
1                                                                                                                                   
2                                                                                                                                   
3                                                                                                                                   
4  /data/Model_Output/pecan.output/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2001.nc
5  /data/Model_Output/pecan.output/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2002.nc
6  /data/Model_Output/pecan.output/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2003.nc
7  /data/Model_Output/pecan.output/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2004.nc
8  /data/Model_Output/pecan.output/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2005.nc
9                                                                                                                                   
10                                                                                                                                  
            host             mimetype     formatname           startdate
1                                                                       
2                                                                       
3                                                                       
4  modex.bnl.gov application/x-netcdf CF (gapfilled) 2001-01-01 00:00:00
5  modex.bnl.gov application/x-netcdf CF (gapfilled) 2002-01-01 00:00:00
6  modex.bnl.gov application/x-netcdf CF (gapfilled) 2003-01-01 00:00:00
7  modex.bnl.gov application/x-netcdf CF (gapfilled) 2004-01-01 00:00:00
8  modex.bnl.gov application/x-netcdf CF (gapfilled) 2005-01-01 00:00:00
9                                                                       
10                                                                      
               enddate                                    dbfile.name
1                      FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3
2                      FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3
3                      FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3
4  2001-12-31 23:59:59 FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3
5  2002-12-31 23:59:59 FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3
6  2003-12-31 23:59:59 FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3
7  2004-12-31 23:59:59 FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3
8  2005-12-31 23:59:59 FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3
9                      FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3
10                     FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3
> fp <- file('/data/Model_Output/pecan.output/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//pecan-747e96181a13aedaef7e60e553f0d5', 'w')
> ign <- serialize(remoteout, fp)
> close(fp)
> 
[1] 0
2017-11-03 17:28:39 INFO   [PEcAn.utils::convert.input] : 
   RESULTS: Convert.Input 
2017-11-03 17:28:39 INFO   [PEcAn.utils::convert.input] : 
   c("", "", "", 
   "/data/Model_Output/pecan.output/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2001.nc", 
   "/data/Model_Output/pecan.output/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2002.nc", 
   "/data/Model_Output/pecan.output/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2003.nc", 
   "/data/Model_Output/pecan.output/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2004.nc", 
   "/data/Model_Output/pecan.output/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2005.nc", 
   "", "") c("", "", "", "modex.bnl.gov", "modex.bnl.gov", "modex.bnl.gov", 
   "modex.bnl.gov", "modex.bnl.gov", "", "") c("", "", "", 
   "application/x-netcdf", "application/x-netcdf", "application/x-netcdf", 
   "application/x-netcdf", "application/x-netcdf", "", "") c("", "", "", 
   "CF (gapfilled)", "CF (gapfilled)", "CF (gapfilled)", "CF (gapfilled)", 
   "CF (gapfilled)", "", "") c("", "", "", "2001-01-01 00:00:00", 
   "2002-01-01 00:00:00", "2003-01-01 00:00:00", "2004-01-01 00:00:00", 
   "2005-01-01 00:00:00", "", "") c("", "", "", "2001-12-31 23:59:59", 
   "2002-12-31 23:59:59", "2003-12-31 23:59:59", "2004-12-31 23:59:59", 
   "2005-12-31 23:59:59", "", "") 
   c("FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3", 
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3", 
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3", 
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3", 
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3", 
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3", 
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3", 
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3", 
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3", 
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3") 
2017-11-03 17:28:39 INFO   [PEcAn.utils::convert.input] : 
   file host mimetype formatname startdate enddate dbfile.name 
2017-11-03 17:28:39 ERROR  [dbfile.insert] : 
   path to dbfiles: is not a valid full path 
Error in postgresqlExecStatement(conn, statement, ...) : 
  RS-DBI driver: (could not Retrieve the result : ERROR:  new row for relation "dbfiles" violates check constraint "file_path_sanity_check"
DETAIL:  Failing row contains (2000019074, FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3, , null, null, null, 2000000001, 2017-11-03 17:28:39, 2017-11-03 17:28:39, Input, 2000000168).
)
Warning in postgresqlQuickSQL(conn, statement, ...) :
  Could not create execute: INSERT INTO dbfiles (container_type, container_id, file_name, file_path, machine_id, created_at, updated_at) VALUES ('Input', 2000000168, 'FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3', '', 2000000001, '2017-11-03 17:28:39', '2017-11-03 17:28:39')
2017-11-03 17:28:39 SEVERE [db.query] : 
   Error executing db query 'INSERT INTO dbfiles (container_type, 
   container_id, file_name, file_path, machine_id, created_at, updated_at) 
   VALUES ('Input', 2000000168, 
   'FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3', '', 2000000001, 
   '2017-11-03 17:28:39', '2017-11-03 17:28:39')' errorcode=0 
   message='ERROR: new row for relation "dbfiles" violates check constraint 
   "file_path_sanity_check" DETAIL: Failing row contains (2000019074, 
   FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3, , null, null, null, 
   2000000001, 2017-11-03 17:28:39, 2017-11-03 17:28:39, Input, 
   2000000168). ' 
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
> proc.time()
   user  system elapsed 
 51.291   7.804  76.209
serbinsh commented 6 years ago

It looks like file_path might not be getting set correctly as it looks to be blank in the log

serbinsh commented 6 years ago

Specifically here:

Warning in postgresqlQuickSQL(conn, statement, ...) :
  Could not create execute: INSERT INTO dbfiles (container_type, container_id, file_name, file_path, machine_id, created_at, updated_at) VALUES ('Input', 2000000168, 'FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3', '', 2000000001, '2017-11-03 17:28:39', '2017-11-03 17:28:39')
serbinsh commented 6 years ago

It seems

dirname(result$file[1])

results in nothing

but

PEcAn.logger::logger.info(dirname(result$file)) 

2017-11-03 17:56:52 INFO   [PEcAn.utils::convert.input] :
   /data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734
   /data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734
   /data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734
   /data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734
   /data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734

So it looks like we need to remove the [1] from the result$file in 

...
      newinput$input.id  <- existing.input$id
      newinput$dbfile.id <- dbfile.id 
    } else {
      newinput <- PEcAn.DB::dbfile.input.insert(in.path = dirname(result$file[1]),
                                      in.prefix = result$dbfile.name[1], 
                                      siteid = site.id, 
                                      startdate = start_date,
                                      enddate = end_date, 
                                      mimetype, 
                                      formatname, 
                                      parentid = parent.id,
                                      con = con, 
                                      hostname = machine$hostname,
                                      allow.conflicting.dates = allow.conflicting.dates)  
    }

in convert.input.R

Thoughts? @tonygardella others?

serbinsh commented 6 years ago

One thing I thought about is that we only need the first row of the results$file, so need to sort out why [1] was resulting in a "" (blank). I suppose it might need [1,] instead if its a dataframe

serbinsh commented 6 years ago

OK, here is the issue I am seeing. dirname(result$file[1]) is empty in the results dataframe, for whatever reason. Take a look

[1] 0
2017-11-06 08:29:34 INFO   [PEcAn.utils::convert.input] :
   RESULTS: Convert.Input
2017-11-06 08:29:34 INFO   [PEcAn.utils::convert.input] :
   c("",
   "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2001.nc",
   "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2002.nc",
   "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2003.nc",
   "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2004.nc",
   "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2005.nc")
   c("", "modex.bnl.gov", "modex.bnl.gov", "modex.bnl.gov",
   "modex.bnl.gov", "modex.bnl.gov") c("", "application/x-netcdf",
   "application/x-netcdf", "application/x-netcdf", "application/x-netcdf",
   "application/x-netcdf") c("", "CF (gapfilled)", "CF (gapfilled)", "CF
   (gapfilled)", "CF (gapfilled)", "CF (gapfilled)") c("", "2001-01-01
   00:00:00", "2002-01-01 00:00:00", "2003-01-01 00:00:00", "2004-01-01
   00:00:00", "2005-01-01 00:00:00") c("", "2001-12-31 23:59:59",
   "2002-12-31 23:59:59", "2003-12-31 23:59:59", "2004-12-31 23:59:59",
   "2005-12-31 23:59:59")
   c("FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3",
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3",
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3",
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3",
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3",
   "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3")
2017-11-06 08:29:34 INFO   [PEcAn.utils::convert.input] :
   file host mimetype formatname startdate enddate dbfile.name
'data.frame':   6 obs. of  7 variables:
 $ file       : chr  "" "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2001.nc" "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2002.nc" "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2003.nc" ...
 $ host       : chr  "" "modex.bnl.gov" "modex.bnl.gov" "modex.bnl.gov" ...
 $ mimetype   : chr  "" "application/x-netcdf" "application/x-netcdf" "application/x-netcdf" ...
 $ formatname : chr  "" "CF (gapfilled)" "CF (gapfilled)" "CF (gapfilled)" ...
 $ startdate  : chr  "" "2001-01-01 00:00:00" "2002-01-01 00:00:00" "2003-01-01 00:00:00" ...
 $ enddate    : chr  "" "2001-12-31 23:59:59" "2002-12-31 23:59:59" "2003-12-31 23:59:59" ...
 $ dbfile.name: chr  "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3" "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3" "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3" "FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3" ...
2017-11-06 08:29:34 INFO   [PEcAn.utils::convert.input] :
2017-11-06 08:29:34 INFO   [PEcAn.utils::convert.input] :
   dirname(result$file[1])
2017-11-06 08:29:34 INFO   [PEcAn.utils::convert.input] :
   /data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734
   /data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734
   /data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734
   /data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734
   /data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734
2017-11-06 08:29:34 ERROR  [dbfile.insert] :
   path to dbfiles: is not a valid full path
Error in postgresqlExecStatement(conn, statement, ...) :
  RS-DBI driver: (could not Retrieve the result : ERROR:  new row for relation "dbfiles" violates check constraint "file_path_sanity_check"
DETAIL:  Failing row contains (2000019098, FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3, , null, null, null, 2000000001, 2017-11-06 08:29:34, 2017-11-06 08:29:34, Input, 2000000169).
)
Warning in postgresqlQuickSQL(conn, statement, ...) :
  Could not create execute: INSERT INTO dbfiles (container_type, container_id, file_name, file_path, machine_id, created_at, updated_at) VALUES ('Input', 2000000169, 'FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3', '', 2000000001, '2017-11-06 08:29:34', '2017-11-06 08:29:34')
2017-11-06 08:29:34 SEVERE [db.query] :
   Error executing db query 'INSERT INTO dbfiles (container_type,
   container_id, file_name, file_path, machine_id, created_at, updated_at)
   VALUES ('Input', 2000000169,
   'FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3', '', 2000000001,
   '2017-11-06 08:29:34', '2017-11-06 08:29:34')' errorcode=0
   message='ERROR: new row for relation "dbfiles" violates check constraint
   "file_path_sanity_check" DETAIL: Failing row contains (2000019098,
   FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3, , null, null, null,
   2000000001, 2017-11-06 08:29:34, 2017-11-06 08:29:34, Input,
   2000000169). '

Note that the DB files (i.e. dbfile.name) is 6 objects long, BUT the rest have a blank "" in the first column of the dataframe. Thus they are only listing 5 of the 6 objects, at least for this site but I wonder if this is a general problem 

'data.frame':   6 obs. of  7 variables:
 $ file       : chr  "" "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2001.nc" "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2002.nc" "/data/home/sserbin/.pecan/dbfiles/Fluxnet2015_CF_gapfill_site_0-734//FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.2003.nc" ...

You can see the missing in the first part of "file" which mentions its a character vector but the first entry is "" thus dirname(result$file[1]) results in "" and thus the db.insert fails

ankurdesai commented 6 years ago

Curious. I will see if I can find the source of the "" being added to the front of list. If this was LISP, we could just write cdr(results$file) and go home. On the other hand, if this was LISP, all of PECAN would be executed in one giant multi-parenthetical function. (((((((((((((((((pecan

serbinsh commented 6 years ago

@ankurdesai thanks! To me it looks like the "" isn't added but instead represents a missing object...perhaps related to indexing in the for loops where we are accidentally out of sync?

ankurdesai commented 6 years ago

Oddly, don't see anything off in download.fluxnet met2CF or metgapfill that would cause either the first entry to not get written or be clobbered. The "results" array is always pre-populated before going into the loop. Something may be happening in convert.inputs or met.process.

ankurdesai commented 6 years ago

So I just ran the same site download, CF, gap fill, model and it ran. Except met2model.SIPNET only produced one day of met output instead of one year. I should note I only chose 1 year (2004).

ankurdesai commented 6 years ago

Ok I now ran 2001-2005 and did not get your error. Not sure what's up now? database issues?

serbinsh commented 6 years ago

@ankurdesai I don't think its a database issue since the error is pre-insert. For me, the issue is a mis-match in the results fields in that the "file" column only has 5 entries but the rest have six, and the first "file" is missing and that is what we are using to define the directory for the dbinsert. We could change that to ignore missing and just grab the first entry that is valid, but that doesn't explain why its not working on my end.

Could you send me you xml or paste your PEcAn commands here so I can try line by line as well? I want to be sure that running outside of PEcAn changes things. How recent is your local install, latest and greatest develop?

ankurdesai commented 6 years ago

Here's my history: http://tree.aos.wisc.edu:6480/pecan/08-finished.php?workflowid=5000000092

serbinsh commented 6 years ago

A new site worked,

https://modex.bnl.gov/pecan_dev/05-running.php?workflowid=2000000750

but it is still failing at this site :/

https://modex.bnl.gov/pecan_dev/08-finished.php?workflowid=2000000746&loglines=20

serbinsh commented 6 years ago

I will try to run through the met process by hand for this site and force flush everything to start from scratch

@ankurdesai I suppose best way to do this is run each command with the purge/clobber/delete arg set to TRUE?

ankurdesai commented 6 years ago

I think so. If you want to be super strict, you could also manually deleted all the folders _CF _CF_gapfill _model for that site and remove those entries from inputs and dbfiles.

serbinsh commented 6 years ago

Ahh....ok. Maybe I will try that.

serbinsh commented 6 years ago

Strange, while trying to purge files in my db I noticed a mis-match (I think). See image

screen shot 2017-11-13 at 9 11 58 am

  <site>
   <id>1000005052</id>
   <met.start>2001/01/01</met.start>
   <met.end>2005/12/31</met.end>
   <name>UCI-1930 burn site (CA-NS2)</name>
   <lat>55.9058</lat>
   <lon>-98.5247</lon>
  </site>

So I thought is was 1000005052 but it is also registered at site 734?

serbinsh commented 6 years ago

So in my servers DB files for site 734

[sserbin@modex Fluxnet2015_site_0-734]$ ll
total 49776
-rw-r--r--. 1 apache test 11765231 Nov  3 17:28 FLX_CA-NS2_FLUXNET2015_SUBSET_2001-2005_1-3.zip
-rw-r--r--. 1 apache test 39200998 Nov  3 17:28 FLX_CA-NS2_FLUXNET2015_SUBSET_HH_2001-2005_1-3.csv
serbinsh commented 6 years ago

Duh, I see there are two entries that are exact. of course. Not sure if that may have led to me problem?

serbinsh commented 6 years ago

All right so deleting the Fluxnet and _CF data and removing the dbfiles did the trick.

Unfortunately, it failed during the run due to an ED2 internal met check :/

 - Simulating:   01/24/2001 00:00:00 UTC
 - Simulating:   01/25/2001 00:00:00 UTC
 - Simulating:   01/26/2001 00:00:00 UTC
 - Simulating:   01/27/2001 00:00:00 UTC
 - Simulating:   01/28/2001 00:00:00 UTC
 - Simulating:   01/29/2001 00:00:00 UTC
 - Simulating:   01/30/2001 00:00:00 UTC
 - Simulating:   01/31/2001 00:00:00 UTC
 - Simulating:   02/01/2001 00:00:00 UTC
---------------------------------------------------
 Air specific humidity doesn't make sense... 

 - Polygon        :      1
 - Site           :      1
 - Date           : 02/01/2001
 - Time           : 13:30:00 UTC
 - Longitude      : -9.85247E+01
 - Latitude       :  5.59058E+01
 - Site spec. hum.:  9.87614E-07
 - Polygon sp. h. :  9.87614E-07
 - Minimum OK     :  1.00000E-06
 - Maximum OK     :  3.20000E-02
---------------------------------------------------

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

--------------------------------------------------------------
                     !!! FATAL ERROR !!!                      
--------------------------------------------------------------
    ---> File:        lapse.f90
    ---> Subroutine:  met_sanity_check
    ---> Reason:      Meteorological forcing has issues (check message).
--------------------------------------------------------------
 ED execution halts (see previous error message)...
--------------------------------------------------------------
Note: The following floating-point exceptions are signalling: IEEE_DENORMAL
STOP fatal_error
ERROR IN MODEL RUN
Logfile is located at '/data/Model_Output/pecan.output/PEcAn_2000000756/out/2000054675/logfile.txt'
************************************************* End Log 2017/11/13_09:21:42
ankurdesai commented 6 years ago

hmm, can you print out some values of sepcific_humidity. Either a conversion went wrong somewhere (which is possible) or there is some consistency problem among the variables reported (which can happen)

github-actions[bot] commented 4 years ago

This issue is stale because it has been open 365 days with no activity.