search

aodn / python-aodntools

Repository for templates and code relating to generating standard NetCDF files for the Australia Ocean Data Network
GNU Lesser General Public License v3.0
10 stars 3 forks source link

hourly product source_file variable lists local paths instead of S3 dest_paths #121

Closed mhidas closed 4 years ago

mhidas commented 4 years ago

The source_file variable in the aggregated and hourly products should only list the relative dest_paths (i.e. S3 ids) of each file, e.g.

 source_file =
  "IMOS/ANMN/QLD/GBRELR/CTD_timeseries/IMOS_ANMN-QLD_CSTZ_20080504T230001Z_GBRELR_FV01_GBRELR-0805-SBE37-SM-18_END-20081024T002501Z_C-20170409T233721Z.nc",
  "IMOS/ANMN/QLD/GBRELR/CTD_timeseries/IMOS_ANMN-QLD_STCPE_20081024T083000Z_GBRELR_FV01_GBRELR-0810-SBE37-SM-17_END-20090418T214300Z_C-20120205T160758Z.nc",
  "IMOS/ANMN/QLD/GBRELR/CTD_timeseries/IMOS_ANMN-QLD_STCPE_20090419T083000Z_GBRELR_FV01_GBRELR-0904-SBE37-SM-282_END-20091019T234500Z_C-20120202T055607Z.nc",

Instead, the hourly files currently produced by the pipeline (aodndata 1.2.1, aodntools 1.3.2) have the temporary local paths (meaningless as soon as the pipeline has completed):

 source_file = 
"/mnt/ebs/tmp/MooringsProductsHandlerr42a0orm/temp/IMOS/ANMN/QLD/GBRELR/Temperature/IMOS_ANMN-QLD_T_20070915T113000Z_GBRELR_FV01_GBRELR-0709-SBE39-74_END-20080504T130000Z_C-20120206T043504Z.nc", 
"/mnt/ebs/tmp/MooringsProductsHandlerr42a0orm/temp/IMOS/ANMN/QLD/GBRELR/Temperature/IMOS_ANMN-QLD_T_20070915T113000Z_GBRELR_FV01_GBRELR-0709-SBE39-83_END-20080504T130000Z_C-20120206T043503Z.nc",
"/mnt/ebs/tmp/MooringsProductsHandlerr42a0orm/temp/IMOS/ANMN/QLD/GBRELR/Temperature/IMOS_ANMN-QLD_T_20070915T113000Z_GBRELR_FV01_GBRELR-0709-SBE39-44_END-20080504T130000Z_C-20120206T043505Z.nc",