search

CSTARS / spatial-cimis

New repository for the DWR Spatial CIMIS program
MIT License
0 stars 1 forks source link

DWR GOES-S CA crop date nomenclature off #83

Closed gjscheer-ucd closed 5 years ago

gjscheer-ucd commented 5 years ago

Noticed on the DWR GOES-S receiver the filename dates for the CA cropped pgm files are not creating the true PST time. They seem off by -7 hours.

-rw-r--r--. 1 cimis cimis 9561699 Aug  2 12:04 20190802T0356PST-B2.pgm
-rw-r--r--. 1 cimis cimis 9561699 Aug  2 12:09 20190802T0401PST-B2.pgm
-rw-r--r--. 1 cimis cimis 9561699 Aug  2 12:14 20190802T0406PST-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 470716436 Aug  2 12:04 conus/satimage-T24D69B13-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 470716436 Aug  2 12:09 conus/satimage-T24D69C3F-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 470716436 Aug  2 12:14 conus/satimage-T24D69D6B-B2.pgm
qjhart commented 5 years ago

Well, that seems to be because there is an inconsistency in the naming scheme for files from automated sciences; Compare the filenames on our two systems. (They use different timezones to mess us up, but these are the same data :)

I need to understand Automated Sciences freewheeling naming scheme to understand the issue.

CSTARS

-rw-r--r-- 1 goesmgr users 941432852 Aug  2 22:51 satimage-T24D6CD5B-B2.pgm
-rw-r--r-- 1 goesmgr users 941432852 Aug  2 23:01 satimage-T24D6CFB3-B2.pgm
-rw-r--r-- 1 goesmgr users 941432852 Aug  2 23:11 satimage-T24D6D20B-B2.pgm
-rw-r--r-- 1 goesmgr users 941432852 Aug  2 23:20 satimage-T24D6D463-B2.pgm

DWR

-rw-rw-r--. 1 goesmgr goesmgr 941432852 Aug  2 15:41 satimage-T24D6CAF1-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 941432852 Aug  2 15:51 satimage-T24D6CD49-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 941432852 Aug  2 16:01 satimage-T24D6CFA1-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 941432852 Aug  2 16:11 satimage-T24D6D1F9-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 941432852 Aug  2 16:21 satimage-T24D6D451-B2.pgm
qjhart commented 5 years ago

@gjscheer-ucd this is back to you

gjscheer-ucd commented 5 years ago

DWR is actually using CONUS data now rather than FULLDISK which looks to be -7 hours off. Maybe GOES-S data is being interpreted as being PST by Automated Sciences?

CSTARS

 rw-r--r-- 1 goesmgr users 941432852 Aug  2 23:51 fulldisk/satimage-T24D6DB6B-B2.pgm
-rw-r--r-- 1 goesmgr users 941432852 Aug  3 00:01 fulldisk/satimage-T24D6DDC3-B2.pgm
-rw-r--r-- 1 goesmgr users 941432852 Aug  3 00:11 fulldisk/satimage-T24D6E01B-B2.pgm
-rw-r--r-- 1 goesmgr users 470716436 Feb 19 23:23 conus/satimage-T23FEA1FD-B2.pgm
-rw-r--r-- 1 goesmgr users 470716436 Feb 19 23:28 conus/satimage-T23FEA329-B2.pgm
-rw-r--r-- 1 goesmgr users 470716436 Feb 19 23:33 conus/satimage-T23FEA455-B2.pgm

DWR

-rw-rw-r--. 1 goesmgr goesmgr 941432852 Aug  2 16:50 fulldisk/satimage-T24D6DB59-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 941432852 Aug  2 17:01 fulldisk/satimage-T24D6DDB1-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 941432852 Aug  2 17:11 fulldisk/satimage-T24D6E009-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 470716436 Aug  2 17:04 conus/satimage-T24D6E163-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 470716436 Aug  2 17:09 conus/satimage-T24D6E28F-B2.pgm
-rw-rw-r--. 1 goesmgr goesmgr 470716436 Aug  2 17:14 conus/satimage-T24D6E3BB-B2.pgm

The FULLDISK data looks more similar, but you're right, shouldn't they be the same data?

I'll contact Shawn but for now I'd like to modify your convert.mk script to not set back 8 hours.

https://github.com/CSTARS/spatial-cimis/blob/master/grb-box/convert.mk#L21

I'm assuming changing:

$(shell date --date="2000-01-01T12:00 UTC + $$(( 16#$(patsubst satimage-T%-B2.pgm,%,$(notdir $1)) )) seconds - 8 hours" +%Y%m%dT%H%MPST)

to

$(shell date --date="2000-01-01T12:00 UTC + $$(( 16#$(patsubst satimage-T%-B2.pgm,%,$(notdir $1)) ))" +%Y%m%dT%H%MPST)

I don't fully understand the middle part of the date command BTW.

gjscheer-ucd commented 5 years ago

DWR CONUS captures seem to be set to PDT. I've modified the convert.mk code to subtract 1 hour to standardize on PST.

qjhart commented 5 years ago

@gjscheer-ucd the problem with your solution is that we don't know why they are different. If it's a function of the TIMEDOMAIN that the two computers are using, then it's quite possible that the time will be off when we fall back to PST. If that's the case, then we need instead to set the DWR computer to UTC, but why didn't Automated Science do that already?

gjscheer-ucd commented 5 years ago

I agree with this assessment. The fix is temporary until how the time is determined. I'll keep the issue open until then.

gjscheer-ucd commented 5 years ago

Automated Sciences confirmed that the local host locale determines the hour for filenames. Shawn will set the receiver timezone to UTC and I'll adjust the CIMIS script to make sure filenames are created in timezone PST.