Gogoduck fails with CARS aggregation

[anguss00]

Steps to reproduce: Select layer CARS 2009 - Australian weekly sea water temperature, select the subset 18 Dec 2009, 22:09 UTC - 25 Dec 2009, 23:04 UTC. Run gogoduck job. Expected outcome: receive an aggregated netcdf file with no errors Observed outcome: error with no aggregation report recieved

[danfruehauf]

Alright, I think I've gotten to the bottom of the problem. It looks like yes, the last date/time measurement in CARS is 2009-12-25:23:04, HOWEVER, if you look at ncdump of CARS, you find that:

 DAY_OF_YEAR = 0, 30.5, 61, 91.5, 122, 152.5, 183, 213.5, 244, 274.5, 305, 335.5 ;

Which means the last DAY_OF_YEAR that we're actually allowed to subset on is around the 1st of Dec. The above subset will fail because both the beginning - 2009-12-18:22:09 and the end 2009-12-25:23:04 are beyond the 335.5 DAY_OF_YEAR.

This is more of a data problem IMO.

[anguss00]

Nice spotting Mr @danfruehauf

[danfruehauf]

I don't think there's much we can do here @anguss00. I've opened an issue in the content repo.

[anguss00]

Agreed, it's a problem with the data

[ggalibert]

It's a problem with the request :

there's no data between 18 Dec 2009, 22:09 UTC and 25 Dec 2009, 23:04 UTC... So GogoDuck is obviously struggling.

[danfruehauf]

Then why is the portal suggesting there is data on those date?

[ggalibert]

On Godiva you can see that there is none data out of the DAY_OF_YEAR = "2009-01-01", "2009-01-31 12", "2009-03-03", "2009-04-02 12", "2009-05-03", "2009-06-02 12", "2009-07-03", "2009-08-02 12", "2009-09-02", "2009-10-02 12", "2009-11-02", "2009-12-02 12" ;

[ggalibert]

By the way Dan, you're talking about the monthly while Angus started on the weekly... (see first post above)

[danfruehauf]

Ohhhhhhh, I see what you're talking now. So there are actually 2 CARS layers? One for weekly and one for monthly?

@anguss00, did you get that? It means we'll have to have configuration for 2 layers for aggregation. One is cars_weekly and the other is cars_monthly.

[ggalibert]

And for this weekly guy, the subset 18 Dec 2009, 22:09 UTC - 25 Dec 2009, 23:04 UTC makes sense.

[anguss00]

Yep, I follow. We'll have to consider that for the metadata as well for CARS

[ggalibert]

DAY_OF_YEAR = "2009-01-01", "2009-01-08 00:55:23.076923", "2009-01-15 01:50:46.153846", "2009-01-22 02:46:9.230769", "2009-01-29 03:41:32.307692", "2009-02-05 04:36:55.384615", "2009-02-12 05:32:18.461538", "2009-02-19 06:27:41.538462", "2009-02-26 07:23:4.615385", "2009-03-05 08:18:27.692308", "2009-03-12 09:13:50.769231", "2009-03-19 10:09:13.846154", "2009-03-26 11:04:36.923077", "2009-04-02 12", "2009-04-09 12:55:23.076923", "2009-04-16 13:50:46.153846", "2009-04-23 14:46:9.230769", "2009-04-30 15:41:32.307692", "2009-05-07 16:36:55.384615", "2009-05-14 17:32:18.461538", "2009-05-21 18:27:41.538462", "2009-05-28 19:23:4.615385", "2009-06-04 20:18:27.692308", "2009-06-11 21:13:50.769231", "2009-06-18 22:09:13.846154", "2009-06-25 23:04:36.923077", "2009-07-03", "2009-07-10 00:55:23.076923", "2009-07-17 01:50:46.153846", "2009-07-24 02:46:9.230769", "2009-07-31 03:41:32.307692", "2009-08-07 04:36:55.384615", "2009-08-14 05:32:18.461538", "2009-08-21 06:27:41.538462", "2009-08-28 07:23:4.615385", "2009-09-04 08:18:27.692308", "2009-09-11 09:13:50.769231", "2009-09-18 10:09:13.846154", "2009-09-25 11:04:36.923077", "2009-10-02 12", "2009-10-09 12:55:23.076923", "2009-10-16 13:50:46.153846", "2009-10-23 14:46:9.230769", "2009-10-30 15:41:32.307692", "2009-11-06 16:36:55.384615", "2009-11-13 17:32:18.461538", "2009-11-20 18:27:41.538462", "2009-11-27 19:23:4.615385", "2009-12-04 20:18:27.692308", "2009-12-11 21:13:50.769231", "2009-12-18 22:09:13.846154", "2009-12-25 23:04:36.923077" ;

But as you can see there are some decimals in the seconds... Sorry about that. I hope GogoDuck can deal with it...

[danfruehauf]

I think the problem is we just have a cars aggreated layer, but we need to have cars_weekly and cars_monthly...

[ggalibert]

Haha! yep. alright, I get it.

[anguss00]

Makes sense, we'll have to do minor portal changes but that's ok

[ggalibert]

CARS2009_Australia_weekly.nc CARS2009_World_monthly.nc

[danfruehauf]

@anguss00 Would you like to open a portal bug then? I'll make the modifications to support cars_weekly and cars_monthly in GoGoDuck.

[ggalibert]

Could you call them cars_australia_weekly and cars_world_monthly?

[danfruehauf]

Yep. That can be done!

[anguss00]

Done, will do

[danfruehauf]

@anguss00 Would you like to inherit the bug from here? Since I know you're doing already stuff around the cars layers in portal?

[anguss00]

Sure thing, I can add that to the current fix (hack) for CARS in the portal. When the time comes to go stateless, we can add it to the metadata as appropriate.

[anguss00]

Actually, we can get away with not changing any portal code. As it stands, this can be fixed by configuring the gogoduckOverrideLayerName as appropriate in the portal (either weekly or monthly depending on the layer).

[danfruehauf]

Where do you do that? Somewhere in the config?

[anguss00]

Yep, portal config. Can I just set it up now?

[danfruehauf]

Yep, set it, I'll deploy the go-go-duck fix then.

[anguss00]

rc first?

[danfruehauf]

Yeah, do RC :)