search

iridl / python-maprooms

Dash maprooms and tools
0 stars 4 forks source link

Remic/entozud #404

Closed remicousin closed 9 months ago

remicousin commented 9 months ago

I did a bit of cleaning by factoring out some functionality, removing all CONFIG entries from being inside functions, adding Documentation, and introducing system arguments rather than a loop on vars and time_res.

All of that to have convert either create a new zarr, or append to an existing one. At present this is assuming that there is no gap between the new set and the old set... Not sure what happens if there is. Maybe that's where the region case come into play? (I haven't looked into that).

I only choose to use to_zarr because it's what we were using in the first place. I am wondering if the 2 cases (create/append) could be even a lot closer if appending to nothing happens to create... or in the case reading the zarr store returns nothing, then make an empty zarr store and that's all the 2 cases differ from...

remicousin commented 9 months ago

I fixed all but the use of strftime to compare times... I ran out of options/ideas to compare comparable objects of same type...

I'll probably leave the other 2 recommendations as Issues, depending when the rest is validated and what's going on elsewhere.

aaron-kaplan commented 9 months ago

Thinking about how to build a workflow around this... when I tried the scipt it was extremely slow. Is it still slow when there's nothing to update?

remicousin commented 9 months ago

no. It didn't feel like it. I've been updating either nothing or one month at a time (or creating up to 5 months now)

aaron-kaplan commented 9 months ago

I'm asking because I'm wondering whether we should add the ENACTS zarr update step to update_datalib. Is it fast enough for that not to be annoying?

remicousin commented 9 months ago

It doesn't take time, but we already ran in a case where the filename pattern expected by enactstozarr was not respected (Guyana). So at least for that reason, we might want to wait until we parameterize that aspect or something?

aaron-kaplan commented 9 months ago

It will be optional, because not everyone has both ENACTS and python maprooms. So we can put it in the update script with a conditional such that it only runs on servers where it's been enabeld. Then we can enable it at one partner at a time, and if/when we need to enable it on one that requires that the script be generalized, we can make those changes then.

aaron-kaplan commented 9 months ago

Which country was the impetus for doing this now?

remicousin commented 9 months ago

Kenya MD