norlandrhagen
commented
1 month ago Thanks I'd like to give 🙌
- Sammy for diving into thinking about data-proximate compute etc.
- Julius for super speed credential help
Cycle Review 🔎
- GODAS ran in production on dataflow. A review would be awesome, then I can add it to the catalog.
import xarray as xr
mutli = xr.open_zarr("gs://leap-persistent/data-library/feedstocks/GODAS/GODAS_multi_levels.zarr", chunks={}) surface = xr.open_zarr( "gs://leap-persistent/data-library/feedstocks/GODAS/GODAS_surface_level.zarr", chunks={})
- I tried out running dataflow with ARM based instances and they work! A bit cheaper, but more limited in adjustment knobs.
**My goals for the two upcoming weeks 💪**
- Get GODAS merged + cataloged
- Start to explore separate caching recipe. In the standard pgf caching step, it seems like the worker's don't scale. We could speed this up so much..
**My availability for this week 🗓**
- I'm out for Th, Fri and maybe into next week 🏥
This is a LEAP Data and Computation Team weekly standup 🎉🎉🎉! This is a way to share feedback from the past cycle, set goals for the next work cycle and coordinate work with others in the group.
Each member should aim to put down 2-3 goals going into the meeting. We then try to discuss how to best align activities around the group to facilitate collaboration as much as possible.
Expecting responses from all members of @leap-stc/data-and-compute.
Copy and paste the template below, and answer questions as you wish!
Response Template
``` **Thanks I'd like to give 🙌** - So-and-so helped me out a lot with XXX... - Thanks for Jo's work on the XXX... **User Interactions/Concerns** - So and so found a gnarly bug. I have raised an issue here ... - We got some awesome feedback on the new ... feature. **Cycle Review 🔎** - I finished implementing ... - **My goals for the two upcoming weeks 💪** - [ ] A goal/project I need to work on in the next two weeks: - [ ] An issue/project that needs help/coordination from another member **Questions for the group ❓** **My availability for this week 🗓** ```