FALL 2024 & SPRING 2025: Project Organization

[andrewfagerheim]

This issue contains meeting notes, action items, and updates starting September 2024. For similar issues from previous semesters see:

• 1

• 8

• 10

• 13

[andrewfagerheim]

23 Sept 2024: Meeting w @dhruvbalwada

Notes: parallelization

• Use dask (that's built into xarray) to parallelize the interpolation process. If this automatic integration doesn't work, then we might need to use the dask library directly. Hopefully this is not the case, it would warrant further discussion.
• Choose the profile dimension for chunking; this is the parallelizable dimension because our analysis treats each profile separately.
• Ensure that multiple cores are being used by checking the argo dashboard. Start with simple test cases in the xarray documentation to make sure functionality is working as expected. Also check that the LEAP hub and gyre both give the same results to make sure results are not machine-dependent.

Notes: vertical coordinate

• A broader note on using density as the vertical coordinate; there are two reasons to do this.
  • The first is mechanistic---to remove the influence of internal waves stretching and squeezing isopycnals but not stirring the tracers.
  • The second is analytic---to more readily visualize if bands of high variance are "sliding" along isopycnals into the interior in regions of deepwater formation, which would connect these variance results with ventilation.
• The process for this coordinate transformation is as follows:
  • C(z,x,t), ρ(z,x,t) --> C(ρ,x,t)
  • C(ρ,x,t) --> C(zρ,x,t)
  • Then either C(zρ,x,t) is used to calculate variance
  • Or C(zρ,x,t) --> stretching of N2 is considered
  • Then this metric would be used to calculate variance

Next steps:

• [x] reread Ferrari 2005 and take notes here, see Dhruv's notes here
• [ ] read SWOT proposal and take notes here
• [x] test dask parallelization with example from xarray documentation
• [x] test dask parallelization with interpolation function

[andrewfagerheim]

14 Oct 2024: Meeting w @dhruvbalwada

Notes:

• Process is likely crashing because of a memory error (especially because the dataset I'm creating is pretty gigantic). The first thing to test is simply updating the packages. Dhruv mentioned that dask memory management issues have been addressed recently, so hopefully updating the packages fixes this too.
• apply_ufunc: function should be written so you could prove a np array (no coordinate information, etc that xr provides) and it would run completely fine. apply_ufunc then tries to adapt the behavior of this function on a np array to provide the same behavior with an xr dataset, including using dask if you tell it to.

Unresolved questions:

• Why does dask not run in parallel for xarray function that takes a long time (minutes) to run on one CPU?
• Why does the kernal crash when a large dataset is used? (Shouldn't dask be able to resolve memory load?)
• What are the barriers to running apply_ufunc?

Next steps:

• [x] update versions of xarray and dask, see if this resolves memory crashing issue
• [ ] compare how long it takes to run get_anom function on an Argo section with and without dask
• [ ] (assuming dask is quicker) use dask to perform a coordinate transformation

[andrewfagerheim]

22 Oct 2024: Meeting w @dhruvbalwada

Notes:

• Ocean Mixing Ch3: lots of connections with Ferrari & Polzin which is great. Start with a general overview of the chapter and all the sections, so people know where to go if they'd like more info about everything. Then can go through a more detailed description of different sections:
  • Mixing conversation: walk through of tracer equation, double and triple decompositions
  • Non-dissipative theories: quick orientation to theories (where they occur, their assumptions)
  • Mixing and circulation: hand-wavy description of three mechanisms mentioned
  • Mixing and circulation locations: make a chart summarizing isotropic mixing vs mesoscale stirring at different depths
• What is pseudo-depth? It seems like it may be discussed in Gouretski & Koltermann 2004? We're wondering if it could be 1) roughly analogous to mean isopycnal depth across basins, or 2) accounting for irregularities in the number and value of density measurements across many depths. If it's the former this would be really relevant to our Argo sections, however the later would be a lot less interesting.

Next steps:

• [ ] work on dask steps listed above
• [x] read Castro et al. 2024 and take good notes
• [x] prepare for discussion on how to apply Castro et al. and Ferrari & Polzin to Argo data
• [ ] look into "pseudo-depth"

[andrewfagerheim]

31 Oct 2024: Meeting w @dhruvbalwada

Notes:

• Castro 2024: uses the triple decomposition on profiles of temperature and shear from the North Atlantic to create estimates for diapycnal and isopycnal mixing. Connects the rates of mixing to large-scale ocean circulation, in particular to AMOC.
• This is a great study and there's a lot of detail that I should dive into further:
  • is $\overline{\theta}$ the same as $\theta^m$
  • how is $\overline{\theta}$ computed in this study
  • it feels like some of the plots are using a double decomposition and some are using a triple decomposition, understand better what is being used for each part of the analysis
• But even more broadly speaking, there is a lot of work here that I think can be replicated using Argo. For example:
  • Fig 3: these are plots of $\overline{\theta}$ and $\theta^m$, which seems like what I've done already
  • Fig 6: top panel is stratification which can definitely be calculated using argo; bottom panel is the contribution of the mean state to total variance which can also be calculated
  • Fig 8-10: dissipation (black) can't be calculated by spectra like they do because you'd need microstructure measurements, but we could use the Osborn and Cox (1972) formula: $\chi{\theta} = P{\theta^2} \approx 2K{\rho} (\frac{\partial \overline{\theta}}{\partial z})^2$ (eqn 3) diapycnal production (blue) I think can be calculated from argo using the formula: $P{\theta^2}^{\perp} = 2K_{\rho} (\frac{\partial \theta^m}{\partial z})^2$ (eqn 8)
• I feel like it's worth thinking more about the role of filtering in my analysis. Here they defined the mean state using a 4-degree polynomial fit, but spatial averaging would work too. I guess I'm wondering if spatial averaging alone would be adequate to define the mean vs anomaly for our analysis? I guess this also depends on if we're taking a double or triple decomposition approach.

Nest steps:

• [ ] write up notes for Castro 2024
• [x] plan time for each day next week using GCal
• [ ] continue tackling dask checklist
• [x] download and run latest version of argopy

Next 1-2 weeks:

• [ ] run vertical coordinate interpolation function in parallel with dask
• [ ] have one netdf/zarr file with all argo data, processed

Next 3-4 weeks:

• [ ] make variance section plots with 3 vertical coordinates (depth, density, average density depth)
• [ ] make plots like Fig 6 in Castro

Next 4-8 weeks:

• make plots like Fig 8-10 in Castro

[andrewfagerheim]

6 Nov 2024: Mental Organization

Notes:

• I think I've successfully updated argopy to version 1.0.0 which is great and should allow parallel loading with dask (will need to test this). However a few things to note:
  • I couldn't do this by running conda update --all alone, this would keep the version at 0.1.15 I believe. I had to do 'conda install conda-forge::argopy` to get version 1.0.0
  • Additionally, this new version of argopy seems to not allow for the newest version of xarray. So currently I'm running xarray version 2023.6.0, but I'm wondering if this doesn't have the more recent updates to dask integration. Maybe it would be worth it to figure out how to run 2023.6.0< xarray < 2024.3.0 with argopy 1.0.0
• As for the dask tests, I can't work on that yet because the gyre dashboard appears to be down. I emailed IT, so hopefully I will hear back soon on this.

[andrewfagerheim]

7 Nov 2024: Meeting w @dhruvbalwada

Notes:

• things to try on the argopy front:
  • run latest version of argopy with newest version possible of dask
  • it would be great to have one netcdf file with all data globally, HOWEVER if I end up spending too much more time on this it's not going to be worth it
  • backup would be to load regions bit by bit without using dask, then concatenate them together into one dataset
• on the dask front:
  • main priority is to be able to use apply_ufunc to run my own functions in parallel using dask
  • this will be useful for lots of things: vertical coordinate transformations, filtering, polynomial fit, etc
• on the readings front:
  • there are a lot of directions we could go with this work and I feel like I'm getting a little lost in the options. goal for today is to create a chart of each "offshoot," which papers they're most closely based on, and what the next steps and eventual outcomes would be
  • discuss with Dhruv and Andreas

Immediate next steps:

• [x] make project summary table
• [x] (if possible) update env file and load global box (still getting error)
• [ ] (if necessary) concatenate regions together

Slightly longer term next steps:

• [x] compare polynomial fit to filtering to spatial averaging --> pick Castro region, compare to Fig 3 (compare profiles and vertical gradient) [tracer gradient decomposition] --> temp & salinity
• [ ] learn apply_ufunc [any depth operation on many profiles] --> this is where spice would become more relevant

[andrewfagerheim]

11 Nov 2024: Meeting w @dhruvbalwada

Notes:

• Notes on the project table:
  • Add vertical coord. stuff to the multi-scale column
  • Maybe flag seasonality as a potential thing to explore somewhere on the table
  • Refine information in the tracer decomp & dissipation column using notes I jotted at the end
  • Big picture idea here is to deeply understand how to calculate all 4 arrows in triple decomposition diagram using argo. Create a list of what I would still need to perform all of these calculations
• Notes on smoothing notebook:
  • Initially, it seems like filtering is definitely winning, especially toward the boundaries (as visible with the first derivative in particular). It would be good to continue testing this however---look at lots of profiles with different shapes, levels of variance, etc. Maybe using the synthetic profiles notebook would be particularly helpful for this.
  • Try over the same depth region as the paper
  • Compare lots of different orders and filter scales
  • For spatial averaging: use Roemmich & Gilson's climatology instead of creating a box around each profile

Next steps:

• [ ] make edits to project table
• [ ] run through checklist above for smoothing tests
• [ ] concatenate global argo dataset
• [ ] write up how to calculate all four pathways/terms of the triple decomposition