search

Thomas-Moore-Creative / shared_sandbox

openly shared code for troubleshooting
MIT License
0 stars 1 forks source link

Review detection code #4

Open pearseb opened 4 months ago

pearseb commented 4 months ago

@ChrisC28 would you mind reviewing the detection code that I uploaded?

Is this similar to what you've done previously? I may try to see how my code deals with doing this at the global scale, rather than over a small region like I've chosen. We could then scale up in both space and to different variables (e.g., BGC)?

Once we are happy with this approach, we might be able to apply it to other datasets for different problems?

Thomas-Moore-Creative commented 4 months ago

@pearseb - over in my branch starting to refactor the code without for loops, with flox and optimising methods and chunking. So far just looking at the single problem of running the climatological mean on the full globe SST. Currently:

next steps I'm going to look at:

pearseb commented 4 months ago

This is so cool @Thomas-Moore-Creative. Nice job. I would love to understand this code a little better, particularly what you are doing with printing out the number of computations? I don't quite understand what the process is here. You seem to be able to searching for the most efficient way to do the calculation and are using some functions to check this that I have not seen before. When you're next in, can we sit down for 30 minutes and go through your edits?

With your work here it may be possible to make these calculations efficiently over many variables now and many years! Exciting stuff for any work in extremes that we want to do.

Thomas-Moore-Creative commented 4 months ago

This is so cool @Thomas-Moore-Creative. Nice job. I would love to understand this code a little better, particularly what you are doing with printing out the number of computations? I don't quite understand what the process is here. You seem to be able to searching for the most efficient way to do the calculation and are using some functions to check this that I have not seen before. When you're next in, can we sit down for 30 minutes and go through your edits?

Absolutely, I want to discuss this - was thinking a small informal meeting - once I think I understand things more fully. The idea is 100% to create workflows, practices, and codes that will apply to ALL our "big climate / big ocean" problems.

With your work here it may be possible to make these calculations efficiently over many variables now and many years! Exciting stuff for any work in extremes that we want to do.

"make these calculations efficiently over many variables now and many years" 100% the idea, absolutely. This is really all covered under what I've been learning (the hard way) with the BRAN2020 climatological descriptive statistics.

The abstract that I have in for the coming 2024 Ocean Modelling and Observations Workshop with @ChrisC28 is to discuss these things - Are our workflows “analysis ready”? Thoughts on data science practices for huge CMIP & COSIMA archives.

Hoping to present a number of compelling use cases and this calculation might be one of them. Would be great to have your eye and contributions on this presentation so if you're happy to be co-author let me know.

pearseb commented 4 months ago

Would love to be involved if you think it'll be useful! An informal chat would be great. Happy to do so any day of the week. Once I understand things a bit better, I'd like to apply these calculations for pH and aragonite saturation state and some other BGC variables too on whatever datasets we can get our hands on. I'll have a chat with Richard to see what daily output he has.