Update PyAOS lessons to use CMIP6 data instead of CMIP5

[hot007]

Updated lessons to use CMIP6 data instead of CMIP5. Included CMIP6 data files in /data, and removed the CMIP5 data. Have not updated any screenshots, only output plots. Added Episode 10 to work with dask, but it needs a lot of reviewing. Please test all changes, I have tested locally but I may have made copy-paste mistakes. Thanks! -Claire

[hot007]

New notebook looks much better, thanks! I'm a bit worried that the parallel version came out with all NaNs, that shouldn't have happened! Does .max sometimes suffer from the same NaN problem as .mean and need to use something like np.nanmean? I'll try to revisit this week but have a few other things to do first. I assume the initial serial calculation (~1hr) would NOT be run by students in class?! I like the idea of adding a bit on dask aware functions, though not sure if it'll fit in time-wise. Worth at least mentioning though I think.

[DamienIrving]

Yeah, the students wouldn't be running the serial calculation. In fact, for all of these lessons (besides the version control ones) the idea is that the students just watch as the instructor live codes what's in the notes, then the students do whatever is in the exercises. In this case the instructor wouldn't actually execute the serial command, they'd just say it takes an hour and move on (something else for the instructor notes document).

I agree that at the very least we'd have an information box introducing the idea of creating your own dask aware functions, with a link to a good tutorial on the topic. We can probably then just create an "enhancement" issue in the repo flagging that we could replace that information box with an actual dask aware function that we build from scratch sometime in future.

It would be good to have this PR merged by 9am Monday at the latest, because that's when ticket "sales" for the workshop open (we are sending an email to the 130+ conference attendees who expressed an interest in attending the workshop this week telling them that 35 places are available, so please make sure you really need to come). Once people register they'll get the data download instructions, and this PR has the new data files in it. https://www.eventbrite.com.au/e/python-for-atmosphere-and-ocean-science-tickets-137505758425

So I guess for now we just need to:

• Figure out the Nan issue so we can plot the daily maximum SST
• Create the information box for Dask aware functions
• Write some exercises (with solutions)

Exercise ideas:

• Get them to run the following to see how many cores and how much memory they have available on their own computer (unless there is a better way to find that information?):

  from dask.distributed import Client
  client = Client()
  client

• Ask them to list other things they could do to avoid memory errors besides resorting to dask (e.g. subsetting the data, processing and outputting one netcdf file at a time). What are the pros and cons of these options? (One pro being you can use non-dask-aware functions.)
• Explore a problem where parallel processing doesn't really speed things up (e.g. if you do the daily precipitation maximum instead of daily SST maximum for ACCESS-CM2 the serial and parallel approach takes about the same amount of time)
• Get them to take the code we developed for plotting the daily maximum SST and turn it into a command line program (i.e. this incorporates everything from the previous lessons)

[hot007]

I've just added a dask daskboard screenshot to the 'fig' directory, but haven't changed the lesson to incorporate it, just letting you know it's there if you want to put it in your notebook when you're preparing it @DamienIrving .

[hot007]

Oh, it’s completely fine, my plot didn’t work because we didn’t remap to a Cartesian grid so it’s still tripolar, just doing imshow indicates the data is fine. So if you want to deal with remapping, we can add that in and make a nice plot, or we can just do imshow to demonstrate we did the thing.

So I guess I’ve at least partially solved the NaN problem and maybe having it on a tripolar grid leads to a good exercise for the students to map it back to Cartesian and plot as per 02-visualisation.

[DamienIrving]

I like the idea of having the students do the remapping as an exercise. We can have a "hint" in the exercise pointing to some online documentation about the relevant remapping function. (They can read about the remapping while they wait the 20 minutes for the parallel tos max calculation to happen!)

[hot007]

I'll admit I ran it on the VDI to check (but still using THREDDS) and it ran in under 4min!! I assume I messed up anyone else on the node at that time, but it was so much quicker than my laptop!