Shrink file sizes and improve memory usage (easy steps)

[bpbond]

Changes to trackingC_setup.Rmd:

Step	Runs	model_output.csv	/run	trk_output.csv	/run
(Initial)	70	36.4	0.52	120.2	1.72
Remove unneeded columns	70	6.8	0.1	37.1	0.53
Skip years (=5)	70	1.4	0.02	7.4	0.11

Changes to trackingC.Rmd:

Step	Runs	mem_used()
(Initial)	70	478
After file-shrink steps above	70	326
Remove unneeded objects	70	313

[leeyap]

Thanks for making these changes! I'll try running a 5000 run csv through and see what happens now.

[bpbond]

@leeyap re running through, um, feel free to try! That said I marked this PR as draft because there's more to do—I was just exploring options—but also thought it might be useful for us to discuss next week. Anyway, up to you, thanks.

[bpbond]

I will try to fix this check failure later.

[bpbond]

Back to you @leeyap — I'm not sure what has changed from your #55 PR, but trackingC.Rmd is breaking at the ggplot call in line 357, because atm_summary has zero rows, because atmosphere_pool has zero rows, because trk_output_final has zero rows...I didn't trace it further back than that.

Pushing a7ac38d38e5fa8f894e4c896a084d2f0668e9cfb fixing the error in trackingC_setup.Rmd.

[bpbond]

100% of model runs are failing the CMIP temperature test in line 245.

I did 5000 runs without a problem.

[leeyap]

I'm not having either of those issues? Looks like only 55 runs are dropped from the temperature tests in total. And, my ggplot in 357 is running just fine. If all of your runs are failing the test, then trk_output_final would indeed be empty, but that shouldn't be the case 🤔

[bpbond]

Hmm. Whatever is happening for me is also happening on GitHub Actions, hence failing test.

[leeyap]

I'm... not sure. I don't really know how to debug if it's working locally?

[bpbond]

Just a guess: it looks like that in #55 you updated the CO2 file, but not CMIP6_annual_tas_global.csv. Is this correct? I'm wondering whether you have an updated version of that file locally that allows the code to run without an error.

[leeyap]

Yes, I updated the CO2 file to include the SSPs it was missing previously, but made no changes to the tas file. I don't see any differences in the file on my computer, or between main and this branch. Hmm.

[bpbond]

And your local repo is exactly at 332c2e8 with no uncommitted changes?

[leeyap]

Correct!

[bpbond]

If you haven't done this already, could you please:

• Delete all files in output_files/
• Re-run trackingC_setup.Rmd after doing Sys.setenv(CI = "true")
• Re-run trackingC.Rmd

Thank you.

[leeyap]

Well, this is why the test is failing. Hector temperature suddenly looks like this. Not sure if this could be due to the fact that there are so few runs (only 14 per SSP with CI = "true")? Otherwise, not sure why that's happening 🤔

Also - it seems that when I deleted the files in output_files in this branch, without committing anything, all output_files on all branches are gone (help?)

[bpbond]

That is indeed. Wow. What the heck? I don't think there's any way this is due to the only-14 issue, but you can try a slightly higher number.

Also - it seems that when I deleted the files in output_files in this branch, without committing anything, all output_files on all branches are gone (help?)

Uh, crap, yes. They're not tracked by Git so when you delete them, they're gone. I'm sorry is I inadvertently gave you bad advice! I can send you my big (n=5000) output files, would that help?

[leeyap]

Ahh, okay. No worries, but yes please pass along the 5000 file! I'll run it through and see if it's weird, too.

[bpbond]

☝️ that's just bizarre. The Hector world is at +2 °C in 2000?!?

[bpbond]

OK, sending your way, hang on.

[bpbond]

Set save_every to one to prevent errors

Bummer.

[bpbond]

@leeyap are there other things you'd like done here? Please lmk if anything else I should/can do.

[leeyap]

@bpbond Just pushed up my last changes. Okay to merge once Rmarkdown check clears.