CLM-NEON Tutorial run time error

[cwsauburn]

Hello Will:

I am trying to run CLM-NEON Tool using the tutorial referred in the manuscript. But it shows the following error and runs is not complete, further log file is not present in the run directory.

Any suggestion how to resolve this issue!

---

Exception from case_run: ERROR: RUN FAIL: Command ' /home/user/CLM-NEON/TALL/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /home/user/CLM-NEON/TALL.transient/run/cesm.log.230125-011043 Submit job case.st_archive Cannot find a TALL.transient.cpl.r..nc file in directory /home/user/CLM-NEON/TALL.transient/run

Below is fill error message after running the run_neon command in jupyter lab run_neon --neon-sites $site --output-root ~/CLM-NEON --overwrite

/opt/ncar/cesm/tools/site_and_regional/run_neon.py:535: SyntaxWarning: "is not" with a literal. Did you mean "!="? if run_type is not "transient": ---- building a base case ------- Removing the existing case at: /home/user/CLM-NEON/TALL ---- creating a base case ------- ---- base case created ------ ---- base case setup ------ ---- base case build ------ WARNING: No .input_data_list files found in dir 'Buildconf' Time required to building the base case: 267.76820278167725 s. using this version: latest ---- removing the existing case ------- ---- cloning the base case in /home/user/CLM-NEON/TALL.transient varname datafiles is already in stream mod dictionary varname year_first is already in stream mod dictionary varname year_last is already in stream mod dictionary varname year_align is already in stream mod dictionary varname dtlimit is already in stream mod dictionary varname datafiles is already in stream mod dictionary varname year_first is already in stream mod dictionary varname year_last is already in stream mod dictionary varname year_align is already in stream mod dictionary varname dtlimit is already in stream mod dictionary varname year_first is already in stream mod dictionary varname year_last is already in stream mod dictionary varname year_align is already in stream mod dictionary varname dtlimit is already in stream mod dictionary Model ctsm missing file finidat = '/home/user/CLM-NEON/TALL.transient/run/inputdata/lnd/ctsm/initdata/TALL.2022-11-11.clm2.r.0318-01-01-00000.nc' Model datm missing file file1 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-01.nc' Model datm missing file file2 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-02.nc' Model datm missing file file3 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-03.nc' Model datm missing file file4 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-04.nc' Model datm missing file file5 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-05.nc' Model datm missing file file6 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-06.nc' Model datm missing file file7 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-07.nc' Model datm missing file file8 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-08.nc' Model datm missing file file9 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-09.nc' Model datm missing file file10 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-10.nc' Model datm missing file file11 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-11.nc' Model datm missing file file12 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2018-12.nc' Model datm missing file file13 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-01.nc' Model datm missing file file14 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-02.nc' Model datm missing file file15 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-03.nc' Model datm missing file file16 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-04.nc' Model datm missing file file17 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-05.nc' Model datm missing file file18 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-06.nc' Model datm missing file file19 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-07.nc' Model datm missing file file20 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-08.nc' Model datm missing file file21 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-09.nc' Model datm missing file file22 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-10.nc' Model datm missing file file23 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-11.nc' Model datm missing file file24 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2019-12.nc' Model datm missing file file25 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-01.nc' Model datm missing file file26 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-02.nc' Model datm missing file file27 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-03.nc' Model datm missing file file28 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-04.nc' Model datm missing file file29 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-05.nc' Model datm missing file file30 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-06.nc' Model datm missing file file31 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-07.nc' Model datm missing file file32 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-08.nc' Model datm missing file file33 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-09.nc' Model datm missing file file34 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-10.nc' Model datm missing file file35 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-11.nc' Model datm missing file file36 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2020-12.nc' Model datm missing file file37 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-01.nc' Model datm missing file file38 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-02.nc' Model datm missing file file39 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-03.nc' Model datm missing file file40 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-04.nc' Model datm missing file file41 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-05.nc' Model datm missing file file42 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-06.nc' Model datm missing file file43 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-07.nc' Model datm missing file file44 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-08.nc' Model datm missing file file45 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-09.nc' Model datm missing file file46 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-10.nc' Model datm missing file file47 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-11.nc' Model datm missing file file48 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2021-12.nc' Model datm missing file file49 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2022-01.nc' Model datm missing file file50 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2022-02.nc' Model datm missing file file51 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2022-03.nc' Model datm missing file file52 = '/home/user/CLM-NEON/TALL.transient/run/inputdata/atm/cdeps/v2/TALL/TALL_atm_2022-04.nc' varname datafiles is already in stream mod dictionary varname year_first is already in stream mod dictionary varname year_last is already in stream mod dictionary varname year_align is already in stream mod dictionary varname dtlimit is already in stream mod dictionary varname datafiles is already in stream mod dictionary varname year_first is already in stream mod dictionary varname year_last is already in stream mod dictionary varname year_align is already in stream mod dictionary varname dtlimit is already in stream mod dictionary varname year_first is already in stream mod dictionary varname year_last is already in stream mod dictionary varname year_align is already in stream mod dictionary varname dtlimit is already in stream mod dictionary submit_jobs case.run Submit job case.run varname datafiles is already in stream mod dictionary varname year_first is already in stream mod dictionary varname year_last is already in stream mod dictionary varname year_align is already in stream mod dictionary varname dtlimit is already in stream mod dictionary varname datafiles is already in stream mod dictionary varname year_first is already in stream mod dictionary varname year_last is already in stream mod dictionary varname year_align is already in stream mod dictionary varname dtlimit is already in stream mod dictionary varname year_first is already in stream mod dictionary varname year_last is already in stream mod dictionary varname year_align is already in stream mod dictionary varname dtlimit is already in stream mod dictionary Exception from case_run: ERROR: RUN FAIL: Command ' /home/user/CLM-NEON/TALL/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /home/user/CLM-NEON/TALL.transient/run/cesm.log.230125-011043 Submit job case.st_archive Cannot find a TALL.transient.cpl.r..nc file in directory /home/user/CLM-NEON/TALL.transient/run

Thank You Sanjiv Kumar

[cwsauburn]

Additional update related to above issue

(1) we are trying to use latest image 'escomp/cesm-lab-neon' to run our model, but it continues to fail in laptop computer (mac and PCs)

ERROR: RUN FAIL: Command ' /home/user/CLM-NEON/TALL/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed

See log file for details: /home/user/CLM-NEON/TALL.transient/run/cesm.log.230125-011043

(2) Please ignore my coupler related issue; this is likely to related to short term archiving issue; since run is not complete so they can not find the coupler restart file.

(3) We did find a work around for the simulation part; the older container image 'escomp/ctsm-lab-2.3-preview’' runs fine. and we completed the simulation using the ctsm-lab-2.3-preview; however the data visualization part fails here in the neon evaluation data download part; i think some the neon data link is broken, in the older version

data visualization script works with the newer image 'escomp/cesm-lab-neon'

(4) In summary, i am not able to complete the simulation and visualization script together using the new container image. I do not know if the recent HPC update caused any issue with cesm-lab-neon image or not?

[wwieder]

My simulations on Cheyenne are working as intended, but I have the same issues described above when running in the updated container. I'll poke around a bit, but @TeaganKing, if you need help maybe @briandobbins or @danicalombardozzi can help?

Regardless, we should likely rebuild the container given issues with lat-lon that have now been resolved on the main CTSM development branch.

[TeaganKing]

Thanks @wwieder . That makes sense to update the container with the new lat-lon values.

Just as a heads up, I'll probably be able to look at this in more detail tomorrow.

[briandobbins]

I'm happy to rebuild the container, or if you'd like to learn, show you how to do that, Teagan?

Additionally, Jim has a ton of experience on Github actions now - maybe we can automate the build of the container on PRs or releases, which would be nice. Does that sound good to you all?

• Brian

On Mon, Jan 30, 2023 at 9:55 AM Teagan King @.***> wrote:

Thanks @wwieder https://github.com/wwieder . That makes sense to update the container with the new lat-lon values.

Just as a heads up, I'll probably be able to look at this in more detail tomorrow.

— Reply to this email directly, view it on GitHub https://github.com/NCAR/NEON-visualization/issues/21#issuecomment-1408978943, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACL2HPOMKX7GWHOTRF3GRULWU7XBHANCNFSM6AAAAAAUFYO25Q . You are receiving this because you were mentioned.Message ID: @.***>

[TeaganKing]

I'd like to learn how you rebuild the container, Brian! Maybe we could tag up to do this tomorrow at 10am or 3:30pm? Automating this also seems like a great idea to make sure that it gets updated with new PRs.

[briandobbins]

Sending you an invite now for 10am tomorrow, Teagan. Per the invite, I'll walk you through the container and how to build it, but we likely do want to get Jim's input at some point to automate the build on releases (release versions) and PRs being added, perhaps (development version).

• Brian

On Mon, Jan 30, 2023 at 10:17 AM Teagan King @.***> wrote:

I'd like to learn how you rebuild the container, Brian! Maybe we could tag up to do this tomorrow at 10am or 3:30pm?

— Reply to this email directly, view it on GitHub https://github.com/NCAR/NEON-visualization/issues/21#issuecomment-1409011816, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACL2HPJPXW75JOGYFEPM6IDWU7ZS7ANCNFSM6AAAAAAUFYO25Q . You are receiving this because you were mentioned.Message ID: @.***>

[wwieder]

Thanks for doing this! I like the automation idea, but worry a bit about being able to test to containerized system regularly enough to make sure nothing breaks in the process. This may be a naive concern, although I'm concerned because it seems like the container is broken now...

[briandobbins]

Hi Will,

I think there are some fair elements to that concern -- but they're mitigated a bit by two things:

1) For 'release' tags, they go through tons of tests -- any code issues will be caught well before we get to the container 2) For something updated whenever a PR is merged, it's more of a 'use at your own risk' type thing, and tagged as such (eg, 'CTSM-Lab-devel', or something)

I think the bigger challenge we need to be aware of is that this is only tracking CTSM versions, whereas we also need to track the container environment versions (eg, OS, compilers, libraries, Jupyter, etc). Thankfully, that moves at a much slower pace, so I think we can still make progress on automating the builds without concern about what could really muck things up in the container (the environment!).

That said, to further mitigate this, let's get the infrastructure in place in the coming weeks, but make sure we build to newly-tagged images

• we'll keep the current official release version updated by hand, where we can do testing, while also creating new ones automatically, so we can test them periodically and, if no errors pop up, switch to that down the road.

  Cheers,

  • Brian

On Mon, Jan 30, 2023 at 12:43 PM will wieder @.***> wrote:

Thanks for doing this! I like the automation idea, but worry a bit about being able to test to containerized system regularly enough to make sure nothing breaks in the process. This may be a naive concern, although I'm concerned because it seems like the container is broken now...

— Reply to this email directly, view it on GitHub https://github.com/NCAR/NEON-visualization/issues/21#issuecomment-1409232412, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACL2HPJECF2QSEVHUXCZCGDWVAKWNANCNFSM6AAAAAAUFYO25Q . You are receiving this because you were mentioned.Message ID: @.***>

[TeaganKing]

Hi all,

Quick update: @briandobbins and I were able to create a new local container with the latest tag (ctsm5.1.dev116).

Unfortunately, this new container maintained the same errors that Sanjiv was previously noticing. An additional test involved running another NEON site, ABBY; running this case also resulted in the same error (a missing cpl*.r.*.nc file and it also lacks a log file with a more detailed description).

It seems that it would eventually be useful to provide a container with the updated lat/lon values, but we should probably first figure out what exactly is causing the error Sanjiv found (which I'm guessing is a bug in the code between ctsm-lab-2.3-preview and the most recent container release).

[wwieder]

Thanks for this update, @TeaganKing. Are you able to run as expected with tag dev116 on Cheyenne? If so, does this suggest that something isn't put together correctly on the container, or is there a larger issue with the code for some reason?

[TeaganKing]

Good idea, I'll try that out and let you know what the results are.

[TeaganKing]

Hi @wwieder and @briandobbins , I was indeed able to run the TALL transient case as expected with tag ctsm5.1.dev116 on Cheyenne. @briandobbins we discussed changing the line of code in the Dockerfile that was set to a particular tag, but was there anything else that would have been changed between these versions on the container side of things?

[wwieder]

Reaching for straws here, but does the container use all the same externals (e.g. CDEPS) as we're getting with a current tag on cheyenne?

[TeaganKing]

Thanks for the idea to check that, @wwieder ! Perhaps @briandobbins has more details, but I do see the following line in CTSM-NEON-Lab/ctsm/Dockerfile: ./manage_externals/checkout_externals, which I assume means that we do use the same externals that are being used on Cheyenne (at least, this was the same command I was using to checkout externals in my TALL case that worked successfully on Cheyenne)

[briandobbins]

We use the same version of the code in the container and on Cheyenne, so it's likely not that -- my guess is either it's something in the compiler or in the environment. On Cheyenne, we default to Intel. In the container, we default to GNU. I'll try to set up a test tonight and send an update tomorrow or so.

On Tue, Jan 31, 2023 at 4:26 PM Teagan King @.***> wrote:

Thanks for the idea to check that, @wwieder https://github.com/wwieder ! Perhaps @briandobbins https://github.com/briandobbins has more details, but I do see the following line in CTSM-NEON-Lab/ctsm/Dockerfile: ./manage_externals/checkout_externals, which I assume means that we do use the same externals that are being used on Cheyenne (at least, this was the same command I was using to checkout externals in my TALL case that worked successfully on Cheyenne)

— Reply to this email directly, view it on GitHub https://github.com/NCAR/NEON-visualization/issues/21#issuecomment-1411212818, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACL2HPOMU6PNSK3PRJLF2O3WVGNT3ANCNFSM6AAAAAAUFYO25Q . You are receiving this because you were mentioned.Message ID: @.***>

[TeaganKing]

Hi @briandobbins , I wanted to check in with you in regards to the compiler/environment testing; is there anything I can help test?

Also, after meeting with Sanjiv and Thomas, I wanted to add a few additional details to this issue. 1) the cesm-lab-neon container fails with run fail error and no detailed log file, as described above (the data visualization scripts works with this container). 2) the ctsm-lab-2.3-preview container runs smoothly for run_neon, but the data visualization tutorial fails on download_eval_files() with an HTTPSConnectionPool error. 3) Sanjiv is looking for a solution for his class on Tuesday; if we don't have a working container at that point, I will share some in depth instructions for running neon on Cheyenne. 4) Another potential workaround if we are still having difficulties on Monday may be to download the evaluation files while using the cesm-lab-neon container, saving these files, and then run with the ctsm-lab-2.3-preview container.

[briandobbins]

My apologies - some other things came up and I hadn't realized this was impacting a class. I'm looking at it now (downloading data, slowly), and will update this issue tonight with more info. @cwsauburn, can you share whether you're doing this and showing students vs having them do it on their own system? That info might be helpful once we identify the problem and decide how best to provide the solution - eg, if it's just you downloading a full new container, or a bunch of students where perhaps just a smaller 'update' to the existing container is better / faster.

Thanks!

[briandobbins]

Brief update -- I've duplicated this, and I'm guessing it's an odd interplay between ESMF and OpenMPI. On Cheyenne, we typically run with an older ESMF tag, and use MPT (MPICH-based) vs OpenMPI. I've got a new test building now and will give another update tomorrow when I hope to have tested a successful workaround for the issue.

[TeaganKing]

Thank you very much for looking at this, @briandobbins !

[briandobbins]

Good news -- I have the case running by switching back to MPICH (vs OpenMPI). We'll do two next steps - one, I'll build a new version with MPICH so we can do a full test (including visualizations), and two, I'll work with the ESMF folks on understanding the issue with OpenMPI and hopefully fixing that.

I've got meetings until 11am, but hope to have a new version built for testing then.

[TeaganKing]

That’s great news! Thank you Brian! Let me know if I can help test once that version is built!

On Feb 3, 2023, at 8:58 AM, briandobbins @.***> wrote:

 Good news -- I have the case running by switching back to MPICH (vs OpenMPI). We'll do two next steps - one, I'll build a new version with MPICH so we can do a full test (including visualizations), and two, I'll work with the ESMF folks on understanding the issue with OpenMPI and hopefully fixing that.

I've got meetings until 11am, but hope to have a new version built for testing then.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[briandobbins]

One more update - I also spoke with Jim and he'd seen the original issue before, and came up with two suggestions. I'm going to try to put those in the current OpenMPI-based container, since that keeps the primary infrastructure the same. This may take a little longer, but I should have something by sometime this afternoon.

[wwieder]

@briandobbins low priority, but if you're rebuilding the container to address this issue can you also bring in the source code from CTSM dev117? This will correct the lat-lon bug that @TeaganKing found and that's been corrected on main.

[briandobbins]

I've uploaded a new version (escomp/cesm-lab-neon-fixed) as a test, and once it looks fine, I'll update the main image. The 'fix' in this one was to disable the 'xmlchange' lines in usermods that set 'mpi-serial', but for the next major release this should be redone in the container so mpi-serial is doable - this is likely an easy change to ccs_config, but this was easier and more in line with how the container has been operating for now.

@wwieder This includes the new 117 tag. I have not tested the visualization, but will try to do so tomorrow.

[wwieder]

@briandobbins any updates on this? Thomas is actually in the building today and Sanjiv is hoping to use this in class tomorrow :)

[briandobbins]

Hi Will,

It should work; I've run cases, but not had the time to run the visualization yet. I'm in meetings until 330pm today, but happy to chat after?

If someone wants to give it a try, they should be able to just do: docker pull escomp/cesm-lab-neon-fixed

And then do a 'docker run' on the cesm-lab-neon-fixed container, much like they did the other one.

I'll also try to do the visualizations tonight.

• Brian

On Wed, Feb 8, 2023 at 1:26 PM will wieder @.***> wrote:

@briandobbins https://github.com/briandobbins any updates on this? Thomas is actually in the building today and Sanjiv is hoping to use this in class tomorrow :)

— Reply to this email directly, view it on GitHub https://github.com/NCAR/NEON-visualization/issues/21#issuecomment-1423196392, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACL2HPLNOMG65QDMCHNUJJTWWP6RHANCNFSM6AAAAAAUFYO25Q . You are receiving this because you were mentioned.Message ID: @.***>

[tkavoo]

Hello Brian I have tried pulling the image but seems I do not have access, is there a way I could get access to test the updated image? I am getting this error

docker pull escomp/cesm-lab-neon-fixed Using default tag: latest Error response from daemon: pull access denied for escomp/cesm-lab-neon-fixed, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

[TeaganKing]

Hi @briandobbins and @tkavoo , I'm also running into the same access issue that Thomas mentioned after logging in to docker with my UCAR email.

[briandobbins]

@TeaganKing @tkavoo

How strange - it worked for me, but perhaps that's because it was on the system I built it? I'll try from another system soon. I just got back from my meetings at the Foothills, but if either of you are available any time tonight between 6pm - 10pm, I'm happy to set up a Zoom call and ensure we've got this working for you too. I expect it'll be quick (and don't feel obligated; I hate asking people to do things off hours!).

I'll send you both emails now, in case you have some availability.

[TeaganKing]

Hi @briandobbins I can be available around 9pm if need be, or I'll be online around 7am tomorrow.

[tkavoo]

Hello @briandobbins and @TeaganKing I can be available at 9.00pm as well

[briandobbins]

Excellent, thanks - I’ll send an invite for 9pm in a little bit.  On Feb 8, 2023, at 4:22 PM, Thomas Kavoo @.***> wrote: Hello @briandobbins and @TeaganKing I can be available at 9.00pm as well

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[briandobbins]

Well, looks like this was an 'oops' on my part -- the actual image name is 'cesm-lab-neon-fix', not '-fixed'. So try:

docker pull escomp/cesm-lab-neon-fix

And that should work, and allow you to run things. I'll also try to check the visualization shortly, and if both aspects work, the 9pm Zoom call likely isn't necessary, but feel free to hop on anyway in case there are other questions, or if either of you have suggestions for improvements. My apologies again for sending incorrect information before - I had typed it from memory without checking, which is always a dangerous gamble!

[TeaganKing]

Thank you very much @briandobbins for all of your work on this. I verified that the updated docker command worked for me and that I was able to successfully run the visualization component with these updates. It also appears that run_neon is working smoothly. With all of that tested, I think we should be good to go. @tkavoo , perhaps you can just double check that it works on your system as well (although I wouldn't expect us to get different results). Given this update, I agree that the 9pm zoom call should not be necessary.

[tkavoo]

Thank you guys, will as well confirm from my end and give a feedback but I m sure it will work. Yes, I agree we do not need the 9pm meeting. Thank you so much @briandobbins and @TeaganKing.

[briandobbins]

Glad to hear it's working for you, @TeaganKing ! I'd caution that I'm still getting an occasional issue on the second notebook (NEON_Visualization_Tutorial), which seems to happen when loading the files via xarray. Removing the 'parallel=True' option has worked for me, and makes sense here, but I don't know why it's failing (sometimes) when set to True.

Once you and @tkavoo have both verified it's working as expected, I'll add that fix (removing 'parallel=True'), and update the 'official' tag, 'cesm-lab-neon'. And I'll still plan on being online a little at 9pm, just in case. :)

[TeaganKing]

Yes, I did also remove that 'parallel=True' to get things working, thanks for pointing that out in this thread. Previously, @tkavoo and @cwsauburn were having issues with the download_eval_files() function itself, which is properly downloading files now. The rest of the notebook also was working as I expected. Although, I also don't understand the 'why' behind the parallel option failure.

[briandobbins]

OK, I'll let @tkavoo try things out before making a final version with the only 'new' fix being to remove the 'parallel=True' bit from the second notebook.

I'm guessing the 'download_eval_files' function issue was related to the change in the NEON S3 bucket, and this tag has the correct location now? If problems persist there, we should look at a more resilient solution which doesn't need updating a container / tag to find the data (eg, some redirect we control).

[briandobbins]

OK, I've pushed an updated 'escomp/cesm-lab-neon' with these fixes. @cwsauburn you'll need to do a 'docker pull escomp/cesm-lab-neon' to grab the update, but everything should work out of the box again.

Later on, @wwieder and @TeaganKing we should also update these notebooks to switch from the deprecated functions (which give warnings) to the newer syntax, but that can wait for a new release. I'd also like to understand why the parallel loading via xarray was problematic, and do some performance testing of it, but I don't expect differences on a single hard-drive (vs a parallel file system).

Thank you, Teagan and @tkavoo !

[tkavoo]

Thank you so much @briandobbins and @TeaganKing. I have successfully run the 'escomp/cesm-lab-neon' image on docker successfully both the simulation and visualization notebooks. Much appreciation.

[wwieder]

Seems like this issue has been resolved