search

NOAA-GFDL / pace

Re-write of FV3GFS weather/climate model in Python
Apache License 2.0
12 stars 11 forks source link

Unit test: test_restart_serial.py OSError regarding RESTART directory not empty #25

Closed fmalatino closed 11 months ago

fmalatino commented 11 months ago

Upon a run of the test_restart_serial.py unit test in the tests/main/driver directory, on both the main repo and my fork for fv3core reorganization, the test fails due to an OSError:

FAILED tests/main/driver/test_restart_serial.py::test_restart_save_to_disk - OSError: [Errno 39] Directory not empty: 'RESTART'

The RESTART directory does appear empty during investigation although.

bensonr commented 11 months ago

Should we be using the built-in error handling functionality of shutil.rmtree and also invoke a buffer flush by using "import os ; os.sync()" prior to calling shutil.rmtree in finally?

fmalatino commented 11 months ago

I implemented this and the error still persists unfortunately.

lharris4 commented 11 months ago

Does it have some invisible files? This has hit me in the past a few times.

fmalatino commented 11 months ago

It is empty; ls -a only returns the links to the current and parent directories

fmalatino commented 11 months ago

Interestingly, the test passes in CI

bensonr commented 11 months ago

@fmalatino - since you're tests are potentially failing only on gaea, this may be an issue with either the lustre scratch filesystem or the NFS-mounted home filesystem - especially if they are using NFS caching.

FlorianDeconinck commented 11 months ago

Arguably the shutil.rmtree("RESTART") should be shutil.rmtree("RESTART", ignore_errors=True) since this is a utest clean up and shouldn't fail the test since the test is testing something else.

This is fine on my box and on NCCS's Discover (NFS cached filesystem)