basic python help needed

[wwieder]

I started working on a quick plot tutorial

Specifically I'm setting up a path the the folder where files are located

neon_site = 'CPER'  #NEON site we're going to look at
archive = '~/scratch/NEON_cases/archive/'  #Path to the directory with archived cases
data_folder = archive + neon_site + '.transient/lnd/hist'  #directory with input data we're going to open
data_folder

Here all looks correct, pointing to ~/scratch/NEON_cases/archive/CPER.transient/lnd/hist

When I try to glob this data folder, I only get an empty list in return

glob(os.path.join(data_folder,'*.nc'))

[ ]

Suggestions here would be most helpful.

[wwieder]

@teaganking or @samsrabin can you help with this basic question?

[samsrabin]

Does it work if you write out the full path to your home directory instead of using ~?

If not, and sorry if this is insulting but it's always good to check: Is ~/scratch/NEON_cases/archive/CPER.transient/lnd/hist actually a directory on your system?

[wwieder]

oh you're right, it doesn't like the ~ and the softlink that's in home to my scratch directory.

[wwieder]

is there another convenient way to do this in a way that's agnostic to running in the cloud or on cheyenne. I've been using a softlink in home to go to scratch ~/scratch. This let's other parts of the tutorial work on either machine (once users add a similar to softlink to /glade/scratch/$USER/. Is there a similar python work around?

[samsrabin]

Not sure if this will work, but try putting the path with ~/scratch through os.path.realpath().

Failing that, you could instruct users to (bash) export SCRATCH="/path/to/scratch/dir" before starting Python. Then in the Python script they could do os.environ["SCRATCH"] to get its value.

[wwieder]

yep, os.path.realpath() gets rid the the ~, but glob still does't like the soft link that it points to, I'll keep trying. Thanks for your suggestions.

[samsrabin]

Okay, it looks like this should work:

os.path.realpath(os.path.expanduser("~/scratch"))

For example, on Cheyenne:

>>> os.path.realpath(os.path.expanduser("~samrabin/inputdata"))
'/glade/p/cesmdata/cseg/inputdata'
>>> glob.glob(os.path.realpath(os.path.expanduser("~samrabin/inputdata/*txt")))
['/glade/p/cesmdata/cseg/inputdata/list_prism_restart.txt', '/glade/p/cesmdata/cseg/inputdata/files_201.txt', '/glade/p/cesmdata/cseg/inputdata/list_prism.txt', '/glade/p/cesmdata/cseg/inputdata/files.txt', '/glade/p/cesmdata/cseg/inputdata/files_200.txt', '/glade/p/cesmdata/cseg/inputdata/lake_list.txt']

[wwieder]

Thanks @samsrabin that did the trick!