Modification to S3Download() and more...

[ghost]

This branch started as my modifications to the S3Download() class but now holds more than just that. As for the S3Download() mods:

• Hardcoded the destination for downloaded files to /mnt. It also does chown ubuntu:ubuntu when the class is instantiated.
• The methods usedspace() and freespace() are turned to properties and use the shutil.disk_usage() to collect information.

Other notable changes so far:

• summary.py checks whether HDF5 file exists first prior to checking if the same file is valid HDF5. Error messages include node name (using platform.node()) so now one can ssh into that VM and examine the error cause.
• ncep_summary.sh runs summary.py in the cluster mode.

[jreadey]

Nice updates!

Hardcoding to /mnt seems a little dangerous (esp. as we have a function to rm -rf all files in this directory).

How about reverting to the env variable and I'll update the engine snapshot to create /mnt/s3 on startup if it is not present?

BTW - I was planning to update summary.py to use s3download today.

[ghost]

Hardcoding to /mnt seems a little dangerous (esp. as we have a function to rm -rf all files in this directory).

/mnt is meant to be place for ephemeral data.

How about reverting to the env variable and I'll update the engine snapshot to create /mnt/s3 on startup if it is not present?

I wanted to avoid having to build a new snapshot each time this location changes but am not religious about it. Change if you want.

BTW - I was planning to update summary.py to use s3download today.

Great! Go for it.