Unable to download Met on remote

[mccabete]

When doing a remote run using ED2 with a GFDL met source, the workflow succesfully executes :

remotefunc <- function() {PEcAn.data.atmosphere::download.GFDL(site_id=853, lat.in=35.9667, lon.in=-79.0833, model=NULL, scenario=NULL, ensemble_member=NULL, overwrite=FALSE, outfolder='/projectnb/dietzelab/pecan.data/output/tmccabe/GFDL_site_0-853/', start_date='2006-01-01', end_date='2100-12-31')}
> remoteout <- remotefunc()

But then the connection fails almost immediately.

fp <- file('/projectnb/dietzelab/pecan.data/output/tmccabe/GFDL_site_0-853//pecan-5af1a61b62004a12fa0de5a03d269e', 'w')
Error in file("/projectnb/dietzelab/pecan.data/output/tmccabe/GFDL_site_0-853//pecan-5af1a61b62004a12fa0de5a03d269e",  : 
  cannot open the connection
In addition: Warning message:
In file("/projectnb/dietzelab/pecan.data/output/tmccabe/GFDL_site_0-853//pecan-5af1a61b62004a12fa0de5a03d269e",  :
  cannot open file '/projectnb/dietzelab/pecan.data/output/tmccabe/GFDL_site_0-853//pecan-5af1a61b62004a12fa0de5a03d269e': No such file or directory
Execution halted

Full workflow can be found here.

[mccabete]

This is also true of Jules. See workflow.

[mccabete]

This is also true across met: Cruncep gives the same error. See workflow

[mccabete]

Seems to be a general met bug. Doesn't work for fluxnet2015, or AmerifluxLBL

[ashiklom]

There are a few of things that could be happening here:

• Target directories are not being created correctly on the remote. E.g. R tries to write to .../GFDL_site5/filename, but the GFDL_site5 directory doesn't exist. The recently created fs package provides some really nice utilities that facilitate a lot of these operations (for instance, creating directory hierarchies recursively; nothing you can't do with base R packages, but fs makes it a lot easier and the syntax is much more consistent).
• Thredds connections, for some bizarre reason, may work on some systems but not others. This could be caused by firewalls, issues with curl, and/or issues with the netcdf library. This is a much harder problem to solve.

[ashiklom]

Mental note: This may be related to #1288.

[github-actions[bot]]

This issue is stale because it has been open 365 days with no activity.