Globus Transfers

[nocollier]

We should provide an option to transfer data via globus transfers, in the form: to_dataset_dict(globus_endpoint='My Kewl Endpoint', dest_path='additional/path').

• [x] get refresh tokens for transfer clients
• [x] prototype transfer functionality
• [x] partition file info by the fastest globus endpoint(s) that is(are) active, and those that have only https
• [x] monitor progress and block until completion
• [x] while waiting for globus, allow any http only files to download
• [x] to_dataset_dict() is too big, refactor and lift out major components to make room for globus transfers
• [x] once transfer is complete, check for file existence, first in cache in case it was setup appropriately, but also in root if we cannot find.
• [ ] add paths to locations we check in the config
• [x] rework config #37 to allow for more locations to be added easily, also fix #26 and #30

This feature is a precursor for enabling globus flows / connect.

[jsta]

This is a great feature! For whatever reason, the ORNL endpoint is not working for me, only the ANL one.

> file not found:

{
  "context": [
    {
      "operation": "Directory List / File Scan",
      "path": "/~/css03_data/CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p1f1/Amon/pr/gn/v20190429/pr_Amon_CanESM5_historical_r1i1p1f1_gn_185001-201412.nc"
    }
  ],
  "error": {
    "body": "550-GlobusError: v=1 c=PATH_NOT_FOUND%0D%0A550-GridFTP-Errno: 2%0D%0A550-GridFTP-Reason: System error in stat%0D%0A550-GridFTP-Error-String: No such file or directory%0D%0A550 End.%0D%0A",
    "code": 550,
    "endpoint": "ORNL ESGF Data Node (dea29ae8-bb92-4c63-bdbc-260522c92fe8)",
    "server": "gsiftp://128.219.128.11:443",
    "type": "FTPServerError"
  }
}

The endpoint works if navigate to an arbitrary file through the web interface (app.globus.org) so I'm fairly certain it's not a firewall issue on my end.

[nocollier]

Thanks for trying this out. I am finding the same thing. I will bring this up again with our team--I believe that it is an issue with the data not being where it is supposed to be. So you can find it, but the way it is stored in the index does not match reality. For the moment, just remove the 'ornl-dev' from your indices :/ Hopefully short term solution.

On Thu, Apr 25, 2024 at 11:11 AM Jemma Stachelek @.***> wrote:

This is a great feature! For whatever reason, the ORNL endpoint is not working for me, only the ANL one.

file not found:

{ "context": [ { "operation": "Directory List / File Scan", "path": "/~/css03_data/CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p1f1/Amon/pr/gn/v20190429/pr_Amon_CanESM5_historical_r1i1p1f1_gn_185001-201412.nc" } ], "error": { "body": "550-GlobusError: v=1 c=PATH_NOT_FOUND%0D%0A550-GridFTP-Errno: 2%0D%0A550-GridFTP-Reason: System error in stat%0D%0A550-GridFTP-Error-String: No such file or directory%0D%0A550 End.%0D%0A", "code": 550, "endpoint": "ORNL ESGF Data Node (dea29ae8-bb92-4c63-bdbc-260522c92fe8)", "server": "gsiftp://128.219.128.11:443", "type": "FTPServerError" } }

The endpoint works if navigate to an arbitrary file through the web interface (app.globus.org) so I'm fairly certain it's not a firewall issue on my end.

— Reply to this email directly, view it on GitHub https://github.com/esgf2-us/intake-esgf/issues/19#issuecomment-2077521876, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKFCBYZZVXDXFGZFZEVRMLY7EMKTAVCNFSM6AAAAABCJAHQPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZXGUZDCOBXGY . You are receiving this because you authored the thread.Message ID: @.***>