Move data and processing from ROGER to Condo

terraref / computing-pipeline

Pipeline to Extract Plant Phenotypes from Reference Data

BSD 3-Clause "New" or "Revised" License

21 stars 13 forks source link

Move data and processing from ROGER to Condo #378

Closed craig-willis closed 6 years ago

craig-willis commented 6 years ago

We are in the process of syncing data from ROGER to the Condo. This issue tracks all steps to move both data and processing to use the Condo mounts by 12/10.

Tasks:

[x] Change Globus to use Condo endpoint
- [x] Create endpoint on Condo (JD)
- [x] Change scripts in AZ (JD)
- [x] Disable/confirm Danforth
[x] Complete sync from ROGER to Condo
[x] Clowder/Kubernetes
- [x] Shutdown RabbitMQ
- [x] Change Clowder mounts
- [x] Change Kubernetes mounts
- [x] Start RabbitMQ

To start processing, prioritize the following:

[ ] RGB raw, level_1, fullfield
[ ] FLIR raw, level_1, fullfield

jdmaloney commented 6 years ago

@craig-willis NFS mount information: host: cnfs.ncsa.illinois.edu Note: Since we have our own file system on the condo, paths are truncated. All paths start at /terraref/...

max-zilla commented 6 years ago

Folder requirements for extractors...

canopycover/meantemp

Level_1/fullfield

bin2tif

raw_data/stereoTop
Level_1/rgb_geotiff

flir2tif

raw_data/flirIrCamera
Level_1/ir_geotiff

ply2las

raw_data/scanner3DTop
Level_1/scanner3DTop
Level_1/laser3d_mergedlas

plant_height

raw_data/scanner3DTop
Level_1/scanner3DTop
Level_1/laser3d_heightmap

craig-willis commented 6 years ago

Update on this -- there has been some delay, but the endpoint will likely be up on Condo by end of week.

craig-willis commented 6 years ago

Globus endpoint is available, but there will be another migration in ~2 months (tracked as separate issue). Nothing to do for Danforth until the endpoint changes. Starting back on full-field analysis now.

max-zilla commented 6 years ago

gpfs scratch is last remaining data, covered specifically in issue #394