Open dhardestylewis opened 2 years ago
include in 2nd grand retiling if possible
updating TACC's TNRIS Lidar DEM collection against the current TNRIS AWS S3 now
All TNRIS Lidar DEMs & LPCs can be found at the following location at TACC:
stampede2.tacc.utexas.edu:/scratch/projects/tnris/tnris-lidardata
Please contact me if you have any issues with accessing the data.
The following Lidar collections are being pulled now:
stratmap-2021-28cm-50cm-bexar-travis
stratmap-2021-laguna-madre-bathy
stratmap-2022-50cm-ellis-hill-johnson-navarro
stratmap-2022-address-points
stratmap-2022-land-parcels
usgs-hydrography
utbeg-2017-70cm-laguna-madre
Here is how they are being pulled:
cd /scratch/projects/tnris
while IFS= read -r line ; do aws s3 cp s3://tnris-public-data/production-data/${line}/ tnris-lidardata/${line} --recursive & done < tnris-missing-data.csv
tnris-missing-data.csv
was produced using
cd /scratch/projects/tnris
aws s3 ls s3://tnris-public-data/production-data/ > tnris-public-data.csv
## use vim magic to modify above file slightly to strip out everything but the collection names
ls tnris-lidardata > tnris-tacc-data.csv
diff --suppress-common-lines -y tnris-public-data.csv tnris-tacc-data.csv > tnris-missing-data.csv
## use vim magic to capture the missing collections on the left side of the diff
I have applied the following permissions against this dataset:
cd /scratch/projects/tnris
find . -type d -exec bash -c "chgrp G-822180 {} ; chmod 755 {} ; chmod g+s {} ; setfacl -d -m g::rx,o::rx {}" \;
find . -type f -exec bash -c "chgrp G-822180 {} ; chmod 644 {}" \;
https://github.com/dhardestylewis/terrain_aggregator/issues/79
terrain_aggregator
DB paused to return TNRIS, TDIS, & CWE requests for small-scale 1m accuracy results https://github.com/dhardestylewis/terrain_aggregator/issues/39#issuecomment-1220088846
[x] TNRIS Lidar DEM tile collections have been updated successfully!
[x] This includes all TNRIS Lidar point-cloud data!
cd /scratch/projects/tnris
while IFS= read -r directory ; do echo "${directory} $(~/local/bin/aws/aws s3 ls s3://tnris-public-data/production-data/${directory}/ --recursive | wc -l)" ; done < tnris-missing-data.csv
stratmap-2021-28cm-50cm-bexar-travis 48922
stratmap-2021-laguna-madre-bathy 189742
stratmap-2022-50cm-ellis-hill-johnson-navarro 13156
stratmap-2022-address-points 19459
stratmap-2022-land-parcels 18532
usgs-hydrography 956
utbeg-2017-70cm-laguna-madre 680
cd /scratch/projects/tnris
while IFS= read -r directory ; do echo "${directory} $(find tnris-lidardata/${directory} -type f | wc -l)" ; done < tnris-missing-data.csv
stratmap-2021-28cm-50cm-bexar-travis 48922
stratmap-2021-laguna-madre-bathy 189742
stratmap-2022-50cm-ellis-hill-johnson-navarro 13156
stratmap-2022-address-points 19459
stratmap-2022-land-parcels 18532
usgs-hydrography 956
utbeg-2017-70cm-laguna-madre 680
Update TNRIS Lidar dataset & database against new projects in TNRIS S3 bucket
for example,
stratmap21-50cm
is reported in the 2022/05/09 TNRIS availability file but not interrain_aggregator
DB