Update orthoing code to handle SWIR and CAVIS

[clairecporter]

While making the changes needed to use 14 bits of input data for SWIR and CAVIS data, I took the opportunity to refactor some of the messy internals of ortho_functions.py and the godawful tests. This includes

1. Aligning the metadata reading structure and eliminating reading the text files twice
2. Making the ImageInfo class into a proper class instead of an empty bucket to throw things in
3. Adding checks for incompatible arguments
4. Some reformating of un-pythonic function names

Sadly, some of the tests may be beyond rescue, but I have left those I could not repair in the repo for now.

The recent NoData fix branch is also rolled in.

[bakkerbakker]

@bakkerbakker once this PR gets tested and merged, and Ali's auto DEM functionality gets merged, make a release for imagery_utils

[clairecporter]

@power720 I updated the README with info on how to run tests. At PGC the tests are in data/common/repos/repo_testdata/imagery_utils. Let me know if we should add more to the readme. We should convert the old tests to pytest and ditch the unusable ones at some point. But that is for another PR.

[power720]

Outputs from pgc_ortho.py appear unchanged. (as intended)

How I checked: I ran a pgc_ortho.py on a batch of images using code from this PR and the same batch using the common/repos code. Once created, I diff'd the outputs from gdalinfo from the corresponding images.

All of the diffs were some variations of the example below. Differences appear limited to insignificant floating-point values. Manual inspection of a few output images in QGIS also indicated that the outputs were unchanged.

2c2
< Files: /mnt/pgc/data/scratch/devin/ortho-pr/ortho-common-repos/GE01_20090705215213_1050410001E50F00_09JUL05215213-P1BS-507561715020_01_P001_u16rf32606.tif
---
> Files: /mnt/pgc/data/scratch/devin/ortho-pr/ortho-swir-bit-depth/GE01_20090705215213_1050410001E50F00_09JUL05215213-P1BS-507561715020_01_P001_u16rf32606.tif
44,45c44,45
< Origin = (422035.505074626125861,7782720.200997613370419)
< Pixel Size = (0.413395693300000,-0.416917544832000)
---
> Origin = (422035.505074621294625,7782720.200997664593160)
> Pixel Size = (0.413395693301000,-0.416917544833000)
181c181
<     STATISTICS_MEAN=206.22784829604
---
>     STATISTICS_MEAN=206.22784832118
183c183
<     STATISTICS_STDDEV=31.592251872396
---
>     STATISTICS_STDDEV=31.592251819676

New logs are much more verbose

The logging messages I commented on create a lot of more standard output than in the previous version of the script. Should this extra output be off by default and available to opt-in via the --verbose flag?

Full standard output from previous version:

$ bash pgc_ortho_common_repos.sh
/home/power720/miniforge3/envs/pgc/lib/python3.11/site-packages/osgeo/osr.py:385: FutureWarning: Neither osr.UseExceptions() nor osr.DontUseExceptions() has been explicitly called. In GDAL 4.0, exceptions will be enabled by default.
  warnings.warn(
04-16-2024 16:01:30 INFO- Running command: /mnt/pgc/data/common/repos/imagery_utils/pgc_ortho.py --format GTiff --outtype UInt16 --epsg=utm --dem /mnt/pgc/data/elev/dem/ned/alaskaNED/alaskaned_mosaic_vwgs84.tif --stretch rf --resample near --tasks-per-job 100 --slurm /home/power720/scratch/ortho-pr/renamed /home/power720/scratch/ortho-pr/ortho-common-repos
04-16-2024 16:01:30 INFO- Processing command: /mnt/pgc/data/common/repos/imagery_utils/pgc_ortho.py --format GTiff --outtype UInt16 --epsg=utm --dem /mnt/pgc/data/elev/dem/ned/alaskaNED/alaskaned_mosaic_vwgs84.tif --stretch rf --resample near --tasks-per-job 100 --slurm /home/power720/scratch/ortho-pr/renamed /home/power720/scratch/ortho-pr/ortho-common-repos
04-16-2024 16:01:31 INFO- Number of src images: 38
04-16-2024 16:01:33 INFO- Number of incomplete tasks: 38
Writing task bundle text files in directory: /home/power720/scratch/task_bundles
04-16-2024 16:01:33 INFO- Submitting Tasks
Submitted batch job 1219050
04-16-2024 16:01:33 INFO- Done

Trimmed standard output from PR:

$ bash pgc_ortho_swir_bit_depth.sh
04-16-2024 16:32:26 INFO- Running command: /home/power720/scratch/ortho-pr/imagery_utils/pgc_ortho.py --format GTiff --outtype UInt16 --epsg=utm --dem /mnt/pgc/data/elev/dem/ned/alaskaNED/alaskaned_mosaic_vwgs84.tif --stretch rf --resample near --tasks-per-job 100 --slurm /home/power720/scratch/ortho-pr/renamed /home/power720/scratch/ortho-pr/ortho-swir-bit-depth
04-16-2024 16:32:26 INFO- Processing command: /home/power720/scratch/ortho-pr/imagery_utils/pgc_ortho.py --format GTiff --outtype UInt16 --epsg=utm --dem /mnt/pgc/data/elev/dem/ned/alaskaNED/alaskaned_mosaic_vwgs84.tif --stretch rf --resample near --tasks-per-job 100 --slurm /home/power720/scratch/ortho-pr/renamed /home/power720/scratch/ortho-pr/ortho-swir-bit-depth
04-16-2024 16:32:26 INFO- Number of src images: 38
04-16-2024 16:32:26 INFO- Geographic extent: POLYGON Z ((-149.041111111111 70.1394444444445 0,-148.628055555556 70.1402777777778 0,-148.627777777778 69.9844444444444 0,-149.041111111111 69.9825 0,-149.041111111111 70.1394444444445 0))
04-16-2024 16:32:26 INFO- Automatically selected output projection EPSG code: 32606
04-16-2024 16:32:26 INFO- Projected extent: POLYGON Z ((422621.591274233 7782720.20099766 0,438279.334519624 7782341.48383028 0,437825.755827772 7764968.33373176 0,422035.505074621 7765226.56673174 0,422621.591274233 7782720.20099766 0))
04-16-2024 16:32:26 INFO- Original image size: 15662.322632 x 17503.449285, res: 1.529673076676 x 1.554066348641
04-16-2024 16:32:26 INFO- Geographic extent: POLYGON Z ((-149.038738521025 68.2403487243692 0,-148.599328719604 68.2367892215348 0,-148.597703543621 68.083735579242 0,-149.04073376696 68.0900191861167 0,-149.038738521025 68.2403487243692 0))
04-16-2024 16:32:26 INFO- Automatically selected output projection EPSG code: 32606
04-16-2024 16:32:26 INFO- Projected extent: POLYGON Z ((415668.677663328 7571055.08411419 0,433830.445491495 7570122.33000709 0,433455.988476124 7553060.93281567 0,415032.041568806 7554304.75990041 0,415668.677663328 7571055.08411419 0))
04-16-2024 16:32:26 INFO- Original image size: 18185.704299 x 16762.418286, res: 0.516947733008 x 0.574547327700
...
04-16-2024 16:32:28 INFO- Geographic extent: POLYGON Z ((-149.030105116295 69.075920977549 0,-148.596194255114 69.0704927328091 0,-148.597167624743 68.9910969066389 0,-149.031187048465 68.9965849112837 0,-149.030105116295 69.075920977549 0))
04-16-2024 16:32:28 INFO- Automatically selected output projection EPSG code: 32606
04-16-2024 16:32:28 INFO- Projected extent: POLYGON Z ((419100.226717474 7664166.59184101 0,436372.057123829 7663050.32058574 0,436102.8292286 7654200.86479256 0,418764.286570471 7655325.78864324 0,419100.226717474 7664166.59184101 0))
04-16-2024 16:32:28 INFO- Original image size: 17307.864891 x 8847.183561, res: 2.004152951768 x 2.062280550377
04-16-2024 16:32:28 INFO- Number of incomplete tasks: 38
Writing task bundle text files in directory: /home/power720/scratch/task_bundles
04-16-2024 16:32:28 INFO- Submitting Tasks
Submitted batch job 1219085
04-16-2024 16:32:28 INFO- Done

[power720]

I was able to get the tests setup and running.

I think I've reached the limit of my relevant repo knowledge, so I do not have any further comments. Let me know if you'd like to discuss them or if you have any more specific items that you'd like me to review closely.

[clairecporter]

The logging issue is because a lot of the work that used to be done by several different functions (all with different logging practices) and now they are all in a single class. When the class is instantiated as you did for submission to Rookery, the logging messages are overkill, but when the class is instantiated later on in the code within the same run (if not submitting to slurm) or within the slurm job, the log messages are expected even without --verbose set. I'll have to think about how to change the info level of logging depending on the context.

[clairecporter]

@power720, I simplified the logging for pgc_ortho.py to make it more useful. I also found an inefficiency in how the ImageInfo class was being used that I ironed out. I'll be merging this in shortly.