search

tnigon / hs_process

An open-source Python package for geospatial processing of aerial hyperspectral imagery
https://hs-process.readthedocs.io/en/latest/
MIT License
6 stars 1 forks source link

Crop image issue #2

Closed Yazhou-Sun closed 3 years ago

Yazhou-Sun commented 3 years ago

Hi, I am cropping UAV hyperspectral imagery with plot boundary files, there is one issue with this line of code:

array_crop, metadata = my_spatial_mod.crop_single(plot_id=1)

KeyError                                  Traceback (most recent call last)
<ipython-input-43-458553a456fd> in <module>
----> 1 array_crop, metadata = my_spatial_mod.crop_single(plot_id=1)

~\Anaconda3\envs\spec\lib\site-packages\hs_process\spatial_mod.py in crop_single(self, pix_e_ul, pix_n_ul, crop_e_pix, crop_n_pix, crop_e_m, crop_n_m, buf_e_pix, buf_n_pix, buf_e_m, buf_n_m, spyfile, plot_id, gdf, name_append)
    909                     "".format(pix_e_ul, pix_n_ul, pix_e_lr, pix_n_lr))
    910         # If "..crop_single" is already included in the history, remove it
--> 911         idx_remove = metadata['history'].find(
    912                 ' -> hs_process.crop_single[<')
    913         if idx_remove == -1:

KeyError: 'history'

Any idea how to solve the issue? Thanks.

tnigon commented 3 years ago

The error tells me that the image does not have a "history" key in its metadata. A workaround might be to simply add a "history" tag to the metadata, even if it is blank or just has a dummy variable. Can you provide more of your code? What is the image type you have loaded into spatial_mod? Can you share the contents of your metadata? I'll try to help you out if I can. Thanks!

Yazhou-Sun commented 3 years ago

The error tells me that the image does not have a "history" key in its metadata. A workaround might be to simply add a "history" tag to the metadata, even if it is blank or just has a dummy variable. Can you provide more of your code? What is the image type you have loaded into spatial_mod? Can you share the contents of your metadata? I'll try to help you out if I can. Thanks!

My code is about the same as section 6. Tutorial: spatial_mod: The metadata seems not defined. But I am not quite sure how to create the metadata. 

import geopandas as gpd
import os
from hs_process import hsio
from hs_process import spatial_mod

fname_in = r'E:\hyperspectral_imagery1.hdr'
fname_gdf = r'E:\plotboundary.geojson'
gdf = gpd.read_file(fname_gdf)
io = hsio(fname_in)
my_spatial_mod = spatial_mod(io.spyfile)
dir_out = os.path.join(io.base_dir, 'spatial_mod', 'crop_many_gdf')
if not os.path.isdir(os.path.join(io.base_dir, 'spatial_mod')):  # create a new folder named "spatial_mod" if it does not exist
    os.mkdir(os.path.join(io.base_dir, 'spatial_mod'))
if not os.path.isdir(dir_out):  # create a new folder named "crop_many_gdf" if it does not exist
    os.mkdir(dir_out)

name_append = '-crop-many-gdf'
df_plots = my_spatial_mod.crop_many_gdf(spyfile=io.spyfile, gdf=gdf)
array_crop, metadata = my_spatial_mod.crop_single(plot_id=1)
tnigon commented 3 years ago

Thanks for the information. Can you open up the .hdr file (E:\hyperspectral_imagery1.hdr) with a text editor and look at the contents? Here is a sample of the image I used in the example/tutorial:

ENVI
samples = 3
lines = 3
bands = 240
header offset = 0
file type = ENVI Standard
data type = 4
interleave = bip
byte order = 0
reflectance scale factor = 1.0
map info = {UTM, 1.0, 1.0, 441371.00707299996, 4855941.7317699995, 0.04, 0.04, 15, T, WGS-84, units  meters, rotation  0.000}
history = input cube -> Radiance Conversion<SpecGroup>[<SpecFilename label:'Imager Calibration' value:J:\\\\Shared Work\\\\Data\\\\PikaImagery2_Radiance\\\\Pika II 100106_11 Radiometric Calibration September 2016\\\\8mm\\\\PikaII_100106_11_8mm_RadiometricCalibration_Sept2016.icp>, <SpecCube label:'Dark Noise Cube' value:None>, <SpecBool label:'Auto Remove Dark Noise?' value:True>, <SpecBool label:'Return floating point?' value:False>] -> Georectify Airborne Datacube<SpecGroup>[[<SpecBool label:'Use Flat Earth Altitude?' value:False>, <SpecFloatText label:'Flat Earth Alt (m)' value:340>, <SpecFloatText label:'Field of View (deg)' value:33.0>, <SpecInfo label:'Calculated Resolution (m)' value:Not Available in Batch Mode>, <SpecFloatText label:'Map Resolution (m)' value:0.04>, <SpecBool label:'Select DEM File' value:False>], [<SpecFloatText label:'Sync Offset (s)' value:-0.05>, <SpecFloatText label:'Imager Roll Offset (deg)' value:0>, <SpecFloatText label:'Imager Pitch Offset (deg)' value:2.25>, <SpecFloatText label:'Imager Heading Offset (deg)' value:0>], [<SpecBool label:'Crop Cube to Start/End Lines?' value:False>, <SpecIntText label:'Start Line' value:0>, <SpecIntText label:'End Line' value:1906>], [<SpecBool label:'Select Folder to Save Output' value:False>, <SpecBool label:'Save Products in Source Folder?' value:False>, <SpecBool label:'Generate Full Datacube?' value:True>, <SpecBool label:'Generate KML of Render?' value:True>, <SpecBool label:'Generate GeoTIFF Render?' value:True>, <SpecBool label:'Generate Swath Outline?' value:True>, <SpecChoice label:'Interpolate Image?' value:nearest>, <SpecChoice label:'Interpolate Datacube?' value:linear>, <SpecChoice label:'Render To Use in KML/GeoTIFF' value:NA>, <SpecBool label:'Apply Stretch/Filter?' value:True>, <SpecBool label:'Save Interpolated LCF?' value:False>, <SpecBool label:'Save Mask of Interpolated Pixels?' value:False>]] -> Convert Radiance Cube to Reflectance from Measured Reference Spectrum<SpecGroup>[<SpecSpectrum label:'ROI Spec.' value:spec28>, <SpecBool label:'Measured reflectivity in Percentage?' value:False>, <SpecChoice label:'Scale 100% Reflectivity To:' value:1.0 (floats)>] -> hs_process.crop_single[<SpecPyFloatText label: 'pix_e_ul?' value:342; SpecPyFloatText label: 'pix_n_ul?' value:75; SpecPyFloatText label: 'pix_e_lr?' value:345; SpecPyFloatText label: 'pix_n_lr?' value:78>]
label = Wells_rep2_20180628_16h56m_test_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip
wavelength = {394.6, 396.6528, 398.7056, 400.7584, 402.8112...}
band names = {1, 2, 3, 4, 5...}

Notice the "history" tag that describes the history this particular image has gone through to get where it is now. All of the images I work with have this history tag, but if yours do not, then that is probably the cause of the problem.

Yazhou-Sun commented 3 years ago

Thanks for the information. Can you open up the .hdr file (E:\hyperspectral_imagery1.hdr) with a text editor and look at the contents? Here is a sample of the image I used in the example/tutorial:

ENVI
samples = 3
lines = 3
bands = 240
header offset = 0
file type = ENVI Standard
data type = 4
interleave = bip
byte order = 0
reflectance scale factor = 1.0
map info = {UTM, 1.0, 1.0, 441371.00707299996, 4855941.7317699995, 0.04, 0.04, 15, T, WGS-84, units  meters, rotation  0.000}
history = input cube -> Radiance Conversion<SpecGroup>[<SpecFilename label:'Imager Calibration' value:J:\\\\Shared Work\\\\Data\\\\PikaImagery2_Radiance\\\\Pika II 100106_11 Radiometric Calibration September 2016\\\\8mm\\\\PikaII_100106_11_8mm_RadiometricCalibration_Sept2016.icp>, <SpecCube label:'Dark Noise Cube' value:None>, <SpecBool label:'Auto Remove Dark Noise?' value:True>, <SpecBool label:'Return floating point?' value:False>] -> Georectify Airborne Datacube<SpecGroup>[[<SpecBool label:'Use Flat Earth Altitude?' value:False>, <SpecFloatText label:'Flat Earth Alt (m)' value:340>, <SpecFloatText label:'Field of View (deg)' value:33.0>, <SpecInfo label:'Calculated Resolution (m)' value:Not Available in Batch Mode>, <SpecFloatText label:'Map Resolution (m)' value:0.04>, <SpecBool label:'Select DEM File' value:False>], [<SpecFloatText label:'Sync Offset (s)' value:-0.05>, <SpecFloatText label:'Imager Roll Offset (deg)' value:0>, <SpecFloatText label:'Imager Pitch Offset (deg)' value:2.25>, <SpecFloatText label:'Imager Heading Offset (deg)' value:0>], [<SpecBool label:'Crop Cube to Start/End Lines?' value:False>, <SpecIntText label:'Start Line' value:0>, <SpecIntText label:'End Line' value:1906>], [<SpecBool label:'Select Folder to Save Output' value:False>, <SpecBool label:'Save Products in Source Folder?' value:False>, <SpecBool label:'Generate Full Datacube?' value:True>, <SpecBool label:'Generate KML of Render?' value:True>, <SpecBool label:'Generate GeoTIFF Render?' value:True>, <SpecBool label:'Generate Swath Outline?' value:True>, <SpecChoice label:'Interpolate Image?' value:nearest>, <SpecChoice label:'Interpolate Datacube?' value:linear>, <SpecChoice label:'Render To Use in KML/GeoTIFF' value:NA>, <SpecBool label:'Apply Stretch/Filter?' value:True>, <SpecBool label:'Save Interpolated LCF?' value:False>, <SpecBool label:'Save Mask of Interpolated Pixels?' value:False>]] -> Convert Radiance Cube to Reflectance from Measured Reference Spectrum<SpecGroup>[<SpecSpectrum label:'ROI Spec.' value:spec28>, <SpecBool label:'Measured reflectivity in Percentage?' value:False>, <SpecChoice label:'Scale 100% Reflectivity To:' value:1.0 (floats)>] -> hs_process.crop_single[<SpecPyFloatText label: 'pix_e_ul?' value:342; SpecPyFloatText label: 'pix_n_ul?' value:75; SpecPyFloatText label: 'pix_e_lr?' value:345; SpecPyFloatText label: 'pix_n_lr?' value:78>]
label = Wells_rep2_20180628_16h56m_test_pika_gige_7-Radiance Conversion-Georectify Airborne Datacube-Convert Radiance Cube to Reflectance from Measured Reference Spectrum.bip
wavelength = {394.6, 396.6528, 398.7056, 400.7584, 402.8112...}
band names = {1, 2, 3, 4, 5...}

Notice the "history" tag that describes the history this particular image has gone through to get where it is now. All of the images I work with have this history tag, but if yours do not, then that is probably the cause of the problem.

Yes, here it is:

ENVI
description = {HEADWALL Hyperspec III[HEADWALL Hyperspec III],[HEADWALL Hyperspec III OR]}
samples = 720
lines = 11182
bands = 274
header offset = 0
file type = ENVI Standard
data type = 12
interleave = bsq
sensor type = Unknown
byte order = 0
wavelength units = nm
map info = {Geographic Lat/Lon,1,1,-89.38681637930115,43.29648597737143,3.080735373867722e-7,2.250256219444636e-7,WGS84,units=Degrees}
wavelength = {
399.309
,401.521
,403.732
,405.944
... ...
,998.646
,1000.86
,1003.07
}
;Correct Position = 1
;Firmware = build: Oct 10 2019_09:59:10
;GSDO = 0.025
;HO = 307.0516052246094
;Invert Columns = 1
;Latitude Offset (m) = 0
;Longitude Offset (m) = 0
;Ortho DEM File = C:\Headwall\dem\usgs_ned_13_n44w090_gridfloat.hdr
;Ortho GPS Offset Altitude Offset(m) = 33
;Ortho GPS Offset Front Up Positive = 0
;Ortho GPS Offset Geoid Correction = 1
;Ortho GPS Offset North-East Positive = 1
;Ortho GPS Offset Pitch = -0.4
;Ortho GPS Offset Right Positive = 1
;Ortho GPS Offset Roll = 0
;Ortho GPS Offset Time Offset(ms) = 0
;Ortho GPS Offset Yaw = 3
;Ortho Lems EFL (mm) = 12
;Ortho Lens Pixel Pitch(um) = 7.4
;Ortho Lens Spatial Binning = 1
;Ortho Resolution (m) = 0.025
;Ortho Running Avg. of Altitude = 0
;Ortho Running Avg. of Latitude = 0
;Ortho Running Avg. of Longitude = 0
;Ortho Running Avg. of Pitch = 0
;Ortho Running Avg. of Raw = 0
;Ortho Running Avg. of Roll = 0
;Ortho Running Avg. of X = 0
;Ortho Running Avg. of Y = 0
;Ortho Running Avg. of Yaw = 0
;Ortho Running Avg. of Z = 0
;Ortho Sensor Alpha = 0
;Ortho Sensor Beta = 0
;Ortho Sensor Gamma = 0
;Ortho Sensor Invert Columns = 1
;Row binning = 1
;Serial Number = nHS-213
;averagePixelDisp = 2.21157549723685
;bBinaryMap = 000007FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFE
;framingCamera = 0
;gain = 3
;hsiii_version = Hyperspec v3.1.0
;invertColumns = 1
;left = 0
;pixel0 = 158.247496218906
;top = 88
tnigon commented 3 years ago

As I suspected, it looks to be missing the "history" tag. For now until I am able to fix the code, can you manually add history = none into the .hdr file?

I see all the extra metadata with a leading semicolon - I'm also not sure how hs_process will behave with these metadata tags. It would be good to get a sample image from you to debug. I've only tested with Resonon imagery.

tnigon commented 3 years ago

@Yazhou-Sun, would you mind uploading a single hyperspectral image to this cloud directory? I can use it as a test image to make sure the code all works with your images. Thanks!

Yazhou-Sun commented 3 years ago

@Yazhou-Sun, would you mind uploading a single hyperspectral image to this cloud directory? I can use it as a test image to make sure the code all works with your images. Thanks!

Thank you so much for your help! I don't have permission to upload files to this drive. would you mind adding me (ysun382@wisc.edu) as an editor or so?

tnigon commented 3 years ago

@Yazhou-Sun, would you mind uploading a single hyperspectral image to this cloud directory? I can use it as a test image to make sure the code all works with your images. Thanks!

Thank you so much for your help! I don't have permission to upload files to this drive. would you mind adding me (ysun382@wisc.edu) as an editor or so?

Okay, give it a try now - sorry!

tnigon commented 3 years ago

The fixes to this issue are merged into the dev branch. Give it a try and let me know how it goes with your Headwall imagery!