Closed gerritholl closed 4 years ago
gerritholl
commented
4 years ago Tests, even including slow ones, are still succeeding:
87 passed, 2 skipped, 1 xfailed, 327 warnings in 786.84s (0:13:06)
so there must be something going on not covered by any tests.
gerritholl
commented
4 years ago This appears to happen when some of the neighbours are masked:
ids == array([[False, False, False],
[False, True, True],
[False, False, True]])hence we have tneigh.shape = (6,). But idmargin is set based on idneigh, set before tneigh and apparently resulting in assumptions of a different shape under circumstances yet to be clarified.
gerritholl
commented
4 years ago Fogpy appears to look for the IR108 BT of non-cloudy neighbours of a cloudy pixel. When one or more of those are masked, the result is incorrect or there is an IndexError:
In : print(self.clusters[index[0]-2:index[0]+3, index[1]-2:index[1]+3].data)
[[0 0 0 0 0]
[0 0 0 0 0]
[0 1 1 0 0]
[0 1 1 0 0]
[1 1 0 0 0]]
In : with np.printoptions(precision=3): print(self.ir108[index[0]-2:index[0]+3, index[1]-2:inde
x[1]+3].data)
...:
[[ nan 260.581 260.581 260.581 nan]
[ nan 260.581 260.581 260.581 nan]
[261.668 261.668 261.668 nan nan]
[261.668 261.668 261.668 nan 261.668]
[262.385 262.385 nan nan 261.848]]The neighbours are labelled 0..7, and cloudfree neighbours are in the variable idmargin. The variable tneigh contains the temperatures for the neighbours. The code assumes that tneigh can be safely indexed with idmargin, but when there are any masked data among the neighbours in ir108, then tneigh will have a size smaller than 8, and the indexing may be wrong.
The solution would be to extract the temperatures for the neighbours in a different way.
My next step will be to refactor the relevant code, put the functionality in smaller units and unit test those, including for this case. That unit test should cover at least:
IndexError (wrong temperature extracted)IndexError (case here)
When trying to regenerate the fog composite for 2019-04-15, that I have done before, I get an
IndexError. The full command below uses theshow-fogcommand from fogtools:It fails in
fogpy.algorithms.LowCloudHeightAlgorithm.procedurewhenidmargin == [0, 1, 2, 4, 7]buttneigh.shape = (6,):