Error when using Sum after NormalizeMeanToMid

[zcanter]

When trying to do a regular Sum with results from the NormalizeMeanToMid command I get the below error. I have been able to work around it by using a WeightedSum and setting the input weights to 1; effectively mimicking a simple Sum.

TypeError                                 Traceback (most recent call last)
~\AppData\Local\ESRI\conda\envs\cbi\lib\site-packages\mpilot\commands.py in run(self)
    139                     **self.validate_params(
--> 140                         {arg.name: arg.value for arg in self.arguments}
    141                     )

~\AppData\Local\ESRI\conda\envs\cbi\lib\site-packages\mpilot\libraries\eems\basic.py in execute(self, **kwargs)
    363             where_idx = numpy.where(
--> 364                 numpy.logical_and(arr.data > prev_raw, arr.data <= raw)
    365             )

TypeError: '>' not supported between instances of 'memoryview' and 'int'

The above exception was the direct cause of the following exception:

UnexpectedError                           Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_23388/3467011737.py in <module>
     10 p.add_command(EEMSWrite, 'out', {'OutFileName': output_path, 'OutFieldNames': list(p.commands.keys())})
     11 
---> 12 p.run()
     13 
     14 p.to_file('EEMS_CMD_FILE_FULL_1.mpt')

~\AppData\Local\ESRI\conda\envs\cbi\lib\site-packages\mpilot\program.py in run(self)
    290             if not dependents.get(command.result_name)
    291         ):
--> 292             command.run()

~\AppData\Local\ESRI\conda\envs\cbi\lib\site-packages\mpilot\commands.py in run(self)
    144                 if isinstance(exc, MPilotError):
    145                     raise
--> 146                 raise_from(UnexpectedError(exc, format_exc(), self.lineno), exc)
    147 
    148             self.is_finished = True

~\AppData\Local\ESRI\conda\envs\cbi\lib\site-packages\six.py in raise_from(value, from_value)

UnexpectedError: Problem: An unexpected error occurred while running the model: '>' not supported between instances of 'memoryview' and 'int'

Traceback (most recent call last):
  File "C:\Users\zrcan\AppData\Local\ESRI\conda\envs\cbi\lib\site-packages\mpilot\commands.py", line 140, in run
    {arg.name: arg.value for arg in self.arguments}
  File "C:\Users\zrcan\AppData\Local\ESRI\conda\envs\cbi\lib\site-packages\mpilot\libraries\eems\basic.py", line 364, in execute
    numpy.logical_and(arr.data > prev_raw, arr.data <= raw)
TypeError: '>' not supported between instances of 'memoryview' and 'int'

Solution: Report this issue at https://github.com/consbio/mpilot/issues

[nikmolnar]

You say the problem is Sum after NormalizeMeanToMid, but it looks like this error is occuring while processing the NormalizeMeanToMid, which would suggest the problem is unrelated to the Sum that comes after it.

What comes before the NormalizeMeanToMid?

[nikmolnar]

MPilot does not use memoryview, so it's likely that your script is passing in a memoryview object rather than a numpy array as a data input at some point in the script. It might be intermittent, since this should have still failed when you switched to WeightedSum if the same memoryview object were still passed in.

[zcanter]

Ah, I misspoke it is not NormalizeMeanToMid that precedes the Sum, but NormalizeCat. It's just weird because it works fine if I use a WeightedSum as opposed to a Sum. The relevant code is below.

p.add_command(EEMSRead, 'brn_sev', {'InFileName': reporting_units_path, 
        'InFieldName': 'brn_sev',
        'DataType' : "Float",
        'Metadata': {'DisplayName' : 'Burn+Intensity+Raw'}})
    p.add_command(NormalizeCurve, 'nm_brn_sev', {'InFieldName': 'brn_sev',
        'RawValues': [0, 4, 4.01, 255],
        'NormalValues' : [0, 1, 0, 0],
        'Metadata': {'DisplayName' : 'Norm+Burn+Intensity'}})
    p.add_command(EEMSRead, 'ls_sus', {'InFileName': reporting_units_path, 
        'InFieldName': 'ls_sus',
        'DataType' : "Float",
        'Metadata': {'DisplayName' : 'Landslide+Susceptiblity+Raw'}})
    p.add_command(NormalizeCurve, 'nm_ls_sus', {'InFieldName': 'ls_sus',
        'RawValues': [0.0, 3.0, 6.0, 8.0, 10.0],
        'NormalValues' : [0, 0.25, 0.5, 0.75, 1.0],
        'Metadata': {'DisplayName' : 'Norm+Landslide+Susceptiblity'}})
    p.add_command(EEMSRead, 'lf_slp', {'InFileName': reporting_units_path, 
        'InFieldName': 'lf_slp',
        'DataType' : "Float",
        'Metadata': {'DisplayName' : 'Slope+Raw'}})
    p.add_command(NormalizeCurve, 'nm_lf_slp', {'InFieldName': 'lf_slp',
        'RawValues': [0.0, 22.5, 45.0],
        'NormalValues' : [0, 0.5, 1.0],
        'Metadata': {'DisplayName' : 'Norm+Slope'}})
    p.add_command(EEMSRead, 'shk_pot', {'InFileName': reporting_units_path, 
        'InFieldName': 'shk_pot',
        'DataType' : "Float",
        'Metadata': {'DisplayName' : 'Shaking+Potential+Raw'}})
    p.add_command(NormalizeCurve, 'nm_shk_pot', {'InFieldName': 'shk_pot',
        'RawValues': [0.5, 1.0, 1.25],
        'NormalValues' : [0, 0.5, 1.0],
        'Metadata': {'DisplayName' : 'Norm+Shaking+Potential'}})
    p.add_command(WeightedSum, 'lnd_rsk', {'InFieldNames': ['nm_shk_pot', 'nm_lf_slp', 'nm_ls_sus', 'nm_brn_sev'], 
        'Weights' : [0.2, 0.2, 0.3, 0.3],
        'Metadata': {'DisplayName' : 'Risk+of+Soil+Slips+Unnormalized'}})
    p.add_command(Normalize, "nm_lnd_rsk", {'InFieldName': "lnd_rsk", 
        'Metadata': {'DisplayName' : "Risk+of+Soil+Slips"}})

    # Anecdata
    p.add_command(EEMSRead, 'er_sz', {'InFileName': reporting_units_path, 
    'InFieldName': 'er_sz',
    'DataType' : "Float",
    'Metadata': {'DisplayName' : 'Erosion+Size+Raw'}})
    p.add_command(NormalizeCat, "nm_er_sz", {'InFieldName': "er_sz", 
    'RawValues': [1, 30, 100, 208, 467, 500],
    'NormalValues' : [0.143, 0.286, 0.429, 0.572, 0.715, 1],
    'DefaultNormalValue' : 0,                               
    'Metadata': {'DisplayName' : 'Norm+Erosion+Size'}})

    p.add_command(EEMSRead, 'er_veg', {'InFileName': reporting_units_path, 
    'InFieldName': 'er_veg',
    'DataType' : "Float",
    'Metadata': {'DisplayName' : 'Poorly+Vegetation'}})
    p.add_command(NormalizeCat, "nm_er_veg", {'InFieldName': "er_veg", 
    'RawValues': [2, 3, 1, 0],
    'NormalValues' : [0.66, 1, 0.33, 0],
    'DefaultNormalValue' : 0,                               
    'Metadata': {'DisplayName' : 'Norm+Poorly+Vegetation'}})

    p.add_command(EEMSRead, 'er_acc', {'InFileName': reporting_units_path, 
    'InFieldName': 'er_acc',
    'DataType' : "Float",
    'Metadata': {'DisplayName' : 'Trail+Access'}})
    p.add_command(NormalizeCat, "nm_er_acc", {'InFieldName': "er_acc", 
    'RawValues': [1, 5, 2, 0.5, 999],
    'NormalValues' : [1, 0.5, 0.5, 1, 0],
    'DefaultNormalValue' : 0,                               
    'Metadata': {'DisplayName' : 'Norm+Trail+Access'}})

    p.add_command(EEMSRead, 'er_slo', {'InFileName': reporting_units_path, 
    'InFieldName': 'er_slo',
    'DataType' : "Float",
    'Metadata': {'DisplayName' : 'Erosion+Slope'}})
    p.add_command(NormalizeCat, "nm_er_slo", {'InFieldName': "er_slo", 
    'RawValues': [2,1,0],
    'NormalValues' : [1, 0.5, 0],
    'DefaultNormalValue' : 0,                               
    'Metadata': {'DisplayName' : 'Norm+Erosion+Slope'}})

    p.add_command(EEMSRead, 'er_sli', {'InFileName': reporting_units_path, 
    'InFieldName': 'er_sli',
    'DataType' : "Float",
    'Metadata': {'DisplayName' : 'Soil+Slippage'}})   

    # Geomorph branch
    p.add_command(WeightedSum, 'erosion', {'InFieldNames': ['nm_er_sz','nm_er_slo', 'nm_er_veg', 'nm_er_acc', 'er_sli'], 
    'Weights' : [0.2, 0.2, 0.2, 0.2, 0.2],
    'Metadata': {'DisplayName' : 'Observed+Erosion+Unnormalized'}})
    p.add_command(Normalize, "nm_erosion", {'InFieldName': "erosion", 
        'Metadata': {'DisplayName' : "Observed+Erosion"}})
    # p.add_command(Sum, 'geomorp', {'InFieldNames': ['nm_erosion', 'nm_lnd_rsk'], 
    #     'Metadata': {'DisplayName' : 'Erosion+Management+Priority+Unnormalized'}})
    p.add_command(WeightedSum, 'geomorp', {'InFieldNames': ['nm_erosion', 'nm_lnd_rsk'], 
        'Weights' : [1, 1],
        'Metadata': {'DisplayName' : 'Erosion+Management+Priority+Unnormalized'}})
    p.add_command(NormalizeCurve, 'nm_geomorp', {'InFieldName': 'geomorp',
        'RawValues': [0, 1, 2],
        'NormalValues' : [0, 1, 1],
        'Metadata': {'DisplayName' : 'Erosion+Management+Priority'}})

[nikmolnar]

I found the issue. In Sum, MPilot uses numpy.copy, which by default returns a normal numpy array, even though the input is a masked array. All arrays elsewhere are assumed to be masked, so NormalizeCurve uses arr.data, which for a masked array is a regular numpy array, but for a regular numpy array, .data is a memory view.

I'll have a fix in shortly.

[nikmolnar]

This should be fixed now in version 1.2.0.

[zcanter]

Thanks, Nik!!