Oops! ifgram_inversion dask does not work with watermask!

[falkamelung]

Unfortunatly, dask now throws errors. If I run without dask the example that fails works fine. Unless you have a quick fix I suggest to revert the autopath - waterMask pull request.

I don't understand yet for what the watermask is useful. I just wanted to have it in to be able to try things.

Sorry for the confusion!

Full script that generated the error

PASTE CODE HERE

Full error message

PASTE ERROR MESSAGE HERE

System information

• Operating system:
• Version of Python and relevant dependencies if applicable
• Version of MintPy (output of smallbaselineApp.py -v):
• Your custom / default template file (smallbaselineApp.cfg):

[yunjunz]

@falkamelung please provide more detailed info here.

[falkamelung]

I am getting the error below. It does not say much why it fails. The waterMask.h5 file looks OK. When I run without dask it works fine (though I did not look at results)

However, I tested a different data set (¼ the size). In that case invert_network with dask works fine. Any idea I can try?

//c485-134/scratch/05861/tg851601/testKilaueaCskAT10[1017] info.py mintpy/inputs/geometryRadar.h5 --slice
height
latitude
longitude
incidenceAngle
azimuthAngle
slantRangeDistance
shadowMask
waterMask

umber of pixels to invert: 13408 out of 76000 (17.6%)
inverting network of interferograms into time-series ...
number of pixels to invert: 0 out of 76000 (0.0%)
number of pixels to invert: 0 out of 76000 (0.0%)
skip pixels with zero/nan value in all interferograms
FUTURE #1 complete. Time used: 10 seconds
Traceback (most recent call last):
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/MintPy/mintpy/smallbaselineApp.py", line 1136, in <module>
    main()
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/MintPy/mintpy/smallbaselineApp.py", line 1126, in main
    app.run(steps=inps.runSteps, plot=inps.plot)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/MintPy/mintpy/smallbaselineApp.py", line 1070, in run
    self.run_network_inversion(sname)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/MintPy/mintpy/smallbaselineApp.py", line 594, in run_network_inversion
    mintpy.ifgram_inversion.main(scp_args.split())
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/MintPy/mintpy/ifgram_inversion.py", line 1132, in main
    ifgram_inversion(inps)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/MintPy/mintpy/ifgram_inversion.py", line 1068, in ifgram_inversion
    results=[ts, temp_coh, num_inv_ifg])
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/MintPy/mintpy/objects/cluster.py", line 147, in run
number of pixels to invert: 0 out of 76000 (0.0%)
    return self.collect_result(futures, results, box, submission_time)
  File "/home1/05861/tg851601/test/operations/rsmas_insar/sources/MintPy/mintpy/objects/cluster.py", line 205, in collect_result
    x0, y0, x1, y1 = sub_box
ValueError: too many values to unpack (expected 4)
distributed.scheduler - ERROR - Couldn't gather keys {'ifgram_inversion_patch-ec99f84b3732aedba6c46db4ecbcd548': ['tcp://127.0.0.1:44109']} state: ['erred'] workers: ['tcp://127.0.0.1:44109']
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: ['tcp://127.0.0.1:44109'], ifgram_inversion_patch-ec99f84b3732aedba6c46db4ecbcd548
NoneType: None
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'ifgram_inversion_patch-ec99f84b3732aedba6c46db4ecbcd548': ('tcp://127.0.0.1:44109',)}

[yunjunz]

I can not reproduce this error and will close this issue in a few days since the initiator did not provide more insightful details or result checking.

P.S.: if there is an error like this type, it should have been identified during the development, which includes testing. The default branch here is not for testing purposes.

[falkamelung]

That is strange. Did you actually try? Always when I have the wasterMask files in the directory (CSK) it fails. After removing them it works. I can try once again once again and then just remove the auto loading if that is OK with you. For now I don't see much benefits anyway

[yunjunz]

Hi @falkamelung I having been water mask and dask local cluster regularly, no issue for me. Since you have not filled out enough info as the template described, I can not diagnose. For bug report, or for contributing new features, please follow the contributing guideline.