'homography' subprocess.CalledProcessError

[kanishk-aidash]

All the calls to hoomgraphy command / process are failing with <Signals.SIGABRT: 6>

I am running the s2p python api with two stereo images, but the run crashes with following error:

subprocess.CalledProcessError: Command '['homography', 'img_01.tif', '-h', '0.20885338713571683 -0.9784332562053635 1292.6446381480373 0.9784332562053635 0.20885338713571683 -118.72866433410803 0.0 0.0 1.0', '/private/var/folders/3t/x_8fjlmd3hsddthfkhscjctc0000gn/T/pytest-of-kanishkvarshney/pytest-2/test_rectify_pair_with_matches0/out1.tiff', '2277', '237']' died with <Signals.SIGABRT: 6>.

Full Log:

tile size: 303 297
total number of tiles: 180 (15 x 12)

discarding masked tiles...
done 180 / 180 tiles
Elapsed time: 0:00:00.605927

correcting pointing locally...
done 180 / 180 tiles
Elapsed time: 0:00:24.147351

correcting pointing globally...
Elapsed time: 0:00:00.087892

rectifying tiles...
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/kanishkvarshney/opt/anaconda3/envs/dsm/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/Users/kanishkvarshney/opt/anaconda3/envs/dsm/lib/python3.6/site-packages/s2p/parallel.py", line 43, in tilewise_wrapper
    out = fun(*args)
  File "/Users/kanishkvarshney/opt/anaconda3/envs/dsm/lib/python3.6/site-packages/s2p/__init__.py", line 153, in rectification_pair
    vmargin=cfg['vertical_margin'])
  File "/Users/kanishkvarshney/opt/anaconda3/envs/dsm/lib/python3.6/site-packages/s2p/rectification.py", line 379, in rectify_pair
    common.image_apply_homography(out1, im1, H1, w0 + 2*hmargin, h0 + 2*vmargin)
  File "/Users/kanishkvarshney/opt/anaconda3/envs/dsm/lib/python3.6/site-packages/s2p/common.py", line 228, in image_apply_homography
    run(["homography", im, "-h", hij, out, "%d" % w, "%d" % h])
  File "/Users/kanishkvarshney/opt/anaconda3/envs/dsm/lib/python3.6/site-packages/s2p/common.py", line 93, in run
    env=env, timeout=timeout, check=True)
  File "/Users/kanishkvarshney/opt/anaconda3/envs/dsm/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['homography', '/Users/kanishkvarshney/Desktop/Workspace/Code/dsm/experiments/input/img_1.TIF', '-h', '0.24907806744353794 -0.9867850768446357 313.07516782285677 0.9867850768446357 0.24907806744353794 5.0 0.0 0.0 1.0', '/Users/kanishkvarshney/Desktop/Workspace/Code/dsm/experiments/output/tiles/row_0000000_height_297/col_0000000_width_303/pair_1/rectified_ref.tif', '408', '382']' died with <Signals.SIGABRT: 6>.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kanishkvarshney/Desktop/Workspace/Code/dsm/dsm/main.py", line 10, in <module>
    main()
  File "/Users/kanishkvarshney/Desktop/Workspace/Code/dsm/dsm/main.py", line 6, in main
    s2p.main(cfg)
  File "/Users/kanishkvarshney/opt/anaconda3/envs/dsm/lib/python3.6/site-packages/s2p/__init__.py", line 561, in main
    timeout=timeout)
  File "/Users/kanishkvarshney/opt/anaconda3/envs/dsm/lib/python3.6/site-packages/s2p/parallel.py", line 102, in launch_calls
    outputs.append(r.get(timeout))
  File "/Users/kanishkvarshney/opt/anaconda3/envs/dsm/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
subprocess.CalledProcessError: Command '['homography', '/Users/kanishkvarshney/Desktop/Workspace/Code/dsm/experiments/input/img_1.TIF', '-h', '0.24907806744353794 -0.9867850768446357 313.07516782285677 0.9867850768446357 0.24907806744353794 5.0 0.0 0.0 1.0', '/Users/kanishkvarshney/Desktop/Workspace/Code/dsm/experiments/output/tiles/row_0000000_height_297/col_0000000_width_303/pair_1/rectified_ref.tif', '408', '382']' died with <Signals.SIGABRT: 6>.

What am I missing?

[mnhrdt]

To debug this problem, can you please show the file /Users/kanishkvarshney/Desktop/Workspace/Code/dsm/experiments/input/img_1.TIF ?

[kanishk-aidash]

@mnhrdt Hey Eric,

I can't share the file with you as it is client data. If you need any other relevant information, it I can share that specifically.

It is satellite geotiff, following is the ss from QGIS

From the code, it seemed like it's trying to execute homography command, but my system / terminal doesn't have any such command .. and I don't see any thing in the the installation steps that might have led to this.

[mnhrdt]

The program "homography" should be inside the "s2p/bin" folder. Can you please check that it is there? If you run ./bin/homography inside the s2p directory it should print a short usage message. I guess you have it there since it aborts upon execution (if it didn't exist python would print a different error). We want to understand why this program aborts. It may be due to an unrecognized TIFF image format, but this is unlikely since the other programs recognize it without problem.

[kanishk-aidash]

I ran the homobraphy command ..from s2p/bin .. this is what's happenign

[carlodef]

@kanishk-aidash It seems that your system is missing the GDAL library. You need to install it, as described here: https://github.com/cmla/s2p#on-macos

[kanishk-aidash]

Hey @carlodef I have GDAL installed and working.

I am suspecting that multiple conda environments might be mixing-up some paths in the system and was planning to cleanup. I will update on the thread once I have updated the setup

[kanishk-aidash]

Hey,

So I did the cleanup (shifted to a fresh EC2) .. seems like GDAL accessibility and PyProj version was the issue was the issue.

Now, it worked till this point:

[mnhrdt]

What is dsm_gen.py ?

[kanishk-aidash]

^ ah my bad .. it's simply calling the s2p.main(...) Following is the content of the file:

import s2p

def main():
    cfg = s2p.read_config_file("config.json")
    s2p.main(cfg)

if __name__ == '__main__':
    main()

And my config.json looks like following:

{
  "out_dir" : "output/",
  "images" : [

    {
    "img" : "input/img_1.TIF",
    "rpc": "input/img_1.xml"

  },
    {
    "img" : "input/img_2.TIF",
    "rpc": "input/img_2.xml"

  }
    ],
  "roi" : {
      "x" : 150,
      "y" : 150,
      "w" : 700,
      "h" : 700
  },
  "full_img": true,
  "horizontal_margin": 20,
  "vertical_margin": 5,
  "tile_size" : 600,
  "disp_range_method" : "sift",
  "msk_erosion": 0,
  "dsm_resolution": 0.5
}

[mnhrdt]

This seems an entirely different issue to the title of this one.

It's a bit difficult to debug it that way (for instance, your file dsm_gen.py is not exactly the same as the one you are using, since the line numbers and function names are inconsistent). To make sure that s2p works at all for you, can you run the provided test suite when you clone a fresh repository, according to the instructions of the readme? Then we can work from that.

[kanishk-aidash]

Hey Enric, Sure .. I have updated the exact dsm_gen.py .. I was just running as a pyCharm scratches .. but eventually calling only those two functions ( s2p.read_config_file(...), and s2p.main(...)

Here is the log for test suite: test.log

Do both geotiffs have to be of same dimension? I am trying to create dsm for full rasters, which are different in size

Yes, current issue is different, but I am not sure if it's continuation of same or some other cause. As mentioned by Carlo, the first error got by updating the GDAL and PyProj (switched to new machine), but this led to new issue in the flow

[mnhrdt]

The output of the text suite looks alright and s2p indeed works correctly on your setup.

Do both geotiffs have to be of same dimension? I am trying to create dsm for full rasters, which are different in size

The input images can have different size; of course their coverage should overlap if you want to obtain a dsm otput. Do you still get the error when launching your working copy of s2p with that config file?

[kanishk-aidash]

Yes, I am still getting the ArgumentError: array must have shape (...) The input rasters are the stereo pairs and do overlap. They work fine in PCI tool but fails at triangulating tiles step with the s2p module

[mnhrdt]

I don't see what could be wrong. Can you add the option "debug":true to the config file, re-run the experiment, and upload the output logs? Or provide a minimal example that we can reproduce locally.

[kanishk-aidash]

Hey @mnhrdt

Providing data might not be possible .. and am not sure what exactly is causing this hence am unable to create some dummy data to reproduce the issue. Here's the log for the run with and without debug flag ( I am getting different end error with 'debug': true and without debug flag) run_debug.log run.log

[gfacciol]

This line from the log is a bit preoccupying: WARNING: sift.matches_on_rpc_roi: found no matches.

without giving away the while images you could send us the content of the following folder (with its subfolders)

/home/ubuntu/kanishk/code/../data/output/tiles/row_0001779_height_593/col_0003414_width_569/pair_1/

this will allow us to check what is the issue.

On Fri, Apr 9, 2021 at 9:44 AM kanishk-aidash @.***> wrote:

Hey @mnhrdt https://github.com/mnhrdt

Providing data might not be possible .. and am not sure what exactly is causing this hence am unable to create some dummy data to reproduce the issue. Here's the log for the run with and without debug flag ( I am getting different end error with 'debug': true and without debug flag) run_debug.log https://github.com/cmla/s2p/files/6283982/run_debug.log run.log https://github.com/cmla/s2p/files/6283983/run.log

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cmla/s2p/issues/84#issuecomment-816486325, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGFI2QZSDTD7ZEM6LTAUGDTH2V5DANCNFSM42NC2DGQ .

[kanishk-aidash]

Hey Gabriele,

Please find the requested files. Do let me know if any more data is needed

pair_1.zip

[mnhrdt]

This error apparently happens on an "empty" tile without image data. If that is the case, s2p should definitely give a clearer error message and not stop the pipeline.

To test whether this is the case, can you please remove the full_img:true option in the config file and specify a roi that falls completely within the part of the domain with image data? It should work correctly in that case.

[kanishk-aidash]

Hey @mnhrdt

I am noticing similar behaviour even with a small tile. Here's the new config for the run:

{
  "out_dir" : "../data/output/",
  "images" : [
    {
    "img" : "../data/img_1.TIF",
    "rpc": "../data/img_1.xml"
  },
    {
    "img" : "../data/img_2.TIF",
    "rpc": "../data/img_2.xml"
  }
    ],
  "roi" : {
      "x" : 500,
      "y" : 500,
      "w" : 500,
      "h" : 500
  },
  "debug": true,
  "horizontal_margin": 20,
  "vertical_margin": 5,
  "tile_size" : 600,
  "disp_range_method" : "sift",
  "msk_erosion": 0,
  "dsm_resolution": 0.5
}

The corresponding tile has data, but error remains similar. Attaching the debug logs and output.zip for reference:

run_500x500.log

output_small.zip

[carlodef]

This is a bug in s2p. It occurs when input images have multiple bands. Most users run s2p on single-band panchromatic images, hence the bug didn't show up yet. It should be fixed by commit ab687293.

@kanishk-aidash could you please retry and let us know if that fixes your issue?

[kanishk-aidash]

Hey @carlodef It doesn't work even with the latest build .. now it throws some other error.

Attaching the new logs for your reference:

run_500x500_news2p_debug.log run_full_news2p_debug.log

[carlodef]

Hi @kanishk-aidash, this is a different error. It is due to your input images having 4 color bands (probably RGBI?), while s2p supported only 1 or 3 color bands. This should be fixed by commit f7540c07.

Could you please retry with this new version, and for the full run also set debug to False as your logs show another error related to some debug information failing to be saved?

[kanishk-aidash]

Hey @carlodef
Thanks for the help, I was able to run with both 1 (on R-Band)and 4 bands with the latest build. The run went fine but the results don't look that encouraging ...

You can mark this thread closed if needed

[carlodef]

Hi @kanishk-aidash, as the GSD of your input images seems to be about 2 meters, you should set the input parameter "dsm_resolution" to something larger than the default value of 0.5m. I would try with "dsm_resolution": 2.5

[kanishk-aidash]

Hey @carlodef I did try at multiple dsm_resolutions, the output varies but still not fully generated DSM: At 0.5:

At 1.5:

At 2.5:

At 3.0:

Original raster 1:

Original raster 2:

It still isn't able to generate DSM for the whole raster:

1. Missing the patch near the centre of the raster
2. No Image area is also generating some disparity

[mnhrdt]

It looks good to me. The "black" part of the DSM is the floodplain (you can tell it's flat because there are a lot of meanders). The missing data points here and there are natural, and you will always have them due to inevitable occlusions in the input pair. By default there's no interpolation of these holes in s2p. Regarding the heights generated outside of the intended domain, you can ignore them, they are just artifacts of the filtering, but I agree it should be better to remove them automatically. I don't think we have seen many such images with hand-drawn masks before.

[kanishk-aidash]

@mnhrdt @carlodef Thanks for the help. It seemingly works fine now. I will close this issue.