CARS supports PleiadesNeo stereo imagery?

lionlai1989 commented 2 years ago

Hi team of CARS: I've been testing CARS with PNeo (PleiadesNeo) data lately and found that there exists few issues.

Program breaks with the error IndexError: index -1 is out of bounds for axis 0 with size 0:

22-10-13 10:33:39 :: WARNING :: Could not import pygraphviz backend. Will try graphviz backend next
WARNING ossimRpcModel::lineSampleHeightToWorld: 
NaN detected computing RPC polynomials. Results are invalid.
WARNING ossimRpcModel::lineSampleHeightToWorld: 
NaN detected computing RPC polynomials. Results are invalid.
WARNING ossimRpcModel::lineSampleHeightToWorld: 
NaN detected computing RPC polynomials. Results are invalid.
WARNING ossimRpcModel::lineSampleHeightToWorld: 
NaN detected computing RPC polynomials. Results are invalid.
Computing epipolar grids ...: 100% [**************************************************]
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:218: RuntimeWarning: invalid value encountered in cos
cos_lat0 = np.cos(np.radians(lat0))
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:219: RuntimeWarning: invalid value encountered in sin
sin_lat0 = np.sin(np.radians(lat0))
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:221: RuntimeWarning: invalid value encountered in cos
cos_long0 = np.cos(np.radians(lon0))
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:222: RuntimeWarning: invalid value encountered in sin
sin_long0 = np.sin(np.radians(lon0))
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:227: RuntimeWarning: invalid value encountered in double_scalars
x_east = (-(x_ecef - x0_ecef) * sin_long0) + (
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:228: RuntimeWarning: invalid value encountered in double_scalars
(y_ecef - y0_ecef) * cos_long0
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:231: RuntimeWarning: invalid value encountered in double_scalars
(-cos_long0 * sin_lat0 * (x_ecef - x0_ecef))
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:232: RuntimeWarning: invalid value encountered in double_scalars
- (sin_lat0 * sin_long0 * (y_ecef - y0_ecef))
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:233: RuntimeWarning: invalid value encountered in double_scalars
+ (cos_lat0 * (z_ecef - z0_ecef))
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:236: RuntimeWarning: invalid value encountered in double_scalars
(cos_lat0 * cos_long0 * (x_ecef - x0_ecef))
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:237: RuntimeWarning: invalid value encountered in double_scalars
+ (cos_lat0 * sin_long0 * (y_ecef - y0_ecef))
/cars/venv/lib/python3.8/site-packages/cars/core/projection.py:238: RuntimeWarning: invalid value encountered in double_scalars
+ (sin_lat0 * (z_ecef - z0_ecef))
Processing Tiles : [  ] ...: 0it [00:00, ?it/s]
22-10-13 10:33:40 :: ERROR :: CARS terminated with following error
Traceback (most recent call last):
File "/cars/venv/lib/python3.8/site-packages/cars/cars.py", line 172, in main_cli
used_pipeline.run()
File "/cars/venv/lib/python3.8/site-packages/cars/pipelines/sensor_to_full_resolution_dsm/sensor_to_full_resolution_dsm_pipeline.py", line 296, in run
) = self.resampling_application.run(
File "/cars/venv/lib/python3.8/site-packages/cars/applications/resampling/bicubic_resampling.py", line 363, in run
format_transformation.grid_margins_2_overlaps(
File "/cars/venv/lib/python3.8/site-packages/cars/data_structures/format_transformation.py", line 118, in grid_margins_2_overlaps
row_max = grid[-1, 0, 1]
IndexError: index -1 is out of bounds for axis 0 with size 0

My guts feeling tells me that the warnings and errors here are related to the goemodel file (RPC*.XML). Thus, I compare the PNeo's RPC file with Pleiades RPC file and find that tags of PNeo and Pleiades are not exactly the same. Here is the mapping:

GroundtoImage_Values --- Inverse_Model
ImagetoGround_Values --- Direct_Model
LON_DEN_COEFF_ --- SAMP_DEN_COEFF_
LON_NUM_COEFF_ --- SAMP_NUM_COEFF_
LAT_DEN_COEFF_ --- LINE_DEN_COEFF_
LAT_NUM_COEFF_ --- LINE_NUM_COEFF_

So, I manually modified the RPC file and make sure that tags of PNeo looks exactly the same as Pleiades'. And rerun the pipeline.

CARS is running eternally, and no changes in output folder.
In 1., I described what modification is made and rerun the pipeline. Here is the output:


22-10-11 13:15:31 :: WARNING :: Could not import pygraphviz backend. Will try graphviz backend next
Computing epipolar grids ...: 100% [**************************************************] (4s)

Processing Tiles : [ epi_matches_left ] ...: 0%| | 0/1612 [00:00<?, ?it/s]ERROR 1: /data/2022-07-15/1558/2022-07-11/WO_000028295_1_11_SAL22023513-11_ACQ_PNEO4_01714303116360/000028295_1_11_STD_A/IMG_11_PNEO4_PAN/IMG_PNEO4_202207110043138_PAN_SEN_PWOI_000028295_1_11_F_1_P_R1C1.JP2:Not a TIFF or MDI file, bad magic number 0 (0x0) Processing Tiles : [ epi_matches_left ] ...: 100%|██████████| 1612/1612 [2:05:42<00:00, 4.68s/it] ERROR 1: /data/2022-07-15/1558/2022-07-11/WO_000028295_1_11_SAL22023513-11_ACQ_PNEO4_01714303116360/000028295_1_11_STD_A/IMG_11_PNEO4_PAN/IMG_PNEO4_202207110043138_PAN_SEN_PWOI_000028295_1_11_F_1_P_R1C1.JP2:Not a TIFF or MDI file, bad magic number 0 (0x0) ... WARNING:transitions.extensions.diagrams:Could not import pygraphviz backend. Will try graphviz backend next WARNING:transitions.extensions.diagrams:Could not import pygraphviz backend. Will try graphviz backend next WARNING:transitions.extensions.diagrams:Could not import pygraphviz backend. Will try graphviz backend next WARNING:transitions.extensions.diagrams:Could not import pygraphviz backend. Will try graphviz backend next

- At this point, a small AOI (observe 1612 tiles in the output above) has been running in a 16 CPUs machine for more than 24 hours, and I used `htop` to observe that the program was using only 1 CPU (but 8 CPUs are assigned in config.json).  
- Also at the same time, the size of the output folder doesn't change at all.
- Thus, I decide to terminate the program and here is the output of `22-10-11_13h15m_sensor_to_full_res_dsm.log`:

cat 22-10-11_13h15m_sensor_to_full_res_dsm.log 22-10-12 09:42:01 :: ERROR :: CARS terminated with following error Traceback (most recent call last): File "/cars/venv/lib/python3.8/site-packages/cars/cars.py", line 172, in main_cli used_pipeline.run() File "/cars/venv/lib/python3.8/site-packages/cars/pipelines/sensor_to_full_resolution_dsm/sensor_to_full_resolution_dsmpipeline.py", line 558, in run = self.rasterization_application.run( File "/cars/venv/lib/python3.8/site-packages/cars/orchestrator/orchestrator.py", line 314, in exit self.compute_futures() File "/cars/venv/lib/python3.8/site-packages/cars/orchestrator/orchestrator.py", line 253, in compute_futures for future_obj in self.cluster.future_iterator(future_objects): File "/cars/venv/lib/python3.8/site-packages/cars/orchestrator/cluster/multiprocessing_cluster.py", line 686, in next if item.ready(): KeyboardInterrupt


3. An unusual thing about `content.json`. The resolution of PNeo data is **0.32** meter. In PNeo metadata file `DIM*.XML`,

        <Sampling_Settings>
            <RESAMPLING_SPACING unit="m">0.32</RESAMPLING_SPACING>
            <RESAMPLING_KERNEL>SPLINE</RESAMPLING_KERNEL>
        </Sampling_Settings>

But the CARS's output file `content.json` shows:

"cloud_fusion_params": {
  "method": "mapping_to_terrain_tiles",
  "terrain_tile_size": null,
  "resolution": 0.5
},
"cloud_fusion_run": {
  "epsg": 32755,
  "margins": {
    "radius": 1.0,
    "resolution": 0.5
  },
"rasterization_params": {
  "method": "simple_gaussian",
  "dsm_radius": 1.0,
  "sigma": null,
  "grid_points_division_factor": null,
  "resolution": 0.5
},

It looks like the target resolution should be entered by users. How can I do it?

4. The information of the data:  
- PNeo directory layout:

│ ├── IMG_10_PNEO4_MS-FS │ │ ├── DIM_PNEO4_202207110043428_MS-FS_SEN_PWOI_000028295_1_10_F_1.XML │ │ ├── IMG_PNEO4_202207110043428_MS-FS_SEN_PWOI_000028295_1_10_F_1_NED_R1C1.J2W │ │ ├── IMG_PNEO4_202207110043428_MS-FS_SEN_PWOI_000028295_1_10_F_1_NED_R1C1.JP2 │ │ ├── IMG_PNEO4_202207110043428_MS-FS_SEN_PWOI_000028295_1_10_F_1_RGB_R1C1.J2W │ │ ├── IMG_PNEO4_202207110043428_MS-FS_SEN_PWOI_000028295_1_10_F_1_RGB_R1C1.JP2 │ │ ├── INDEX.HTM │ │ ├── LIBRARY │ │ ├── LINEAGE │ │ ├── LUT_PNEO4_202207110043428_MS-FS_SEN_PWOI_000028295_1_10_F_1_NED.XML │ │ ├── LUT_PNEO4_202207110043428_MS-FS_SEN_PWOI_000028295_1_10_F_1_RGB.XML │ │ ├── MASKS │ │ ├── PREVIEW_PNEO4_202207110043428_MS-FS_SEN_PWOI_000028295_1_10_F_1.JPG │ │ ├── PREVIEW_PNEO4_202207110043428_MS-FS_SEN_PWOI_000028295_1_10_F_1.KMZ │ │ └── RPC_PNEO4_202207110043428_MS-FS_SEN_PWOI_000028295_1_10_F_1.XML │ ├── IMG_10_PNEO4_PAN │ │ ├── DIM_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1.XML │ │ ├── IMG_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1_P_R1C1.J2W │ │ ├── IMG_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1_P_R1C1.JP2 │ │ ├── INDEX.HTM │ │ ├── LIBRARY │ │ ├── LINEAGE │ │ ├── LUT_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1_P.XML │ │ ├── MASKS │ │ ├── PREVIEW_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1.JPG │ │ ├── PREVIEW_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1.KMZ │ │ └── RPC_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1_P.XML │ ├── INDEX.HTM │ ├── LIBRARY │ │ ├── VOL_LOGO.JPG │ │ └── VOL_STYLE.XSL │ └── VOL_PNEO.XML

- `config.json`:

{ "inputs": { "sensors": { "one": { "image": "/data/WO_000028295_1_11_SAL22023513-11_ACQ_PNEO4_01714303116360/000028295_1_11_STD_A/IMG_11_PNEO4_PAN/IMG_PNEO4_202207110043138_PAN_SEN_PWOI_000028295_1_11_F_1_P_R1C1.JP2", "geomodel": "/data/WO_000028295_1_11_SAL22023513-11_ACQ_PNEO4_01714303116360/000028295_1_11_STD_A/IMG_11_PNEO4_PAN/RPC_PNEO4_202207110043138_PAN_SEN_PWOI_000028295_1_11_F_1_P.XML", "no_data": 0 }, "two": { "image": "/data/WO_000028295_1_10_SAL22023513-10_ACQ_PNEO4_01714303116361/000028295_1_10_STD_A/IMG_10_PNEO4_PAN/IMG_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1_P_R1C1.JP2", "geomodel": "/data/WO_000028295_1_10_SAL22023513-10_ACQ_PNEO4_01714303116361/000028295_1_10_STD_A/IMG_10_PNEO4_PAN/RPC_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1_P.XML", "no_data": 0 } }, "pairing": [ [ "one", "two" ] ], "initial_elevation": "/data/2022-07-15_srtm" }, "output": { "out_dir": "/output_dir" }, "orchestrator": { "mode": "mp", "nb_workers": 8 } }



It's a bit long. Thank you for your patience. :smile:

lionlai1989 commented 2 years ago

Another note: This feature request in OTB gitlab page shows that OTB currently doesn't support PleiadesNeo data. Could it be the issue? However, I still think that the difference between Pleiades and PNeo data is the metadata. If the metadata is modified to be exactly the same, CARS and OTB should be able to run it.

dyoussef commented 2 years ago

Hello @lionlai1989,

You are right : OTB currently doesn't not support Pleiades Neo Data so CARS too.

In order to work around the problem, you must :

Modify these keys in each RPC :

sed -i 's/GroundtoImage_Values/Inverse_Model/g' $RPC
sed -i 's/ImagetoGround_Values/Direct_Model/g' $RPC
sed -i 's/GroundtoImage/Inverse_Model/g' $RPC
sed -i 's/ImagetoGround/Direct_Model/g' $RPC

Point to DIMAP to help OTB to make the link between images and RPC :

{
"inputs": {
    "sensors": {
        "one": {
            "image": "/data/WO_000028295_1_11_SAL22023513-11_ACQ_PNEO4_01714303116360/000028295_1_11_STD_A/IMG_11_PNEO4_PAN/DIM_PNEO4_202207110043138_PAN_SEN_PWOI_000028295_1_11_F_1.XML",
            "geomodel": "/data/WO_000028295_1_11_SAL22023513-11_ACQ_PNEO4_01714303116360/000028295_1_11_STD_A/IMG_11_PNEO4_PAN/RPC_PNEO4_202207110043138_PAN_SEN_PWOI_000028295_1_11_F_1_P.XML",
            "no_data": 0
        },
        "two": {
            "image": "/data/WO_000028295_1_10_SAL22023513-10_ACQ_PNEO4_01714303116361/000028295_1_10_STD_A/IMG_10_PNEO4_PAN/DIM_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1.XML",
            "geomodel": "/data/WO_000028295_1_10_SAL22023513-10_ACQ_PNEO4_01714303116361/000028295_1_10_STD_A/IMG_10_PNEO4_PAN/RPC_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1_P.XML",
            "no_data": 0
        }
    },
    "pairing": [
        [
            "one",
            "two"
        ]
    ],
    "initial_elevation": "/data/2022-07-15_srtm"
},
"output": {
    "out_dir": "/output_dir"
},
"orchestrator": {
    "mode": "mp",
    "nb_workers": 8
}
}

NB : To check it works once you have modified the RPC, you can try on a small data using otbcli_ExtractROI : https://cars.readthedocs.io/en/stable/faq.html#how-to-create-cars-compatible-roi-input-data-with-otb :

otbcli_ExtractROI -in /data/WO_000028295_1_11_SAL22023513-11_ACQ_PNEO4_01714303116360/000028295_1_11_STD_A/IMG_11_PNEO4_PAN/DIM_PNEO4_202207110043138_PAN_SEN_PWOI_000028295_1_11_F_1.XML -out img1_xt.tif uint16 -startx startx -starty starty -sizex sizex -sizey sizey

otbcli_ExtractROI -in /data/WO_000028295_1_10_SAL22023513-10_ACQ_PNEO4_01714303116361/000028295_1_10_STD_A/IMG_10_PNEO4_PAN/DIM_PNEO4_202207110043428_PAN_SEN_PWOI_000028295_1_10_F_1.XML -out img2_xt.tif uint16 -mode fit -mode.fit.im img1_xt.tif

Regards, David

dyoussef commented 1 year ago

Hello @lionlai1989, have you tried with the recommendations ? Can we close the issue ? Best regard, David

dyoussef commented 1 year ago

I close this issue : @lionlai1989 If you require any further information, please feel free to (re) open it.

CNES / cars

CARS supports PleiadesNeo stereo imagery? #13