search

DeDop / dedop-core

DeDop is a User Configurable Tool for Processing Delay Doppler Altimeter Data.
http://dedop.org/
GNU General Public License v3.0
14 stars 8 forks source link

Issue with Amazon data #31

Closed hans-permana closed 6 years ago

hans-permana commented 7 years ago

Expected behavior

Able to process Amazon L1A files

Actual behavior

dedop-error

Steps to reproduce the issue

  1. Download the following L1A input files:

  2. In DeDop Studio, run a process for those files with default configuration

Specifications

dedop-0.5.4.dev dedop-studio-0.0.1-dev

mark-ep commented 7 years ago

this his been fixed in commit 2845849 - the Amazon basin files contain some invalid bursts, which the processor now identifies and skips.

nbercher-atk commented 7 years ago

We overcame this issue by using our own ecef2lla() implementation. Notably, our implementation does not involve limitations on a number of iteration (MAX_ITERS). Contact me if you are interested.

nbercher-atk commented 7 years ago

Hi,

Before submitting a new issue on Github, we may discuss the issue we are facing at Aʟᴏɴɢ-Tʀᴀᴄᴋ: we are not able to process L1A files for tha Amazon Case Study unsing the default DeDop configuration:

1. Install Dedop Shell (cf. below)

2. Create & configure new workspace and run DeDop:

Create & configure workspace:

dedop workspace add aca-ddp_wp6400_amazon_case_study dedop config add ddp_conf_01

Add input L1A files:

L1A_PATH=/some/path dedop input add "${L1A_PATH}"/*.DBL.nc

Run DeDop:

dedop run

ISSUE#1: This fails with the following errors in ecef2lla() function:

processing ~/.dedop/workspaces/aca-ddp_wp6400_amazon_case_study/inputs/CS_LTA__SIR1SAR_FR_20150101T075744_20150101T075923_C001.DBL.nc using "ddp_conf_01" processing: [###---------------------------] 10%

/usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/functions/vectors.py:11: RuntimeWarning: invalid value encountered in true_divide (norm(vec1) norm(vec2)) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/functions/vectors.py:11: RuntimeWarning: invalid value encountered in true_divide (norm(vec1) norm(vec2)) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:269: RuntimeWarning: divide by zero encountered in double_scalars (ground_surf_orbit_angle - ground_surf_orbit_angle_prev) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:272: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.time_sar_ku - isp_prev.time_sar_ku) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:274: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.x_sar_surf - isp_prev.x_sar_surf) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:276: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.y_sar_surf - isp_prev.y_sar_surf) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:278: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.z_sar_surf - isp_prev.z_sar_surf) processing: done

dedop run: error: MAX_ITERS reached in ecef2lla

SOLUTION TO ISSUE#1:

Replace DeDop's ecef2lla() Pyhton function with Aʟᴏɴɢ-Tʀᴀᴄᴋ's implementation. Then rerun.

Note: Despite this solution allowed us to reach another issue (cf. ISSUE#2), it is likely that the failure in DeDop's ecef2lla() implementation is related to some unhandleded issue in the Amazon L1A files.

ISSUE#2: Rerun of DeDop with new ecef2lla() function fails with

"divide by zero" errors:

processing: [###---------------------------] 10%

/usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/functions/vectors.py:11: RuntimeWarning: invalid value encountered in true_divide (norm(vec1) norm(vec2)) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/functions/vectors.py:11: RuntimeWarning: invalid value encountered in true_divide (norm(vec1) norm(vec2)) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:269: RuntimeWarning: divide by zero encountered in double_scalars (ground_surf_orbit_angle - ground_surf_orbit_angle_prev) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:272: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.time_sar_ku - isp_prev.time_sar_ku) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:274: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.x_sar_surf - isp_prev.x_sar_surf) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:276: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.y_sar_surf - isp_prev.y_sar_surf) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:278: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.z_sar_surf - isp_prev.z_sar_surf) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:289: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.x_sar_sat - isp_prev.x_sar_sat) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:291: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.y_sar_sat - isp_prev.y_sar_sat) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:293: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.z_sar_sat - isp_prev.z_sar_sat) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:304: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.x_vel_sat_sar - isp_prev.x_vel_sat_sar) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:306: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.y_vel_sat_sar - isp_prev.y_vel_sat_sar) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:308: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.z_vel_sat_sar - isp_prev.z_vel_sat_sar) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:311: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.alt_rate_sat_sar - isp_prev.alt_rate_sat_sar) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:314: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.roll_sar - isp_prev.roll_sar) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:316: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.pitch_sar - isp_prev.pitch_sar) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/surface_locations.py:318: RuntimeWarning: invalid value encountered in double_scalars alpha (isp_curr.yaw_sar - isp_prev.yaw_sar) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/functions/vectors.py:11: RuntimeWarning: invalid value encountered in true_divide (norm(vec1) norm(vec2)) /usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/functions/vectors.py:11: RuntimeWarning: invalid value encountered in true_divide (norm(vec1) norm(vec2)) processing: [###---------------------------] 10% processing gap: 32

/usr/local/dedop/dedop-1.0.0/dedop-backend/lib/python3.5/site-packages/dedop/proc/sar/algorithms/beam_angles.py:34: RuntimeWarning: divide by zero encountered in double_scalars self.chd.freq_ku / isp_record.vel_sat_sar_norm) processing: processing gap: 32

dedop run: error: math domain error

Download DeDop Shell (aka DeDop backend):

wget https://github.com/DeDop/dedop-core/releases/download/v1.0.0/DeDop-backend-1.0.0-Linux-x86_64.sh chmod a+x DeDop-backend-1.0.0-Linux-x86_64.sh

Install DeDop:

mkdir -p /usr/local/dedop/dedop-1.0.0/ ./DeDop-backend-1.0.0-Linux-x86_64.sh

hans-permana commented 7 years ago

hi @nbercher-atk, unfortunately, the fix to this issue came after the v1.0.0 release. This should be included on the next release (v1.1.0).

hans-permana commented 7 years ago

DeDop-core v1.1.0 has been released https://github.com/DeDop/dedop-core/releases/tag/v1.1.0

hans-permana commented 6 years ago

Reopening this issue because I encountered a similar issue (the same error message, but maybe different cause).

(DeDop) $ dedop run
processing /Users/dedop/.dedop/workspaces/default/inputs/CS_LTA__SIR1SAR_FR_20150331T034023_20150331T034235_C001.DBL.nc using "new-config"
processing: [##----------------------------]   7%                                                                                                                     /Users/dedop/dedop-core-1.4.0.dev1/lib/python3.6/site-packages/dedop-1.4.0.dev1-py3.6.egg/dedop/proc/functions/vectors.py:10: RuntimeWarning: invalid value encountered in double_scalars
  (norm(vec1) * norm(vec2))
/Users/dedop/dedop-core-1.4.0.dev1/lib/python3.6/site-packages/dedop-1.4.0.dev1-py3.6.egg/dedop/proc/functions/vectors.py:10: RuntimeWarning: invalid value encountered in double_scalars
  (norm(vec1) * norm(vec2))
/Users/dedop/dedop-core-1.4.0.dev1/lib/python3.6/site-packages/dedop-1.4.0.dev1-py3.6.egg/dedop/proc/sar/algorithms/surface_locations.py:269: RuntimeWarning: divide by zero encountered in double_scalars
  (ground_surf_orbit_angle - ground_surf_orbit_angle_prev)
/Users/dedop/dedop-core-1.4.0.dev1/lib/python3.6/site-packages/dedop-1.4.0.dev1-py3.6.egg/dedop/proc/sar/algorithms/surface_locations.py:272: RuntimeWarning: invalid value encountered in double_scalars
  alpha * (isp_curr.time_sar_ku - isp_prev.time_sar_ku)
/Users/dedop/dedop-core-1.4.0.dev1/lib/python3.6/site-packages/dedop-1.4.0.dev1-py3.6.egg/dedop/proc/sar/algorithms/surface_locations.py:274: RuntimeWarning: invalid value encountered in double_scalars
  alpha * (isp_curr.x_sar_surf - isp_prev.x_sar_surf)
/Users/dedop/dedop-core-1.4.0.dev1/lib/python3.6/site-packages/dedop-1.4.0.dev1-py3.6.egg/dedop/proc/sar/algorithms/surface_locations.py:276: RuntimeWarning: invalid value encountered in double_scalars
  alpha * (isp_curr.y_sar_surf - isp_prev.y_sar_surf)
/Users/dedop/dedop-core-1.4.0.dev1/lib/python3.6/site-packages/dedop-1.4.0.dev1-py3.6.egg/dedop/proc/sar/algorithms/surface_locations.py:278: RuntimeWarning: invalid value encountered in double_scalars
  alpha * (isp_curr.z_sar_surf - isp_prev.z_sar_surf)
processing: done                                                                                                                                                      
dedop run: error: MAX_ITERS reached in ecef2lla

Run with DeDop-core-1.4.0.dev1 in MacOSX 10.13.3

hans-permana commented 6 years ago

It failed when processing this dataset (Amazon): http://dedop.org/data/resources/Amazon/CS_LTA__SIR1SAR_FR_20150331T034023_20150331T034235_C001.DBL.nc

but successfully processed this dataset (Icebergs): http://dedop.org/data/resources/Icebergs/CS_LTA__SIR1SAR_FR_20130303T030418_20130303T030503_C001.DBL.nc

Are there any differences between those two type of data? I thought the only difference is just the lat and lon values.

Run with DeDop-core-1.4.0.dev1 in KDE

mark-ep commented 6 years ago

some time ago, @nbercher-atk provided us with a patch for a non-iterative version of the ecef2lla algorithm, which should solve this issue. I'm going to apply the patch and run some tests on it, as well as checking the specific files you've found this problem with. If everything's good I'll replace the current version of the ecef2lla algorithm with the non-iterative version.

mark-ep commented 6 years ago

It turns out this is actually due to a mistake I made in commit add3663, which re-introduced the original problem. I've got a fix for it though so this should be sorted soon.

The new implementation of ecef2lla has passed the tests, so I'm happy to include that too.

mark-ep commented 6 years ago

this is fixed with commit db9da6f

hans-permana commented 6 years ago

I can confirm that the issue has been fixed. Thanks!