opera-adt / disp-s1

OPERA Displacement workflows
Other
5 stars 6 forks source link

Add new validation checks #54

Open gmgunter opened 5 months ago

gmgunter commented 5 months ago

Closes #12

Add new checks against the frame-to-burst JSON file and phase-linked interferogram to the disp-s1 validate command:

scottstanie commented 4 months ago
  1. Remembering a conversation with SDS, it would be good to add an option to let it run through all problems rather than exit out at the first error. I'm thinking something like --fail-fast as a flag, where the default is run through everything. Then it would print out at the end
  File "/u/aurora-r0/staniewi/repos/disp-s1/src/disp_s1/validate.py", line 761, in validate
    raise ValidationError(combined_msg)
disp_s1.validate.ValidationError: Validation failures:
unwrapped phase dataset '/unwrapped_phase' failed validation: too many nan values (0.9532800448888682 > 0.01)
unwrapped phase dataset failed validation: phase values were not congruent with reference dataset

I started a version of this here, where it's still half-baked from not changing the return types/implementing the fail-fast flag

  1. ~it wasn't changed in this PR, but the nan_threashold=0.01 is way too low because of the geocoding tilt. a normal unw has like 20-30% nans~ edit: the validation as failing due to the new CCL nodata value of 65535. separate bug.

maybe these should be in another PR though

gmgunter commented 4 months ago

I like the --fail-fast idea, but IMO let's make it a separate PR.

In the case where --fail-fast is disabled, would the individual validation errors contain traceback info? Or just the error message?

scottstanie commented 4 months ago

I like the --fail-fast idea, but IMO let's make it a separate PR.

In the case where --fail-fast is disabled, would the individual validation errors contain traceback info? Or just the error message?

The first version just had the error messages join by newlines- do you think the further traceback is helpful?