Different results in DICe interface compared to custom application (Due to version of OpenCV used)

Hello DICe developers,

Thank you for your software, I find it very useful !

I was trying to write a custom application based on the custom_app example provided in the tests/examples folder, and to compare the results I was getting to the ones coming from DICe software (Digital Image Correlation Engine), in order to see if my implementation was good.

I managed to load an input parameter file and a correlation parameter file into my application, and I managed to extract some results from my correlation. I also launched a 2d analysis on DICe software with the same parameters.

My issue is that the results I get from my application are slightly different than the results coming from DICe software, and I can't figure out why. First I thought that maybe some parameters DICe software used were hidden to the user, so I tried to modify some parameters in my application, but I can't get the same results as DICe software... Maybe you would have an idea on what I did wrong here ?

Here is my code :

using namespace DICe::field_enums;

int main(int argc, char* argv[]) {

    std::cout << "Begin custom_app example\n";
    DICe::initialize(argc, argv);

    int NB_POINTS = 25;

    DICe::Schema schema("input.xml", "params.xml");

    schema.set_ref_image("images//aaa_b2_00.tif");

    for (auto indexImage = 1; indexImage <= 2; ++indexImage)
    {
        schema.set_def_image("images//aaa_b2_0" + std::to_string(indexImage) + ".tif");

        schema.execute_correlation();
        schema.execute_post_processors();

        // compute displacement and score of each points in the current frame
        auto const results = schema.mesh();
        Teuchos::RCP<DICe::MultiField> subsetDisplacementsX = results->get_field(SUBSET_DISPLACEMENT_X_FS);
        Teuchos::RCP<DICe::MultiField> subsetDisplacementsY = results->get_field(SUBSET_DISPLACEMENT_Y_FS);
        Teuchos::RCP<DICe::MultiField> subsetsScores = results->get_field(GAMMA_FS);

        std::cout << "[IMAGE " << indexImage << "]" << std::endl;
        for (auto indexPoint = 0; indexPoint < NB_POINTS; ++indexPoint)
        {
            cv::Point2d const displacement(subsetDisplacementsX->local_value(indexPoint), subsetDisplacementsY->local_value(indexPoint));
            auto const score = schema.local_field_value(indexPoint, GAMMA_FS);
            std::cout << "[" << indexPoint << "] =  " << displacement << "  " << score << std::endl;
        }
        std::cout << std::endl;

    }
    DICe::finalize();
    std::cout << "End custom_app example\n";

    return 0;
}

Here is my input.xml file :

<ParameterList>
  <Parameter name="output_folder" type="string" value="E:\WS\Test\Test\results\" /> 
  <Parameter name="correlation_parameters_file" type="string" value="E:\WS\Test\Test\params.xml" />
  <Parameter name="subset_file" type="string" value="./subsets.txt" />
  <Parameter name="subset_size" type="int" value="25" />
  <Parameter name="step_size" type="int" value="100" />
  <Parameter name="separate_output_file_for_each_subset" type="bool" value="false" />
  <Parameter name="create_separate_run_info_file" type="bool" value="true" />
  <Parameter name="image_folder" type="string" value="E:\WS\Test\Test\images\" />
  <Parameter name="reference_image_index" type="int" value="0" />
  <Parameter name="start_image_index" type="int" value="0" />
  <Parameter name="end_image_index" type="int" value="2" />
  <Parameter name="skip_image_index" type="int" value="1" />
  <Parameter name="num_file_suffix_digits" type="int" value="2" />
  <Parameter name="image_file_extension" type="string" value=".tif" />
  <Parameter name="image_file_prefix" type="string" value="aaa_b2_" />
</ParameterList>

Here is the subsets.txt showing the location of the subsets :

BEGIN SUBSET_COORDINATES
24 24
24 124
24 224
24 324
24 424
124 424
124 324
124 224
124 124
124 24
224 24
224 124
224 224
224 324
224 424
324 424
324 324
324 224
324 124
324 24
424 24
424 124
424 224
424 324
424 424
END SUBSET_COORDINATES

And here is the params.xml file :

<ParameterList>
  <Parameter name="interpolation_method" type="string" value="KEYS_FOURTH" />
  <Parameter name="sssig_threshold" type="double" value="148" />
  <Parameter name="optimization_method" type="string" value="GRADIENT_BASED" />
  <Parameter name="initialization_method" type="string" value="USE_FEATURE_MATCHING" />
  <Parameter name="enable_translation" type="bool" value="true" />
  <Parameter name="enable_rotation" type="bool" value="false" />
  <Parameter name="enable_normal_strain" type="bool" value="false" />
  <Parameter name="enable_shear_strain" type="bool" value="false" />
  <Parameter name="output_delimiter" type="string" value="," />
  <ParameterList name="output_spec"> 
    <Parameter name="COORDINATE_X" type="bool" value="true" />
    <Parameter name="COORDINATE_Y" type="bool" value="true" />
    <Parameter name="DISPLACEMENT_X" type="bool" value="true" />
    <Parameter name="DISPLACEMENT_Y" type="bool" value="true" />
    <Parameter name="SIGMA" type="bool" value="true" />
    <Parameter name="GAMMA" type="bool" value="true" />
  </ParameterList>
</ParameterList>

Here is a screenshot of DICe software showing the parameters I entered : Parameters DICe software

The images I used are the 3 first images of Sample 10 from 2D DIC challenge (named _aaa_b200.tif, _aaa_b201.tif and _aaa_b202.tif). I can upload them if they are not accessible to you.

Here is a screenshot of the results I got from DICe software : Results DICe software

And there is a screenshot of the results I got from my implementation : Results implementation

As you can see, the DISPLACEMENT_X for subset #1 is slighly different, even though the DISPLACEMENT_Y and GAMMA values are not different. But some points like #8 or #11 for example are not good in my implementation while they are in DICe software results.

I can not figure out why, so if you had any idea on what went wrong I would really appreciate it.

Sorry for the long message. Thank you in advance for your help.

Thanks for the details in your issue, they are helpful. I'm not sure why the results are different, but I do have some ideas you could try.

Are you using the same input.xml and params.xml files from the DICe GUI and your custom app? I couldn't quite tell from above if that was the case.
If you have a local build of DICe, you can set the -D DICE_DEBUG_MSG:BOOL=ON \ flag in your cmake script and then run both your custom app and the DICe executable on your input.xml file and compare the two side by side. You should get enough debug output to see what is going on.

I'll try to think of other reasons that the results could be different, but I wanted to share those ideas first to see if they help.

Hello, Thank you for your answer.

There is just one difference between the input.xml and params.xml files I used from DICe GUI and in my custom app : DICe GUI did not output the "subset_file" line in input.xml and did not create a subsets.txt file with the coordinates of the subsets. So I manually added this line in the input.xml file for my custom app (and added the subsets.txt file).
I do have a local build of DICe, so I will try to launch my custom app with the -D DICE_DEBUG_MSG:BOOL=ON \ flag in my cmake script. But I do not understand which DICe executable you are reffering to ? If you meant the _DICegui.exe in the DICe software installation folder, then can you give me some more explanations (as I don't think that this executable would be linked to the cmake script I would run) ? Or maybe you meant that there is a DICe_gui executable in the local build ? (But I did not find it)

Again, thank you for you help.

Regarding the DICe executable, what I was referring to is the dice.exe executable that gets built in the build directory. The GUI is just a javascript front end to that executable. If you create an analysis in the GUI all it really does is create an input.txt file and params.txt file and then call dice.exe. You can also run DICe from the command line with

dice.exe -i input.xml

That way you can compare the debugging output to the command prompt with what you get when you build your custom app.

Hello,

Thank you for your answer. I built the library with the flag _DICE_DEBUGMESSAGE ON, and I managed to launch the executable you referred to in your answer.

After comparing the outputs of the executable vs my custom application, it seems that I forgot to call schema.swap_def_prev_images(); in my application before calling schema.set_def_image.

I did not see this instruction in the main.cpp of the custom_app example provided in the library, so I guess it has to be used when working on more than one image, right ?

Thank you for your help, I am very grateful.

You are right. Looks like since the custom app example only has one image, the call to post_execution_tasks() is never made. I've added it to the custom app example in this commit: https://github.com/dicengine/dice/commit/ce2f9e4fba3f398df3c1825b384ce74dc67f0770 so that others can avoid the same pitfall.

Schema::post_execution_tasks() is the better method to call in case in the future other tasks, aside from swap_def_prev_images() are added to the set of post execution tasks.

All right, I will call post_execution_tasks then.

Thank you again for your help !

Hello again,

Sorry to reopen a closed issue, but i find that my problem is not totally resolved. I thought it was, as adding post_execution_tasks instruction worked for this sample. But when I tried on Sample 2 from DIC challenge, I obtain different results from DICe gui and my implementation. I tried launching the DICe executable, but I get the same results as in my implementation.

Here is my process : First, I used DICe gui to launch an analysis. Here is a screenshot of the parameters I used : I worked on the first 2 images and on very few points in order to reduce the calculation time and to try to narrow the problem.

Then, I launched the DICe executable (found in the build directory, as you mentioned earlier) with the input.xml file generated by the gui. Here are the results I get : On the left, you will find the results I get from DICe gui, and on the right the ones I get using DICe executable.

As you can see, most of the values are the same on both tests, but some values may differ for some points (the displacement_x value for point with id #2 for example). I find this odd, as you mentioned the gui was just an interface that actually launched the executable. But I can not figure out where the problem is...

In case you would like to try on your side, there is my input.xml file :

<!-- Auto generated input file from DICe GUI -->
<ParameterList>
<Parameter name="output_folder" type="string" value="E:\Videos\DIC 2D-Challenge 1.0\Sample2\results\" /> 
<Parameter name="correlation_parameters_file" type="string" value="E:\Videos\DIC 2D-Challenge 1.0\Sample2\params.xml" />
<Parameter name="subset_size" type="int" value="21" />
<Parameter name="step_size" type="int" value="100" />
<Parameter name="separate_output_file_for_each_subset" type="bool" value="false" />
<Parameter name="create_separate_run_info_file" type="bool" value="true" />
<Parameter name="image_folder" type="string" value="E:\Videos\DIC 2D-Challenge 1.0\Sample2\" />
<Parameter name="reference_image_index" type="int" value="0" />
<Parameter name="start_image_index" type="int" value="0" />
<Parameter name="end_image_index" type="int" value="1" />
<Parameter name="skip_image_index" type="int" value="1" />
<Parameter name="num_file_suffix_digits" type="int" value="2" />
<Parameter name="image_file_extension" type="string" value=".tif" />
<Parameter name="image_file_prefix" type="string" value="trs2_b8_" />
</ParameterList>

and my params.xml file :

<!-- Auto generated parameters file from DICe GUI -->
<ParameterList>
<Parameter name="interpolation_method" type="string" value="KEYS_FOURTH" />
<Parameter name="sssig_threshold" type="double" value="0" />
<Parameter name="optimization_method" type="string" value="GRADIENT_BASED" />
<Parameter name="initialization_method" type="string" value="USE_FEATURE_MATCHING" />
<Parameter name="enable_translation" type="bool" value="true" />
<Parameter name="enable_rotation" type="bool" value="false" />
<Parameter name="enable_normal_strain" type="bool" value="false" />
<Parameter name="enable_shear_strain" type="bool" value="false" />
<Parameter name="output_delimiter" type="string" value="," />
<ParameterList name="output_spec"> 
<Parameter name="COORDINATE_X" type="bool" value="true" />
<Parameter name="COORDINATE_Y" type="bool" value="true" />
<Parameter name="DISPLACEMENT_X" type="bool" value="true" />
<Parameter name="DISPLACEMENT_Y" type="bool" value="true" />
<Parameter name="SIGMA" type="bool" value="true" />
<Parameter name="GAMMA" type="bool" value="true" />
<Parameter name="BETA" type="bool" value="true" />
<Parameter name="STATUS_FLAG" type="bool" value="true" />
<Parameter name="UNCERTAINTY" type="bool" value="true" />
</ParameterList>
</ParameterList>

I understand these images are not well contrasted and therefore the ssig_threshold parameter should not be set to 0 as it would "force" some low contrasted points to be part of the analysis (if I understood well), but I was trying to find the limitations of my implementation.

If you have any tips regarding this topic, I would be very grateful. Again, I am sorry to reopen this closed issue... Thanks in advance for your help.

That's strange. Here are a few more questions to try and get to the bottom of this:

Is it possible that you have multiple instances of DICe installed on your machine? Check the .dice.js file in your home directory to make sure that the execPathOverrride variable is not set. Also in the global.js file in /Applications/DICe.app/Contents/Resources/app/js/global.js make sure the execPath variable points to where you are running DICe from the command line. (I'm assuming you're on a Mac based on how the GUI looks in the images above)

Also, it looks like your GUI install is from an older version since the copyright at the bottom is from 2017. You might consider updating your install of DICe to the most current beta release.

Let me know if either of those make a difference.

Thank you for your answer.

Sorry I didn't provide that information earlier, but I work on Windows. I tried to check the files you pointed to. I could not find a .dice.js file, but I found a dice.js file in resources/app/js folder in the installation folder of the GUI. I could not find the execPathOverride variable in this file though, so it is probably not the file you were referring to. i found the global.js file, and the execPath variable does point to where I am running DICe from the command line.

I also downloaded the latest version of the GUI as you mentionned. You were right, I was working with a way older version.

To add some informations (as I don't think it was totally clear) :

On one hand, I installed the latest version of the GUI, and used the executable provided in the installation folder.
On the other hand, I followed the steps here to be able to build DICe on my machine. I first downloaded DICe source code from here, then downloaded DICeDevPack, updated the do-cmake.bat script with the correct paths, launched the script, then nmake, then nmake install commands. OpenCV 4.0.0 provided in the DICeDevPack was used. That way I had in my build/bin folder a dice.exe executable to launch in order to test the DICe building I made.
With the executable from the GUI, I do obtain the same results as in the GUI itself. But when using the executable generated when building the library, I obtain the slightly different results provided on the previous comment.

Something I have not tried yet : the DICe source code I work on is version 3.0-beta.3, as it was the latest release at the time I started working on DICe. I see you released a beta.4 version, so I will try to use this version, in order to see if I could get some different results with my build.

Thank you for your help.

I think what you are running into is just the minor differences in DICe between versions. If you are ever curious which version of DICe is actually being executed (by the GUI or in your own build) you can call

dice.exe --version

And you should get the precise commit that represents the snapshot of the code that was used to build that version. For the most part, there should be strong correspondence between various versions of DICe, but in some cases we make changes that actually have an impact on the results you see. Hopefully this is for the better, but let me know if you run into anything suspicious. It's helpful to have feedback from people like you. (Especially when they provide enough detail to track down the issue like you did so thanks!)

You are welcome, I would be glad to help your project moving in the right direction !

I tried running the command you mentionned. With the executable from the GUI, I get this result :

You were right to say that there might be a version issue, as my built executable did not point to the same commit. So I cloned the DICe repository and checked out the same commit as the GUI executable. I rebuilt the library, and relaunched the command to make sure the same commit was pointed to.

But I still got the different results...

The only explanation I saw then was that the different results come from the different versions of OpenCV used in my built and in the GUI. If I am not wrong, the GUI uses OpenCV 3.2.0, but in the DICe DevPack, OpenCV 3.2.0 and 4.0.0 are provided. Previously, I was using OpenCV 4.0.0, so I tried to build DICe using OpenCV 3.2.0 and now I get the same results as in the GUI ! So I guess the problem is solved.

However, I still have a few questions :

I know the differences seem minor, but I assume that they would lead to bigger ones if the analysis was launched on more images. So now that there are (at least) 2 different results, depending on the version of OpenCV, is there one of them you would favor above the other ?
Any idea on what difference between the two versions of OpenCV might lead to a different result in your code ?
Is there a particular reason why DICe is still using OpenCV 3.2.0, and not an upper version ? Of course, I understand that updating to a more recent version would take some time, and that it might not be a top priority for you.

Anyway, I am glad we got to the bottom of this, and I would like to thank you very much for your time.

I know the differences seem minor, but I assume that they would lead to bigger ones if the analysis was launched on more images. So now that there are (at least) 2 different results, depending on the version of OpenCV, is there one of them you would favor above the other ?

I don't think so. It seems like it would be pretty easy to run your example with many more data points and compare the solutions to see if it is substantial. I would want to know if that is the case. What I see in your results above is a difference of about 1/100th of a pixel which is well within the uncertainty bounds of DIC. You should see similar variation from switching computers/operating systems, for example.

Any idea on what difference between the two versions of OpenCV might lead to a different result in your code ?

My impression is that the way the feature matching in OpenCV is done (which DICe uses for initialization of the subset deformation) was changed between version 3 and 4 of OpenCV. I haven't done a study to see what the impact is, but that's my hunch. At this point I don't have a good sense of which one is better, but I'm hoping that it doesn't introduce a tremendous difference.

Is there a particular reason why DICe is still using OpenCV 3.2.0, and not an upper version ? Of course, I understand that updating to a more recent version would take some time, and that it might not be a top priority for you.

The reason that we use 3.2 is that we haven't done the work of updating the DICe code to keep up to date with the latest OpenCV version. In some cases there are depricated functions, etc. that need to be addressed. We would like to stay more up to date with current version so we are working on a build framework that automatically pulls the most recent versions of OpenCV (and the other DICe dependencies) and builds against them nightly. That's probably a few months away from being finished though.

Thanks for your reply.

To give you some more informations on whether the differences would be substantial or not, I ran a few more tests on more samples from DIC challenge, and saw approximately the same kind of differences.

However, on Sample 13, I found that some points seem to still being tracked in one case and not being tracked anymore in the other. Here are my results on frame 9 so you can see : On the left are the results from DICe with opencv 3.2 and on the right results from opencv 4.0.

Again, there are some tiny differences on the displacements of some points, but what is interesting now is for instance point #46 that is not tracked anymore in this image, when it still is with the more recent opencv version. Or point #22 that is still tracked on the left, but not on the right.

I guess the changes in opencv might have some impacts there too.

I just wanted to point that to you so you would know. I will definitely watch the future versions you will release on DICe.

Again, thank you for your help.

dicengine / dice

Different results in DICe interface compared to custom application (Due to version of OpenCV used) #239