Results are not reproducible

[costeladebarata]

Hi folks,

I'm having some trouble trying to reproduce results.

As I initially thought it could be a python version or other machine-specific issue, I've set up a simple for loop to change only my simulation name so that I could save the results with different names.

It turns out that my Wm2Front and Wm2Back are never the same! I ran three rounds with 3 simulations each, restarting the kernel and deleting all variables between each round and I ended up with 9 different sets of values. Again, note that no variable was changed at all, I only changed the simulation name.

Anyone else having this issue? Or better, is there any random process in the source code that will inevitably make the results to be randomly different at each simulation??

Thanks!

[costeladebarata]

For reference, I calculated the average value across the module for each simulation and the standard deviation of Wm2Front results was 1.4%, with a 4% difference between highest and lowest result. The standard deviation for the Wm2back was 0.2%, with a 1% difference between highest and lowest result.

[jalderman9]

@costeladebarata I think what you're seeing is simply the nature of the backwards ray tracing. If you manually run rtrace commands repeatedly you'll see the same variations. Each time a ray interacts with a surface a new path is determined probabilistically.

[jalderman9]

Here's an example of running rtrace on one of the oct files using the exact same command that bifacial_radiance is using. The last 3 numeric columns are RGB irradiance and they get summed as Wm2Front or Wm2Back and you can see that the numbers vary a bit each time the command is run.

jalderman@crunchy ~/Downloads/bifacial_radiance/output $ for i in $(seq 1 10)
> do
> echo '-0.8677099862885846 -5.313191286482546e-17 1.89844479015825 -0.000 -0.101 -0.995 5  5' | rtrace -i -ab 2 -aa .1 -ar 256 -ad 2048 -as 256 -h -oovs 1axis_01_01_08.oct >> rtrace_results.txt
> done

jalderman@crunchy ~/Downloads/bifacial_radiance/output $ cat rtrace_results.txt 
-8.677100e-01   -5.313191e-17   1.898445e+00    2.952460e+00    2.952461e+00    2.952461e+00    a13.3.a0.TorqueTube_1P.6457 
-8.677100e-01   -5.313191e-17   1.898445e+00    2.953019e+00    2.953019e+00    2.953019e+00    a13.3.a0.TorqueTube_1P.6457 
-8.677100e-01   -5.313191e-17   1.898445e+00    2.950714e+00    2.950714e+00    2.950714e+00    a13.3.a0.TorqueTube_1P.6457 
-8.677100e-01   -5.313191e-17   1.898445e+00    2.954782e+00    2.954782e+00    2.954782e+00    a13.3.a0.TorqueTube_1P.6457 
-8.677100e-01   -5.313191e-17   1.898445e+00    2.954652e+00    2.954652e+00    2.954653e+00    a13.3.a0.TorqueTube_1P.6457 
-8.677100e-01   -5.313191e-17   1.898445e+00    2.952657e+00    2.952657e+00    2.952657e+00    a13.3.a0.TorqueTube_1P.6457 
-8.677100e-01   -5.313191e-17   1.898445e+00    2.956000e+00    2.956000e+00    2.956001e+00    a13.3.a0.TorqueTube_1P.6457 
-8.677100e-01   -5.313191e-17   1.898445e+00    2.951638e+00    2.951639e+00    2.951640e+00    a13.3.a0.TorqueTube_1P.6457 
-8.677100e-01   -5.313191e-17   1.898445e+00    2.953385e+00    2.953386e+00    2.953386e+00    a13.3.a0.TorqueTube_1P.6457 
-8.677100e-01   -5.313191e-17   1.898445e+00    2.954435e+00    2.954435e+00    2.954435e+00    a13.3.a0.TorqueTube_1P.6457```

[shirubana]

Hello~

Yes, thank you @jalderman9 . Ray tracing (forward and reverse for the case of radiance) is stochastic in nature, and as such the results will not be exactly the same, but they are inside of the threshold of uncertainty from irradiance measurements, particularly for rear irradiance (you can read more on our validation and this on https://ieeexplore.ieee.org/abstract/document/8534404 )

What we have done in the past to deal with this is run variuos simulations for the same exact scenario and average the results. This has proven better/faster than running a higher accuracy raytrace.

I'l ltry to document this better on the wiki in the upcoming month, thank you for bringing it up.

[costeladebarata]

Sweet! Thanks @jalderman9 and @shirubana ! Appreciate your comments, this I way I know it's not my code or a specific setting that I missed. Hopefully it will be useful for more people.