Same seed yields different results on GPU

[Quasimondo]

Running texturizer with the same seed results in slightly different results when repeated. I would expect the same seed to generated the identical texture every time. Here are three examples:

[Quasimondo]

Looks like this is caused by some non-deterministic CUDA functions - since when running the same seed on the CPU the results are identical.

From https://pytorch.org/docs/stable/notes/randomness.html:

There are some PyTorch functions that use CUDA functions that can be a source of nondeterminism. One class of such CUDA functions are atomic operations, in particular atomicAdd, which can lead to the order of additions being nondetermnistic. Because floating-point addition is not perfectly associative for floating-point operands, atomicAdd with floating-point operands can introduce different floating-point rounding errors on each evaluation, which introduces a source of nondeterministic variance (aka noise) in the result.

[htoyryla]

Try torch.backends.cudnn.deterministic = True ?

I use this all the time, appears to save memory.

[Quasimondo]

Ah, sorry I didn't mention that of course I had tried all the usual settings mentioned in the docs. It does not help.

[htoyryla]

Oh yes... good to know.

[alexjc]

It's good to know the CPU results are identical, that rules out issues & bugs directly in this library ;-)

It's not clear to me what is causing the non-determinism on GPU. torch.nn.Conv2d should become deterministic with torch.backends.cudnn.deterministic = True, but the other candidate in the encoder is only torch.nn.AvgPool2d — which has no warnings in the documentation about determinism.

It may be that the L-BFGS optimizer itself also introduces some non-determinism. Testing a plain SGD optimizer would rule that out...

[Quasimondo]

I had first suspected the LBFGS solver and tentatively replaced it with Adam. The results look like crap, but they also differ when run on CUDA with the same seed.

I forgot to quote the most important line from that "Reprocibility" link above: **

"There is currently no simple way of avoiding nondeterminism in these functions."

**

It's a pity since I was trying to do some seed interpolation animations as well as a sliding window test for getting larger textures than fit in the GPU, but both of them do not work with CUDA since they change on every iteration.

[alexjc]

The interpolation would not be deterministic anyway, as soon as you change the content of the starting tensor, it would introduce different numbers and hence instability. https://arxiv.org/abs/1705.02092v1

As for larger textures, I have a variety of code for tiling but not with gram matrices yet... I will see what's possible!