Refactor sum layers + consistency checks + bug fixes

[loreloc]

As we discussed and to avoid any confusion, this merges dense and mixing layers into a more general sum layer, as formalized in https://arxiv.org/abs/2409.07953.

Moreover, this adds sanity checks on the symbolic representation (symbolic circuit, parameterizations and initializations), thus preventing some bugs (e.g. shape mismatch) to happen after compilation and optimizations.

Finally, as I was already changing the implementation of the sum layer in torch, this PR also fixes the bug causing sum layers to output NaNs when the input to a sum unit are all +-infs.

Closes #309 #316 #319

Detailed changes:

• 309 Replaced symbolic mixing and dense layers with a symbolic sum layer.

  • Add use_mixing_weights boolean flag in image_data template (see here), as to enable a parameterization of sum layers with arity > 1 that matches the one we had with mixing layers. This defaults to True for compatibility with previous results.
  • Add nary_sum_weight_factory in circuit from region graph method as to optionally enable the specification of a different weight factory for sum layers with arity > 1.
  • Fused torch dense and mixing layer implementations into a single one. When compared to cirkit==0.1.0 you might notice an increase in the total number of parameters. This is normal as the sparse weights in mixing layers are now stored in dense weights of sum layers. Performances-wise, it seems the current implementation is still just as fast as the one in cirkit==0.1.0 (briefly tested with the compilation-options.ipynb notebook).
  • Updated torch layer optimization rules as to apply known optimizations only for sum layers having arity=1 (i.e., the previously known dense layers)

• 316 Add a number of sanity checks to the symbolic representation, such as on the graph structures and on the input/output shapes

• 319 Fixed bug causing sum layers to output NaN when entries being log-sum-expd are all infs. This has been done by clamping infs values to be finite (based on the chosen precision).

• Re-run notebooks

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 57.47126% with 37 lines in your changes missing coverage. Please review.

Project coverage is 68.25%. Comparing base (3e51b42) to head (9a48049).

Files with missing lines	Patch %	Lines
cirkit/templates/circuit_templates/utils.py	0.00%	15 Missing :warning:
cirkit/backend/torch/layers/inner.py	45.45%	6 Missing :warning:
cirkit/symbolic/circuit.py	50.00%	4 Missing and 1 partial :warning:
cirkit/backend/torch/parameters/pic.py	0.00%	3 Missing :warning:
cirkit/templates/circuit_templates/data.py	0.00%	3 Missing :warning:
cirkit/backend/torch/compiler.py	50.00%	1 Missing and 1 partial :warning:
cirkit/backend/torch/optimization/registry.py	60.00%	2 Missing :warning:
cirkit/backend/torch/optimization/layers.py	94.73%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #314 +/- ## ========================================== + Coverage 68.18% 68.25% +0.07% ========================================== Files 51 51 Lines 5368 5314 -54 Branches 616 614 -2 ========================================== - Hits 3660 3627 -33 + Misses 1492 1473 -19 + Partials 216 214 -2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.