search

hep-packaging-coordination / packaging-hep-simulation-stack

Tracking project of the goal of packaging the full HEP simulation stack on conda-forge
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

Decoupling FF #4

Open matthewfeickert opened 1 week ago

matthewfeickert commented 1 week ago

FF seems to be a problem in terms of actually packaging things in a way that makes sense when it comes to conda-forge.

The source for FF is distributed on https://www.nikhef.nl/~form/maindir/others/ff/ff.html, but as I have very little faith that unmaintained websites will work forever I'm also attaching a version of it here: ff.tar.gz

FF has some problems:

[dependencies]
fortran-compiler = ">=1.8.0,<2"
make = ">=4.4.1,<5"

running

$ curl -sLO https://www.nikhef.nl/~form/maindir/others/ff/ff.tar.gz
$ tar -xzf ff.tar.gz
$ cd ff
$ make

results in lots of warnings and then fails with

...
$CONDA_PREFIX/bin/x86_64-conda-linux-gnu-gfortran   -c -o ffxd0p.o ffxd0p.f
ffxd0p.f:463:31:

  463 |                    call ffcxra(cs4(169),ipi12(25),xqi,qiDqj,sdel2,3,ier)
      |                                             1
Error: Actual argument contains too few elements for dummy argument 'ipi12' (2/3) at (1)
ffxd0p.f:466:31:

  466 |                    call ffxtra(cs4(169),ipi12(25),xqi,qiDqj,sdel2,3,ier)
      |                                             1
Error: Actual argument contains too few elements for dummy argument 'ipi12' (2/3) at (1)
make: *** [<builtin>: ffxd0p.o] Error 1

However, it gets more complicated as it seems that almost every theory tool vendors their own different version of FF.

For example, LoopTools (also currently without immutable distributions, so LoopTools-2.16.tar.gz) vendors a version of FF that seems to be arranged quite differently in source file content from what the original FF source has

$ curl -sLO https://feynarts.de/looptools/LoopTools-2.16.tar.gz
$ tar -xzf LoopTools-2.16.tar.gz
$ tree -L 1 LoopTools-2.16/src/
LoopTools-2.16/src/
├── A
├── B
├── C
├── D
├── E
├── frontend
├── include
├── makefile
├── tools
└── util

9 directories, 1 file

and (LoopTools author) Thomas Hahn (@t-hahn) mentions in email:

LoopTools embeds the old FF package of Geert Jan van Oldenborgh, and I make no secret out of this in the manual (Appendix A).

Geert Jan actually once deleted his copy of the source and asked me for a backup copy, so in that sense the FF version inside LoopTools may have become the "original" then.

The FF sources are still available here: https://www.nikhef.nl/~form/maindir/others/ff/ff.html but you cannot easily remove the FF code from LoopTools and plug the NIKHEF version, there have been subtle and not-so-subtle changes plus of course the integration into the LoopTools build.

It is worth noting that LoopTools has a license of GNU GPL v3.

Similarly, the Fortran version of QCDLoop (QCDLoop-1.98.tar.gz) also vendors a version of FF (with an entry in the README from 2003-09-08) though the QCDLoop README notes

The directory ff is taken from http://www.xs4all.nl/~gjvo/FF.html. The file ffinit_mine.f replaces the original file ff_init.f. One has to adjust here the path (twice) and set it to the location of the the ff library. The makefile in ff has been modified accordingly.

This does build, but it builds a statically linked library. QCDLoop is also unlicensed software.

So it seems that to move forward we need a version of FF that:

I'm tempted to try to take the version of FF in QCDLoop, put it in a GitHub repository (under MIT), get it up on conda-forge and try to go from there until I realize that I've messed something up.

Thoughts in general @chrisburr @henryiii?

t-hahn commented 1 week ago

So it seems that to move forward we need a version of FF that:

  • Builds with modern gfortran
  • Is hosted in a standalone repository
  • Builds a dynamically linked library
  • Can be used in LoopTools and the Fortran QCDLoop

I'm tempted to try to take the version of FF in QCDLoop, put it in a GitHub repository (under MIT), get it up on conda-forge and try to go from there until I realize that I've messed something up.

Thoughts in general @chrisburr @henryiii?

There are (at least) 2-3 fixes in the FF version of LoopTools w.r.t. that of QCDLoop, that is, not just compiler quirks but places where the numerical output of a function is actually different.

I leave it up to you to distribute whatever version of FF you wish but if you plan to rewire the FF functions inside LoopTools the resulting package must not be called LoopTools.

Best Wishes,

Thomas

matthewfeickert commented 1 week ago

There are (at least) 2-3 fixes in the FF version of LoopTools w.r.t. that of QCDLoop, that is, not just compiler quirks but places where the numerical output of a function is actually different.

@t-hahn can you elaborate, or give context on how your revision of the FF source code structure works?

Given what you've said here, it seems that the concept of FF doesn't really make sense anymore (beyond the existing original library that doesn't compile as is on modern Fortran). There's LoopTools-FF and QCDLoop-FF, but as you've both diverged, and neither provide public version controlled source code it doesn't seem like there is an obvious "FF" at the moment.

but if you plan to rewire the FF functions inside LoopTools the resulting package must not be called LoopTools.

I'm not really sure what you mean by "rewire". Do you use (LoopTools-)FF as a library or not?

matthewfeickert commented 1 week ago

There's LoopTools-FF and QCDLoop-FF, but as you've both diverged, and neither provide public version controlled source code it doesn't seem like there is an obvious "FF" at the moment.

Maybe it is best to lean into this and just make sure that the two "FF" libraries get placed under different lib/<package name> directories so that when they both get installed there won't ever be a collision.

but if you plan to rewire the FF functions inside LoopTools the resulting package must not be called LoopTools.

I'm not really sure what you mean by "rewire".

@t-hahn as you mentioned in email

It seems that by design LoopTools tries to statically link everything. Is it possible to support dynamic linking through a flag given that conda-forge strongly recommends (and works best) for dynamic libraries?

LoopTools 1.x built shared libraries but this became a never-ending source of user confusion, so I dumped it and went for static only. Furthermore shared libraries are IMO the single most significant obstacle to long-term code use in at least Linux.

So no, I don't plan to add shared linking, but you can try of course.

to double check, do you view patching LoopTools to produce share libraries as "rewiring"?

matthewfeickert commented 1 week ago

Maybe it is best to lean into this and just make sure that the two "FF" libraries get placed under different lib/<package name> directories so that when they both get installed there won't ever be a collision.

Oh, I guess that's not relevant for the statically linked version of LoopTools as it just produces a single library libooptools.a that contains FF inside of it. So from this perspective LoopTools doesn't care about any other existing form of FF. That could work to get a statically linked version up for the time being.

t-hahn commented 1 week ago

There are (at least) 2-3 fixes in the FF version of LoopTools w.r.t. that of QCDLoop, that is, not just compiler quirks but places where the numerical output of a function is actually different.

@t-hahn can you elaborate, or give context on how your revision of the FF source code structure works?

I'd have to dig up details from somewhere deep down my mail folders, but over the years I received various bug reports that led to fixes in the FF routines contained in LoopTools.

to double check, do you view patching LoopTools to produce share libraries as "rewiring"?

My impression this morning was...

I'm tempted to try to take the version of FF in QCDLoop, put it in a GitHub repository (under MIT), get it up on conda-forge and try to go from there until I realize that I've messed something up.

... that you were about to refactor QCDLoop's FF into a separate package and then replace the calls inside LoopTools to its own FF routines by calls to that package.

While you can perfectly do so within the licensing terms, I would ask you to rename the package then so I can point folks complaining about inconsistent results to you instead of going through the debugging again.

Maybe it is best to lean into this and just make sure that the two "FF" libraries get placed under different lib/<package name> directories so that when they both get installed there won't ever be a collision.

There is no file-level name collision that I can see, for LoopTools builds a single library libooptools.a.

API-wise LoopTools implements a rudimentary name mangling through the preprocessor (src/include/externals.h and src/tools/mkexternalsh). The effect of this is that a routine 'ffinit' ends up at object level as 'ljffinit' (lj being a combination of letters similar to lt but unlikely used elsewhere intentionally).

matthewfeickert commented 1 week ago

@t-hahn

I'd have to dig up details from somewhere deep down my mail folders, but over the years I received various bug reports that led to fixes in the FF routines contained in LoopTools.

Sure. I think that sounds like a lot of work on your side, so if it isn't something that's "in cache" for you then I would suggest ignoring this.

that you were about to refactor QCDLoop's FF into a separate package and then replace the calls inside LoopTools to its own FF routines by calls to that package.

No. Sorry for the miscommunication. The idea was that as

There's LoopTools-FF and QCDLoop-FF, but as [they]'ve both diverged ...

that I could take QCDLoop-FF and get that on conda-forge as "a version" as the diff between QCDLoop's FF and the 1998 release of FF is small, and so that's something that would be more immediately understandable than trying to figure out how to move from QCDLoop-FF to LoopTools-FF as there's quite a gap there.

I was not suggesting that I would take QCDloop-FF, attempt to rip out LoopTools-FF, and then package that all up and call it "LoopTools". That would be really bad (not to mention I doubt that would even work).

I would ask you to rename the package then so I can point folks complaining about inconsistent results to you instead of going through the debugging again.

Yeah, there is zero point in doing packaging work to create yet more divergences between things. It would be nice if someday there could be a FF on GitHub that both QCDLoop and LoopTools agreed worked and they could both use it as a submodule so that there was a clear source of truth, but I also appreciate that sounds like a lot of work and that unless someone gets funding to pay for a student to do this it will never happen (in any near term).

There is no file-level name collision that I can see, for LoopTools builds a single library libooptools.a.

Yup. I realized this a little bit after posting (I had my mental model of how LoopTools and QCDLoop do things flipped): https://github.com/hep-packaging-coordination/packaging-hep-simulation-stack/issues/4#issuecomment-2489457121

Given that you do this and so FF and LoopTools exist collectively in the static library, I've started the process of packaging LoopTools as is on conda-forge: https://github.com/conda-forge/staged-recipes/pull/28292 as there's some workarounds for libraries that are static only.

If you have any comments before merge, they're welcome, but after that PR gets merged the "feedstock" https://github.com/conda-forge/looptools-feedstock/ will be generated, which is where the packaging scripts to generate and distribute things live, and so things can always get addressed there too.

matthewfeickert commented 1 week ago

The current version of LoopTools "as is" is now packaged on conda-forge as a static library looptools-static: https://github.com/conda-forge/looptools-feedstock