Open FreddieWitherden opened 1 year ago
Can you share the graphs in Metis format to reproduce this locally?
So I sat down and bisected the git revisions and found the culprit was:
https://github.com/KarypisLab/METIS/commit/5ba158081c1373adf6de94d1e7a257f31490b9d3
which causes ABI breakage. Without recompilation, any METIS 5.1 application will pass an incorrect options array with 5.2 due to every option past METIS_OPTION_DBGLVL being shifted down by one.
I'll put together a PR later which gives these enum options explicit values so such breakage can be avoided in the future as/when new options are added.
This reordering of options broke using METIS 5.2.1 from MUMPS for me. They have code like
MUMPS_INT ncon, edgecut, options[40];
ierr=METIS_SetDefaultOptions(options);
options[0] = 0;
/* Use 1-based fortran numbering */
options[17] = 1;
ncon = 1;
ierr = METIS_PartGraphKway(n, &ncon, iptr, jcn,
NULL, NULL, NULL,
k, NULL, NULL, options,
&edgecut, part);
and I got a lot of complaints from Metis about the graph to the log, and then some crash.
Of course, it's not good that the Mumps people assumed that METIS_OPTION_NUMBERING
will always be 17
(they even include metis.h), but it seems that it could have been easily avoided in the metis side, too (or could be fixed in 5.2.2).
If I change
@@ -271,12 +271,10 @@ typedef enum {
METIS_OPTION_IPTYPE,
METIS_OPTION_RTYPE,
METIS_OPTION_DBGLVL,
- METIS_OPTION_NIPARTS,
METIS_OPTION_NITER,
METIS_OPTION_NCUTS,
METIS_OPTION_SEED,
METIS_OPTION_NO2HOP,
- METIS_OPTION_ONDISK,
METIS_OPTION_MINCONN,
METIS_OPTION_CONTIG,
METIS_OPTION_COMPRESS,
@@ -285,6 +283,8 @@ typedef enum {
METIS_OPTION_NSEPS,
METIS_OPTION_UFACTOR,
METIS_OPTION_NUMBERING,
+ METIS_OPTION_NIPARTS,
+ METIS_OPTION_ONDISK,
METIS_OPTION_DROPEDGES,
/* Used for command-line parameter purposes */
Mumps works fine again. (I would have complained there if they had a public issue tracker :))
I am maintaining the conda-forge build of METIS, so when the dust settles here let me know and I can bump the version and/or add a patch.
So I sat down and bisected the git revisions and found the culprit was:
which causes ABI breakage. Without recompilation, any METIS 5.1 application will pass an incorrect options array with 5.2 due to every option past METIS_OPTION_DBGLVL being shifted down by one.
Just for the sake of completeness, that commit is also included in METIS 5.1.1, so even an application built with METIS 5.1.0 will already gave wrong results when used at runtime with METIS 5.1.1 .
For reference, this is moptions_et
in METIS 5.1.0 :
/*! Options codes (i.e., options[]) */
typedef enum {
METIS_OPTION_PTYPE,
METIS_OPTION_OBJTYPE,
METIS_OPTION_CTYPE,
METIS_OPTION_IPTYPE,
METIS_OPTION_RTYPE,
METIS_OPTION_DBGLVL,
METIS_OPTION_NITER,
METIS_OPTION_NCUTS,
METIS_OPTION_SEED,
METIS_OPTION_NO2HOP,
METIS_OPTION_MINCONN,
METIS_OPTION_CONTIG,
METIS_OPTION_COMPRESS,
METIS_OPTION_CCORDER,
METIS_OPTION_PFACTOR,
METIS_OPTION_NSEPS,
METIS_OPTION_UFACTOR,
METIS_OPTION_NUMBERING,
/* Used for command-line parameter purposes */
METIS_OPTION_HELP,
METIS_OPTION_TPWGTS,
METIS_OPTION_NCOMMON,
METIS_OPTION_NOOUTPUT,
METIS_OPTION_BALANCE,
METIS_OPTION_GTYPE,
METIS_OPTION_UBVEC
} moptions_et;
and this is in METIS 5.1.1 and 5.2.1 :
/*! Options codes (i.e., options[]) */
typedef enum {
METIS_OPTION_PTYPE,
METIS_OPTION_OBJTYPE,
METIS_OPTION_CTYPE,
METIS_OPTION_IPTYPE,
METIS_OPTION_RTYPE,
METIS_OPTION_DBGLVL,
METIS_OPTION_NIPARTS,
METIS_OPTION_NITER,
METIS_OPTION_NCUTS,
METIS_OPTION_SEED,
METIS_OPTION_NO2HOP,
METIS_OPTION_ONDISK,
METIS_OPTION_MINCONN,
METIS_OPTION_CONTIG,
METIS_OPTION_COMPRESS,
METIS_OPTION_CCORDER,
METIS_OPTION_PFACTOR,
METIS_OPTION_NSEPS,
METIS_OPTION_UFACTOR,
METIS_OPTION_NUMBERING,
METIS_OPTION_DROPEDGES,
/* Used for command-line parameter purposes */
METIS_OPTION_HELP,
METIS_OPTION_TPWGTS,
METIS_OPTION_NCOMMON,
METIS_OPTION_NOOUTPUT,
METIS_OPTION_BALANCE,
METIS_OPTION_GTYPE,
METIS_OPTION_UBVEC
} moptions_et;
the diff is:
--- 5.1.0
+++ 5.1.1
@@ -6,10 +6,12 @@
METIS_OPTION_IPTYPE,
METIS_OPTION_RTYPE,
METIS_OPTION_DBGLVL,
+ METIS_OPTION_NIPARTS,
METIS_OPTION_NITER,
METIS_OPTION_NCUTS,
METIS_OPTION_SEED,
METIS_OPTION_NO2HOP,
+ METIS_OPTION_ONDISK,
METIS_OPTION_MINCONN,
METIS_OPTION_CONTIG,
METIS_OPTION_COMPRESS,
@@ -18,6 +20,7 @@
METIS_OPTION_NSEPS,
METIS_OPTION_UFACTOR,
METIS_OPTION_NUMBERING,
+ METIS_OPTION_DROPEDGES,
/* Used for command-line parameter purposes */
METIS_OPTION_HELP,
For anyone interested, a patch that make mumps 5.2.1 work with metis 5.1.1 and 5.2.1 (but breaking compatibility with metis 5.1.0) that seems to work is available at https://github.com/conda-forge/mumps-feedstock/blob/c524cb3c71686bee59d9b12df5d9d6ce20782ce4/recipe/mumps_support_only_metis_5_1_1.patch .
Using METIS 5.1:
where the partitioning and renumbering (both of which make calls to
METIS_PartGraphRecursive
) complete almost immediately. By contrast using METIS 5.2.1:where we can see a huge slow down (on the order of ~2000x) for the partition graph portion which makes a single call to
METIS_PartGraphRecursive
. The inputs are identical in both cases, also reproduced withMETIS_PartGraphKway
. Also reproduced on both Linux (x86-64) and macOS (AARCH64).This occurs with all of our grids/meshes. Profiling 5.2.1 with
perf record
we find:whereas with 5.1 (good) we find:
where METIS is just a rounding error in the runtime.