--mfaktc and --mfakto prototype

[brubsby]

Very much a work in progress, but creating this PR mostly as a jumping off point for discussion and feedback. Probably won't be ready for v2.2, and definitely shouldn't hold up the release of CERT support.

Current TODO:

• [x] finish mfaktc changes
  • [x] add os info to json
  • [x] build on linux
  • [x] investigate changing checkpoint time to entire bitlevel instead of class
• [x] switch to new worktype once implemented by James George
• [x] tune default work fetch options around TF assignments
• [x] nudge results.txt to be results.json.txt (not sure if an error or doing this automatically is better), maybe see what p95/mprime case does
• [x] mfakto
• [ ] Iterative work fetch (Teal TODO)
• [ ] Workfetch status line at each iteration
• [x] TF bitrange in completion date messages
• [x] Additional line after setup wizard
• [x] Optparse default usage line
• [x] Default tf days of work = 1

Nice to have:

• [ ] TF1G integration
• [ ] nag user if they have StopAfterFactor=2 (suggest Stages=1 and StopAfterFactor=1)

Testing areas:

• [x] new setup default parameter experience
• [x] multiple bitlevel ini settings
• [x] other mfaktc .ini configuration combinations

Bugs:

• [x] "stall detection" if mfaktc not making progress since last checkin
• [x] work estimation for multiple bitlevel assignments (checkpoints only for single bitlevel)
• [x] don't send d=1 if exponent still in assignments
• [x] fix percentage calculation for Stages=1 and multiple bitlevel assignments

Server-side changes wishlist:

• [ ] have work preference 12 respect max and min arguments to ga
• [ ] have work preference 12 send highest ROI assignment in the 1m bins that each category is currently giving PRP out from.

Probably won't do:

• automatically generate and run TF worktodo lines for PRP or P-1 assignments that are underfactored (very long term stretch goal)
• create configuration options for TF work preferences that are passed to Manual_GPU_Assignment
• delete worktodo lines that fail to get AID via selfassign (current behavior just puts N/A, deleting instead of poaching (even if just an option) might be nice)
• GPU72 integration

[brubsby]

Ended up forking mfaktc to add json results that include the AID, will probably add default logging soon too, so we can parse the logs and use those for work estimation.

[brubsby]

--max-exp (and presumably --min-exp) aren't working with work preference 12.

connectionpool.py: [MainThread 2024-07-25 11:45:42,315]  DEBUG: http://v5.mersenne.org:80 "GET /v5server/?px=GIMPS&v=0.95&t=ga&g=[guid]=0&a=&max=138000000&ss=19191919&sh=ABCDABCDABCDABCDABCDABCDABCDABCD HTTP/11" 200 179
primenet.py: [MainThread 2024-07-25 11:45:42,316]  INFO: PrimeNet success code with additional info:
primenet.py: [MainThread 2024-07-25 11:45:42,316]  INFO: Server assigned trial factoring work.
primenet.py: [MainThread, Worker #1 2024-07-25 11:45:42,316]  INFO: Got assignment D5BBDEFA9AA42BBBC44F3AD5CFA8DA31: Trial factor M146204119
primenet.py: [MainThread, Worker #1 2024-07-25 11:45:42,316]  DEBUG: Fetching using v5 API
primenet.py: [MainThread, Worker #1 2024-07-25 11:45:42,316]  INFO: Getting assignment from server

We can maybe ask George or James to add support for these, as it looks correct from our side, but either way not extremely pressing. Just wanted to test them out since you mentioned them.

[brubsby]

Starting another spot for discussion here, as it's not really mfaktx related per se, but primenet related, and I feel I've saturated my outgoing communications to folks for the time being :)

For my TODO post above:

fix percentage calculation for Stages=1 and multiple bitlevel assignments This raises some question about what exactly we should report as STAGE and percentage in various cases. Let's go through an example of an assignment that is TF from 2^75 to 2^77.

If mfaktc.ini has Stages=0, then my understanding is that it's running every single bitrange in the assignment at once, so for example, if we find a 76.5 "bit" factor, we aren't sure that there isn't also 75.5 "bit" factor to be found in the range as well. So from this, reporting a simple TF75 stage seems a bit misleading, as the percentage associated with this stage is actually the joint percentage of TF75-77.

Similarly if mfaktc.ini has Stages=1, and we're reporting progress on TF from 2^75 to 2^77, we will report progress stage TF75 (and a percentage) during 75-76, and then TF76 (and a percentage) during 76-77. What exactly these percentages should be is unclear to me. If we are reporting 0-100% for each stage, then TF75 (and a percentage) is indistinguishable from the Stages=0 case of TF75 (and a percentage).

Estimated completion date, I would think, should just be when we expect to release the exponent, and not necessarily when the current stage is done. But I'm not sure if percentage should be that as well.

TL;DR: Should percentage be our progress on the current stage, or the progress of the whole assignment?

[brubsby]

As the number of changes in this PR keeps growing, merging upstream changes starts to get a bit more painful (I think it might've been better if I started with a rebase, but my tooling kept telling me a merge was a better idea), would you be open to merging this into the 2.2 branch at some point soon, and doing more changes as another PR? Or would you rather keep a version of the code with CERT stuff separate from the TF stuff, in case we want to release them independently?

[brubsby]

Here's the code I used to regenerate the merged table, on the chance it saves you any amount of manual effort:

import tables
tables.array(
    (
        (4,    "✔",  "✔",   "",   "",   "",  "✔"),
        (12,    "",   "",   "",   "",  "✔",   ""),
        (100,  "✔", "✔*",  "✔",  "✔",   "",   ""),
        (101,  "✔",   "",   "",  "✔",   "",   ""),
        (102,  "✔", "✔*",  "✔",  "✔",   "",   ""),
        (104,  "✔", "✔*",  "✔",  "✔",   "",   ""),
        (106,   "", "✔*",  "✔",  "",   "",   ""),
        (150,  "✔",  "✔",  "✔",   "",   "",   ""),
        (151,  "✔",  "✔",  "✔",   "",   "",   ""),
        (152,  "✔",  "✔",  "✔",   "",   "",   ""),
        (153,  "✔",  "✔",  "✔",   "",   "",   ""),
        (154,  "✔", "✔*",   "",   "",   "",   ""),
        (155,   "",  "✔",  "✔",   "",   "",   ""),
        (156,   "",   "",   "",   "",   "",   ""),
        (160,  "✔",   "",   "",   "",   "",   ""),
        (161,  "✔",   "",   "",   "",   "",   ""),
    ),
    ("Worktype", "Mlucas", "GpuOwl", "PRPLL", "CUDALucas", "mfaktc/mfakto", "CUDAPm1"),
    headerrow=True,
)

(I really wish the check glyph used had a monospaced glyph in most of the fonts used in editors and stuff, lol)

[tdulcet]

would you be open to merging this into the 2.2 branch at some point soon, and doing more changes as another PR?

Sorry about the hassle. Yes, we can merge this soon and then make any additional improvements in another PR. If we are going to support mfakto, it might be good to get that working with the script. For now, I will wait to push any more changes to the 2.2 branch until this is merged.

I really wish the check glyph used had a monospaced glyph in most of the fonts used in editors and stuff, lol

Yeah, I liked the checkmark symbol, but I agree that it is annoying that this table does not align in editors, so feel free to use different symbol. You could also change the appearance of the table by adjusting some of the options for my library.