Newest version of BOINC 7.20.7 slows down processing of tasks

[Noah-Yarmush]

After upgrading BOINC to the v 7.20.7 I noticed that tasks are processed slower. In my case it is about 5% but it's not only an E@H problem. Currently from time to time I also do tasks for Denis@Home and situations there is similar. I didn't change my local settings nor account settings for these projects, however increasing settings isn't help this situation. Problem was also reported by another user on this thread of Einstein@Home project.

My computer:

MacBook M1 8GB RAM, SSD 256GB, Mac OS Ventura 13.4 (updated few days ago) Laptop bought over a year ago and turned on most of the time. However, I'm a volunteer from 2009 and there's no problem on Win machines in past years (currently I don't have one to verify if the difference is present there).

I use local settings:

12.5% of cores (1 core) / 90%

I also do tasks for World Community Grid but unfortunately, now I cannot check it because I don't have long tasks, for example ARP, which normally lasts ~6 hours on my computer.

[AenBleidd]

@CharlieFenton, could you please take a look?

[CharlieFenton]

Apple broke an API [per-thread locale uselocale()] in MacOS 13.3. The workaround requires switching locale on every RPC call which the client uses to communicate with the BOINC Manager or screensaver. This is the only change from 7.20.4 that might slow things down.

Do you still get the slowdown if you close (hide) the BOINC Manager by clicking on the red button at the top left of the window? For fastest processing of project tasks, keep Manage windows closed and avoid running the screensaver; this has always been true.

[Noah-Yarmush]

I never tested processing tasks with closed client (always opened in adv. mode) so I need to check it first for a day or so. I've also never used screensavers (turning off display instead). BOINC screensaver is turned off in system settings.

Apple broke an API [per-thread locale uselocale()] in MacOS 13.3.

So the problem is in switching between threads processing specific tasks when client needs to determine region and language? Or is the switching performed constantly but it immediately returns to processing a single task? However, there was no problem before the last update of BOINC. For few days in a row laptop processed for only E@H tasks which normally takes 2h5m per task without switching between tasks (task A started and after 2h5m when finished another task is taken). Now the difference is about 5 minutes in E@H and dozen/dozens of seconds in Denis@Home (about 5% longer). I'll try to check it for WCG when they'll send new tasks for ARP.

[Noah-Yarmush]

Info from another user:

the slowdown persisted despite closing/hiding BOINC Manager into the background...

[CharlieFenton]

the slowdown persisted despite closing/hiding BOINC Manager into the background...

In that case I have no idea why this would happen. There were no changes to the client from 7.20.4 to 7.20.7, only to the Manager and installer.

Are you sure you are lnot comparing different applications from Einstein? On my M1 iMac I have both Binary Radio Pulsar Search (Arecibo,GBT,long) 1.60 and Multi-Directional Gravitational Wave search on O3 (CPU) 1.03 (GW-SSE2).

Have you tried reverting to 7.20.4 to see if the current Einstein tasks run faster with the older version of BOINC? In other words, to make sure that Einstein didn't changed their tasks at the same time by coincidence? You can download BOINC 7.20.4 for the mac here.

In any case, it is not uncommon for different tasks for the same application to take different amounts of time if the data they are crunching is different, though I don't know whether that variation occurs for Einstein in particular.

[Noah-Yarmush]

I haven't checked downgrading the client but the other person did:

Has anyone noticed the recent release of BOINC 7.20.7 actually slows down E@H crunching...? My crunch time increased from 2.75hrs to 3.5hrs, while CPU temp decreased from 60 to 50 degrees C... Restart didn't do the trick...

After downgrading to 7.20.2 crunch time went back to the normal 2.75hrs...

MacBook Air M1 here...

I'll check it tomorrow personally.

[CharlieFenton]

So the problem is in switching between threads processing specific tasks when client needs to determine region and language?

The locale used for most of the BOINC Manager is your selected natural language, such as English or French. Because decimal points in numbers differ (for example the U.S. uses a period: 123.45 while most of the world uses a comma: 123,45), the Manager must use the "C" locale to ensure using a decimal point while communicating with the client. (The client always uses the "C" locale.) The RPCs used to communicate use a separate thread in the Manager so as not to slow down the responsiveness of the Manager.

Before MacOS 13.3, the RPC thread called uselocale() only once when initializing the RPC thread and that thread always just used the "C" locale. When Apple broke the uselocale() API, we were forced to switch back and forth for every RPC. The Manager uses many RPCs every second to communicate with the client and update its display.

[davidpanderson]

What is the CPU % used by the Manager before and after this change?

[Noah-Yarmush]

I described it in the start post. It was the same previously. The same situation was in case of first person who observed this behaviour on different laptop for different settings. He mentioned that after reversal, things are ok. I'll back when I'll check times for the older version.

[davidpanderson]

When you look at the CPU usage in the Activity Monitor (I think what's what it's called on Mac) 1) how much CPU is the BOINC client using? 2) how much CPU is the Manager using? There should both be 2% or less.

Charlie: is it possible they were compiled with -g or similar?

[Noah-Yarmush]

Reverting to version 7.20.2 fixed the issue.

how much CPU is the BOINC client using?

below 2% (peaks about 1.5, 1.9, normal about 0.4)

Now I'm testing version 7.20.7 again to see if the problem will back. CPU usage for the client is still about 0.4. I need over next two hours to be sure.

how much CPU is the Manager using? There should both be 2% or less.

About 6% but there's no matter if I run CAD, Safari with many tabs, Pycharm or other "high CPU" programs. My BOINC settings pause the client when CPU usage is > 25%. However, it never increased the elapsed and CPU time of tasks.

[davidpanderson]

6% is too high. Charlie, can you see what it's doing? Was there a WxWidgets change?

[CharlieFenton]

Charlie: is it possible they were compiled with -g or similar?

@davidpanderson Looking at the Xcode projects for the 2 builds, I don't see anything that would affect the compiler or linker options used for building the Manager or the client. However, 7.20.7 was built with a slightly newer version of Xcode (14.3) than 7.20.4. (Xcode 14.1.) Xcode 14.3 has a newer version of the compiler. Surprisingly, BOINC Manager 7.20.7(21.7 MB) is smaller than BOINC Manager 7.20.4 (23.6 MB), but BOINC client 7.20.7 (10,013,744 bytes) is slightly larger than BOINC client 7.20.4 (9,983,920 bytes.)

By comparison, BOINC Manager 7.22.2 is 23.8MB and BOINC Client 7.22.2 is 10,001,328 bytes. It was also built with Xcode 14.3. While one would hope a newer compiler would not create less efficient code, it's possible that was true this time.

In any case, since project applications are separate executables run as separate processes, shouldn't the total CPU time be independent of what the client and Manager do? If we are talking about clock (wall) time, might a difference in how throttling is done or how switching among tasks cause the appearance of slower processing? Still, there weren't any code changes to the client from 7.20.4 to 7.20.7.

@Noah-Yarmush When you say tasks are processed slower, do you mean they take more total CPU time or more wall clock (total elapsed) time? (Note that the "Elapsed" time column in BOINC Manager is a different value; it counts only the time while that task is running, not the total difference in the time of day from when it started until it finished.)

There is an option in BOINC's Computing Preferences "Switch between tasks every xx minutes." BOINC stops working on a task and switches to processing a different task that often. Is it possible that the clock time for any one task takes longer, but when you average the time for many tasks it works out the same for 7.20.4 and 7.20.7. Might it be that BOINC is switching among more tasks in a day than before? Note also the "Use at most xx% of CPU time" which is intended to allow you to keep your computer cooler by "throttling" the computation. For example, if you set it to 75% tasks will repeatedly run of 3 seconds and pause for 1 second.

These are all things which can affect the total clock time, though they should theoretically have the same affect in BOINC 7.20.7 as BOINC 7.20.4.

[CharlieFenton]

6% is too high. Charlie, can you see what it's doing? Was there a WxWidgets change?

On my M1 iMac, I see a very occasional high of 2.0% but it usually varies between 0.4% and 1.7%. Both 7.20.4 and 7.20.7 used wxWidgets 3.1.5. When you move the mouse inside a BOINC Manager window, it goes higher, as it always has, but drops again when the mouse stops moving.

@Noah-Yarmush, could you try downloading BOINC 7.22.2 here and see if that has any effect?

[Noah-Yarmush]

@CharlieFenton

Test on version 7.20.7 slowed processing again, not only run time. Here are results from E@H site:

Configuration of my BOINC is set in the way that there's no switching between tasks, so in cases where deadlines aren't close, each task is processed at once. I have the Switch between tasks every ... minutes set to 15 minutes but it has no effect and I don't remember if any values worked on Mac. I'll check the newest version which you've linked and'll back about 3hrs.

@davidpanderson

I saw on commits that there's a new version of widgets using in the client (at least paths are new). I also noticed that items on client's grid controls stopped blinking (however sometimes they are still bad rendered). Hiding client's window or inactive tasks doesn't help.

[Noah-Yarmush]

@CharlieFenton

Version 7.22.2 did task quicker but it still lasted (CPU time) about 5min longer than average for this project. I need four additional hours to verify it for two other tasks.

Additional informations from the other user:

Use at most 45% of the CPU Use at most 50% of CPU time Effectively I am crunching at only 3 out of 8 cores at 50% capacity all the time. And I'm only processing E@H WUs, no other projects at play so there's no switching amongst tasks.

also:

why can't BOINC be set to crunch at a steady half capacity instead of running 1 sec and pausing 1 sec in my CPU case usage...? Won't this pausing-and-running scenario actually shortens CPU lifespan...?

[Noah-Yarmush]

Still no improvement in v7.22.2. Last three tasks longer than usual: two of them over 5min and one over 15. It's not normal situation. In my case they should lasts ~2h5m of CPU usage with dozen seconds of standard deviation (measured last year for over 1200 samples). There's no doubt that something had changed between version 7.20.2 and 7.20.7 in BOINC's code or 3rd party libraries. I'm going back to the 7.20.2 version. If you need more information just let me know what I can test in newest installers.

[CharlieFenton]

@Noah-Yarmush I would like you to determine if the speed difference might be due to the difference in Xcode versions I used to build BOINC 7.20.4 (Xcode 14.1) and BOINC 7.20.7 (Xcode 14.3). I have rebuilt BOINC 7.20.4 using the same source files as before but building it with the latest Xcode version 14.3, I've uploaded this test build of BOINC 7.20.4 to the server. Please download it here and let us know if it also exhibits the slower run times. Thanks in advance for your help in figuring this out.

This build of BOINC 7.20.4 using Xcode 14.3 has a BOINC Manager size of 21.7MB and BOINC client size of 10,014,704 bytes. (As I wrote before, the original Xcode 14.1 build of BOINC 7.20.4 had sizes of 23.6 MB and 9,983,920 bytes.

[Noah-Yarmush]

@CharlieFenton I've just checked the test build for two more tasks and there's no problem - both finished after 2h3m. Currently a task for Rosetta@Home is processed so I'll check the result after 8hrs.

[CharlieFenton]

@Noah-Yarmush Thanks for checking. On one hand I’m relieved that the new version of Xcode doesn’t matter. On the other hand, I’m even more puzzled at what is causing this.

[CharlieFenton]

I’ll be out of town for the next two weeks but will keep thinking about it.

[Noah-Yarmush]

Longer task are also ok so client v7.20.4 doesn't have change which slows down computing.

[Noah-Yarmush]

For a few days I again tested v7.20.7 and I observed such things:

• longer (8h) tasks for Rosetta done in the same time as before update,
• tasks for E@H seems to returning to normal BUT the situation is interesting here:

For example, I performed two E@H tasks in a row for 70% of CPU time and after that I set back 90% of CPU time. But surprisingly, the change wasn't immediate. It seems that the impact of first two tasks is "stretched" out to processing other tasks (time factor is constantly decreasing both for run and cpu time and every next task is convergent to before-update mean).

I read on the forum that something has been changed with throttling mechanism. Can it be linked with? I didn't perform such test for other "longer" projects so unfortunately I cannot prove that the problem isn't E@H-specific.