mypy vs os x vs no sched_getaffinity

[kemsguy7]

Describe the bug When I ran the make test command, I got the error below.

process_worker_pool.py:643: error: Module has no attribute "sched_getaffinity" [attr-defined] parsl/executors/high_throughput/process_worker_pool.py:683: error: Module has no attribute "sched_setaffinity" [attr-defined] parsl/tests/test_python_apps/test_lifted.py:28: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] parsl/tests/test_python_apps/test_lifted.py:29: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]

To Reproduce On MacOs, after creating a virtual environment and running the make test command from the instructions on the readme file, run the make test command.

1. Follow the installation steps from the readme file
2. After creating and activating the virtual environment, run make test inside the virtual environment
3. See error

Expected behavior You'll get the error sched_getaffinity" [attr-defined] declared above.

Environment

• OS: MacOS
• Python version : 3.12
• Parsl version

Distributed Environment

• MacBook Air (13-inch, 2017)

[kemsguy7]

@benclifford At your request from issue #3105, I have created this note. Please review it and let me know if any further action is needed.

[azharcodeit]

I am also seeing this exact error on macos, seems like it is not available on macOS because macOS uses a different process scheduling model compared to Linux

[kemsguy7]

I am also seeing this exact error on macos, seems like it is not available on macOS because macOS uses a different process scheduling model compared to Linux

Well, to resolve this, i ran the smaller test suite : make config_local_test and i was able to get past that, you could try out too

[benclifford]

Tagging @WardLT because he wrote this code originally and might be interested.

As @azharcodeit points out, these methods aren't available on OS X and mypy is getting upset about that.

@WardLT even wrote in some other tests a check to make those other tests be skipped on platforms which don't have scheduler affinity - see parsl/tests/sites/test_affinity.py has this line in it:

@pytest.mark.skipif('sched_getaffinity' not in dir(os), reason='System does not support sched_setaffinity')

mypy is performing a different kind of analysis of the code, and the error that it is reporting here is something like:

"If a user tries to run this code path, they are going to get an error that these sched_get/set_affinity methods do not exist."

Luckily for most users, that code path is only reached if the user has set a process affinity option, because every call to sched_setaffinity or sched_getaffinity is inside this big if statement:

   # If desired, set process affinity
    if cpu_affinity != "none":

So you can interpret the mypy error message a bit more as: if a user on OS X tries to set the cpu_affinity configuration option, then this code will break.

[benclifford]

This is quite an interesting bug to untangle, but I don't expect someone from Outreachy to do it - so for now, we have to accept that the mypy test (which is run as part of the make test tests) will not pass on OS X. You can see from other Outreachy applications (@kemsguy7 above, for example) that there are ways to run just a smaller subset of the tests.