Improve multiprocessing stubs

[hauntsaninja]

stubtest finds a number of issues with the existing multiprocessing stubs. PRs that work toward fixing these (if they need fixing) are welcome, even if they're small!

~/dev/typeshed master λ python3.8 -m mypy.stubtest --custom-typeshed-dir . multiprocessing --concise                     
multiprocessing.Array is not a type
multiprocessing.Event is inconsistent, runtime does not have argument "lock"
multiprocessing.JoinableQueue is not a type
multiprocessing.Queue is not a type
multiprocessing.RawArray is not present in stub
multiprocessing.RawValue is not present in stub
multiprocessing.SimpleQueue is not a type
multiprocessing.Value is not a type
multiprocessing.reducer is not present in stub
multiprocessing.set_executable is inconsistent, stub argument "exe" differs from runtime argument "executable"
multiprocessing.context.BaseContext.Event is inconsistent, runtime does not have argument "lock"
multiprocessing.dummy.Namespace.__getattr__ is not present at runtime
multiprocessing.dummy.Pipe is not present in stub
multiprocessing.managers.BaseManager.shutdown is not present at runtime
multiprocessing.managers.Namespace.__getattr__ is not present at runtime
multiprocessing.managers.Server.__init__ is inconsistent, stub does not have argument "registry"
multiprocessing.managers.Server.__init__ is inconsistent, stub does not have argument "address"
multiprocessing.managers.Server.__init__ is inconsistent, stub does not have argument "authkey"
multiprocessing.managers.Server.__init__ is inconsistent, stub does not have argument "serializer"
multiprocessing.managers.Server.accept_connection is not present in stub
multiprocessing.managers.Server.accepter is not present in stub
multiprocessing.managers.Server.create is not present in stub
multiprocessing.managers.Server.debug_info is not present in stub
multiprocessing.managers.Server.decref is not present in stub
multiprocessing.managers.Server.dummy is not present in stub
multiprocessing.managers.Server.fallback_getvalue is not present in stub
multiprocessing.managers.Server.fallback_mapping is not present in stub
multiprocessing.managers.Server.fallback_repr is not present in stub
multiprocessing.managers.Server.fallback_str is not present in stub
multiprocessing.managers.Server.get_methods is not present in stub
multiprocessing.managers.Server.handle_request is not present in stub
multiprocessing.managers.Server.incref is not present in stub
multiprocessing.managers.Server.number_of_objects is not present in stub
multiprocessing.managers.Server.public is not present in stub
multiprocessing.managers.Server.serve_client is not present in stub
multiprocessing.managers.Server.shutdown is not present in stub
multiprocessing.managers.SharedMemoryServer.__init__ is inconsistent, stub does not have *args argument "args"
multiprocessing.managers.SharedMemoryServer.create is not present in stub
multiprocessing.managers.SharedMemoryServer.list_segments is not present in stub
multiprocessing.managers.SharedMemoryServer.public is not present in stub
multiprocessing.managers.SharedMemoryServer.release_segment is not present in stub
multiprocessing.managers.SharedMemoryServer.shutdown is not present in stub
multiprocessing.managers.SharedMemoryServer.track_segment is not present in stub
multiprocessing.managers.SyncManager.Barrier is not present in stub
multiprocessing.managers.SyncManager.Event is inconsistent, stub does not have *args argument "args"
multiprocessing.managers.SyncManager.JoinableQueue is not present in stub
multiprocessing.managers.SyncManager.Lock is inconsistent, stub does not have *args argument "args"
multiprocessing.managers.SyncManager.Namespace is inconsistent, stub does not have *args argument "args"
multiprocessing.managers.SyncManager.Pool is not present in stub
multiprocessing.managers.SyncManager.RLock is inconsistent, stub does not have *args argument "args"
multiprocessing.pool.ApplyResult.__init__ is inconsistent, stub does not have argument "pool"
multiprocessing.pool.ApplyResult.__init__ is inconsistent, stub does not have argument "callback"
multiprocessing.pool.ApplyResult.__init__ is inconsistent, stub does not have argument "error_callback"
multiprocessing.pool.CLOSE variable differs from runtime type Literal['CLOSE']
multiprocessing.pool.IMapIterator.__init__ is inconsistent, stub does not have argument "pool"
multiprocessing.pool.MapResult.__init__ is inconsistent, stub does not have argument "pool"
multiprocessing.pool.MapResult.__init__ is inconsistent, stub does not have argument "chunksize"
multiprocessing.pool.MapResult.__init__ is inconsistent, stub does not have argument "length"
multiprocessing.pool.MapResult.__init__ is inconsistent, stub does not have argument "callback"
multiprocessing.pool.MapResult.__init__ is inconsistent, stub does not have argument "error_callback"
multiprocessing.pool.Pool.Process is not present in stub
multiprocessing.pool.Pool.imap is inconsistent, stub argument "iterable" has a default value but runtime argument does not
multiprocessing.pool.Pool.imap_unordered is inconsistent, stub argument "iterable" has a default value but runtime argument does not
multiprocessing.pool.Pool.map is inconsistent, stub argument "iterable" has a default value but runtime argument does not
multiprocessing.pool.Pool.map_async is inconsistent, stub argument "iterable" has a default value but runtime argument does not
multiprocessing.pool.Pool.starmap is inconsistent, stub argument "iterable" has a default value but runtime argument does not
multiprocessing.pool.Pool.starmap_async is inconsistent, stub argument "iterable" has a default value but runtime argument does not
multiprocessing.pool.RUN variable differs from runtime type Literal['RUN']
multiprocessing.pool.TERMINATE variable differs from runtime type Literal['TERMINATE']
multiprocessing.pool.ThreadPool.Process is not present in stub
multiprocessing.queues.JoinableQueue.__init__ is inconsistent, stub argument "ctx" has a default value but runtime argument does not
multiprocessing.queues.Queue.__init__ is inconsistent, stub argument "ctx" has a default value but runtime argument does not
multiprocessing.queues.Queue.put_nowait is inconsistent, stub argument "item" differs from runtime argument "obj"
multiprocessing.queues.SimpleQueue.__init__ is inconsistent, stub argument "ctx" has a default value but runtime argument does not
multiprocessing.queues.SimpleQueue.put is inconsistent, stub argument "item" differs from runtime argument "obj"
multiprocessing.spawn._main is inconsistent, stub does not have argument "parent_sentinel"
multiprocessing.synchronize.Barrier.__init__ is inconsistent, runtime does not have *args argument "ctx"
multiprocessing.synchronize.Barrier.__init__ is inconsistent, stub does not have argument "ctx"
multiprocessing.synchronize.Condition.acquire is not present at runtime
multiprocessing.synchronize.Condition.release is not present at runtime
multiprocessing.synchronize.Event.__init__ is inconsistent, runtime does not have argument "lock"
multiprocessing.synchronize.SemLock.__init__ is inconsistent, stub does not have argument "kind"
multiprocessing.synchronize.SemLock.__init__ is inconsistent, stub does not have argument "value"
multiprocessing.synchronize.SemLock.__init__ is inconsistent, stub does not have argument "maxvalue"
multiprocessing.synchronize.SemLock.__init__ is inconsistent, stub does not have argument "ctx"
multiprocessing.synchronize.SemLock.acquire is not present at runtime
multiprocessing.synchronize.SemLock.release is not present at runtime
multiprocessing.synchronize.Semaphore.get_value is not present in stub

[hauntsaninja]

I removed the good-first-issue label since some of these are fairly complicated to fix "right", see discussion in #4282

However, there are likely some easy fixes in here as well that do make good first issues! I'd recommend taking a look at any of the ones complaining about missing or differing arguments :-)

[Dr-Irv]

With respect to multiprocessing.Event, the python docs define that as a class, but it is a method. So if you try

from multiprocessing import Event

myevent: Event = Event()

you get a mypy error, because multiprocessing.Event is a function, not a class. If you leave out the type declaration, it is valid code.

Not sure if that means the typing in typeshed has to fix that, or if it is a documentation bug on the python side.

[srittau]

typeshed tries to follow the implementation, not the documentation. In this case the documentation is at least misleading.

[Dr-Irv]

typeshed tries to follow the implementation, not the documentation. In this case the documentation is at least misleading.

So where do we report the documentation issue?

[Akuli]

See the conversation on #4313

Technically you could report this to bugs.python.org, but Python core devs are already aware of this. See https://bugs.python.org/issue19895 for example.

[Dr-Irv]

@Akuli thanks for the pointers. We can work around it by doing

from multiprocessing import Event
from multiprocessing.synchronize import Event as EventClass
myevent:  EventClass = Event()

[srittau]

Maybe re-exporting the "proper" classes from multiprocessing as _EventType, _QueueType etc. is the best solution.

[Akuli]

Currently typeshed just says that Queue is a class, even though it isn't, so you can write myqueue: Queue = Queue() for example. As can be seen from #4313 etc, changing this now is probably not a good idea.

But for some reason, Event and a few other things are functions in typeshed. Maybe we should make them look like classes as well, so that at least it's consistent, even if it lies?

[srittau]

We should not change those if they are already correct, and we should work towards fixing Queue as well. Lies in typeshed always cause more problems than they solve, in my experience.

[zplizzi]

Is it possible to type a multiprocessing.Value? I can't figure out how without making mypy fail.

[hauntsaninja]

There isn't a great way; how best to type it depends on how exactly you're using it. Also willing to bet that the stubs could be improved (e.g. looks like the lock to Value should have a default value on the Literal[True] overload)

Is the code you're working on open source? That would help us, e.g. we could add it to https://github.com/hauntsaninja/mypy_primer

[zplizzi]

I'm not doing anything too special with it, here's a simple example that I don't know how to properly type (as written mypy will complain):

import multiprocessing

def target(counter: multiprocessing.Value):
    counter += 1

if __name__ == "__main__":
    from ctypes import c_int

    counter = multiprocessing.Value(c_int)

    ps = [multiprocessing.Process(target=target, args=(counter,)) for _ in range(5)]
    for p in ps:
        p.start()
    for p in ps:
        p.join()

    print(counter)

Although in my actual code I'm using lock=False. It's not open source unfortunately.

[hauntsaninja]

The precise type is probably multiprocessing.sharedctypes.SynchronizedBase[ctypes.c_int]

I also made the change I suggested in #8330, which means you'll get a much more precise return type. Right now in your snippet, counter just ends up as Any.

[madig]

I think I found a blind spot? I upgraded mypy recently and now I get the following:

import ctypes
from multiprocessing import Array

a = Array(ctypes.c_char, 32)
with a.get_lock():
    print(a.value)

> mypy a.py
a.py:6: error: "SynchronizedArray[c_char]" has no attribute "value"  [attr-defined]
Found 1 error in 1 file (checked 1 source file)

Where the docs explicitly say "that an array of ctypes.c_char has value and raw attributes which allow one to use it to store and retrieve strings." (https://docs.python.org/3/library/multiprocessing.html?highlight=array#multiprocessing.Array)

[prauscher]

For @madig s problem it looks like SynchronizedString is lost in Array-Constructor: While https://github.com/python/typeshed/blob/main/stdlib/multiprocessing/sharedctypes.pyi#L68 returns a SynchronizedString, none of the Array specifications will return SynchronizedString.

The same seems to be true in https://github.com/python/typeshed/blob/main/stdlib/multiprocessing/sharedctypes.pyi#L66, so that

a = Value(ctypes.c_float, 0.0)
a.value = 1.0  # error: Incompatible types in assignment (expression has type "float", variable has type "c_float") [assignment]

I think both should be quite easy to fix, but the overloads of Value and Array keep getting bigger and bigger...

[prauscher]

See #10319, I think that is the same issue