alex-petrenko / faster-fifo

Faster alternative to Python's multiprocessing.Queue (IPC FIFO queue)
MIT License
179 stars 29 forks source link

Does it support Windows OS? #34

Open Sniper199999 opened 2 years ago

Sniper199999 commented 2 years ago

I tried to install this package on my Windows 10 machine. Got this following error:

PS C:\Users\Work\Desktop\opencv> pip install faster-fifo
Collecting faster-fifo
  Installing build dependencies ... done
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      Compiling faster_fifo.pyx because it depends on C:\Users\Work\AppData\Local\Temp\pip-build-env-lfv2321j\overlay\Lib\site-packages\Cython\Includes\libcpp\__init__.pxd.
      [1/1] Cythonizing faster_fifo.pyx
      Traceback (most recent call last):
        File "C:\users\Work\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 363, in <module>
          main()
        File "C:\users\Work\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "C:\users\Work\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "C:\Users\Work\AppData\Local\Temp\pip-build-env-lfv2321j\overlay\Lib\site-packages\setuptools\build_meta.py", line 177, in get_requires_for_build_wheel
          return self._get_build_requires(
        File "C:\Users\Work\AppData\Local\Temp\pip-build-env-lfv2321j\overlay\Lib\site-packages\setuptools\build_meta.py", line 159, in _get_build_requires
          self.run_setup()
        File "C:\Users\Work\AppData\Local\Temp\pip-build-env-lfv2321j\overlay\Lib\site-packages\setuptools\build_meta.py", line 174, in run_setup
          exec(code, locals())
        File "<string>", line 39, in <module>
        File "C:\Users\Work\AppData\Local\Temp\pip-build-env-lfv2321j\overlay\Lib\site-packages\setuptools\discovery.py", line 103, in find
          convert_path(str(where)),
        File "C:\Users\Work\AppData\Local\Temp\pip-build-env-lfv2321j\overlay\Lib\site-packages\setuptools\_distutils\util.py", line 140, in convert_path
          raise ValueError("path '%s' cannot end with '/'" % pathname)
      ValueError: path './' cannot end with '/'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

Just noticed that Linux and Mac versions are supported, but did not see any mentions of Windows not being supported.

alex-petrenko commented 2 years ago

C++ code of the queue uses POSIX interprocess synchronization primitives, therefore it will only work on a POSIX system. I think you can make it work in a MinGW/Cygwin environment on Windows, or in Windows Subsystem for Linux (WSL). But not in a vanilla Windows environment.

I believe Windows has similar primitives to pthread mutexes as a part of WinAPI (not an expert), so most likely Windows support can be added using some preprocessor conditioned branches. It is a very welcome contribution! :)

Requirements section mentioned supported OSes, perhaps should add an explicit note that Windows is not currently supported. https://github.com/alex-petrenko/faster-fifo#requirements

alex-petrenko commented 2 years ago

Actually Windows implementation can be easier than I thought: https://stackoverflow.com/questions/6672539/linux-pthread-portability-on-windows http://sourceware.org/pthreads-win32/ - looks promising. I never actually looked closely due to lack of time.

alex-petrenko commented 2 years ago

Also found this: https://github.com/marklakata/mqueue-w32

Sniper199999 commented 2 years ago

That is good to hear that it could possibly be implemented on Windows. For now, I have my code working without using queue in Processes, but I still think the queue implementation will be faster.

I looked through the links, and a lot of stuff is new to me. Tell me if I am wrong: Windows is not POSIX sysystem, in order to get POSIX pthreads working on windows I need MinGW64 which supports gcc compiler. Then I need to install pthreads-win32 or https://sourceforge.net/projects/pthreads4w/ in MinGW for getting the POSIX compatibilities in Windows. Then it should work just fine?

alex-petrenko commented 2 years ago

@Sniper199999 I know very little about IPC in Windows. I am not at all confident that pthread-win32 has full support for all POSIX features, such as inter-process mutexes (in fact, looks like it doesn't). I am confident enough that in principle Windows implementation is possible, although it might require using some Win32 API.

At this time Windows implementation is beyond both my expertise and capacity, hope you understand :) But I love to see interest and attention to this repository. If you'd be willing to work on Windows support, I'd be glad to help!

If you're looking at a simple plug-and-play solution that would make your Windows app faster, this is probably not it. Moreover, I'd generally look for performance outside of Python world (unless you absolutely have to use Python). I also think that cross-platform support goes a super long way towards popularity of software and I can't encourage it enough if you'd like to work on that! :)