Closed visr closed 2 months ago
@visr, thanks for opening the issue -
@savingoyal re:
Windows support would be a complex undertaking (given our liberal use of multiprocessing for starters). We are definitely open to contributions and a good way to kick start the conversation would be to do a POC.
FWIW, all data scientists may not be running a *nix Linux or mac. Having official support for Windows would be a big plus. I wish I could coerce them though but that's a dream. (And FWIW, WSL or Cygwin are their own can of frankenstein worms)
Multiprocessing may not be the main issue IMHO but rather:
keeping Python 2 and 3 compatibility is going to be a major pain on Windows
using signals such as here https://github.com/Netflix/metaflow/blob/c2f70c73782549fdce6c418410ecd9b305199b28/metaflow/plugins/timeout_decorator.py
possibly some assumptions about calling some functions and file system semantics that do not apply or are not available on Windows
For 2. Windows is a pain but I have some POSIX/windows utilities that have been battle tested for timeouts here https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/interrupt.py
For 1. we did that in https://github.com/nexB/scancode-toolkit and I would never do it again which would mean to support Python 3 for Windows. Anything else is a sure way to destroy grey matter very quickly ;)
And what about news about Windows supporting Linux subsystem?
My latest thinking is that Windows support is too mucho pita and with the new WSL2, running on Windows means running on Linux, which is a sane thing. I would close this once some doc is added that WSL2 must be used
I was hoping that a full remote k8s execution of a flow would work on windows but it doesn't. I stubbed out fcntl but hit this polling dependency. I am in a corporate setup so windows is unfortunately my client. I could develop flows locally in docker but it would be nice to be able to execute the flow in k8s from windows.
$ python 00-helloworld/helloworld.py run --with kubernetes:service_account=auser
Metaflow 2.5.2.post7+git6b07668 executing HelloFlow for user:U038178
Validating your flow...
The graph looks good!
Running pylint...
Pylint is happy!
Polling is not supported on your operating system (Windows)
2022-03-17 13:23:24.242 Workflow starting (run-id 1647523400851411):
2022-03-17 13:23:24.628 [1647523400851411/start/1 (pid 34040)] Task is starting.
2022-03-17 13:23:24.629 Workflow failed.
2022-03-17 13:23:24.630 Terminating 1 active tasks...
2022-03-17 13:23:24.631 [1647523400851411/start/1 (pid 34040)] [KILLED BY ORCHESTRATOR]
2022-03-17 13:23:24.632 [1647523400851411/start/1 (pid 34040)] [KILLED BY ORCHESTRATOR]
2022-03-17 13:23:29.000 Killing 1 remaining tasks after having waited for 5 seconds -- some tasks may not exit clean
2022-03-17 13:23:29.002 Flushing logs...
Internal error
Traceback (most recent call last):
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 209, in execute
self._launch_workers()
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 564, in _launch_workers
self._launch_worker(task)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 584, in _launch_worker
self._poll.add(fd)
AttributeError: 'NoneType' object has no attribute 'add'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 1102, in main
start(auto_envvar_prefix="METAFLOW", obj=state)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 829, in __call__
return self.main(args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 610, in invoke
return callback(args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 666, in wrapper
return func(args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\decorators.py", line 33, in new_func
return f(get_current_context().obj, args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 789, in run
runtime.execute()
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 230, in execute
self._killall()
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 275, in _killall
list(self._poll_workers())
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 521, in _poll_workers
for event in self._poll.poll(PROGRESS_INTERVAL):
AttributeError: 'NoneType' object has no attribute 'poll'
I see Argo support might be forthcoming #50 would this be an option for starting flows in k8s from windows or is windows as a client completely out?
Also its not clear to me how the code is packaged? I see decorators for conda & image and reference to the code being tarball'd to S3. Can you explain or point me to the docs that explain the code packaging for k8s. Thanks.
@erowan Would you like to try out #992 in your windows environment? Here is the code that packages up your flow.
Metaflow works well in WSL2 - closing this issue since that seems like the most reasonable way forward. Many modern ML packages don't play well with windows and have to rely on WSL2, so even if Metaflow somehow worked well within Windows (but outside of WSL2), it's utility will be somewhat limited.
Sounds fine to me. I'm not currently using metaflow, but the reason I opened the issue because it wasn't documented that it doesn't work on Windows. Is this now documented? I had a quick look and I don't see anything in https://docs.metaflow.org/getting-started/install, though for R it is: https://docs.metaflow.org/v/r/getting-started/install#windows-support.
Thanks for open sourcing this library. I was quite excited to take it for a spin, only to get an error "no module named 'fcntl'", and learning through #10, #23 and #46 that Windows is not supported, and there are no active plans for Windows support.
That is of course fine, but I have a few related questions.
I see #10 has a
wontfix
and #46 has ahelp wanted
label. That begs the question; would you be open to accepting contributions that add Windows support?Do you know what the major technical obstacles to Windows support are?
Just now I see on the Installing Metaflow page "Metaflow is available as a Python package for MacOS and Linux." Perhaps if it was followed by a more explicit "Windows is not supported.", fewer people would miss this.
Would it be suitable to place this on the roadmap, perhaps stating that there are no Netflix plans but outside contributions are welcome?