Windows support - Githubissues

visr commented 4 years ago

Thanks for open sourcing this library. I was quite excited to take it for a spin, only to get an error "no module named 'fcntl'", and learning through #10, #23 and #46 that Windows is not supported, and there are no active plans for Windows support.

That is of course fine, but I have a few related questions.

I see #10 has a wontfix and #46 has a help wanted label. That begs the question; would you be open to accepting contributions that add Windows support?
Do you know what the major technical obstacles to Windows support are?
Just now I see on the Installing Metaflow page "Metaflow is available as a Python package for MacOS and Linux." Perhaps if it was followed by a more explicit "Windows is not supported.", fewer people would miss this.
Would it be suitable to place this on the roadmap, perhaps stating that there are no Netflix plans but outside contributions are welcome?

savingoyal commented 4 years ago

@visr, thanks for opening the issue -

Windows support would be a complex undertaking (given our liberal use of multiprocessing for starters). We are definitely open to contributions and a good way to kick start the conversation would be to do a POC.
We are updating our contributing guidelines to reflect this. Expect some progress on that front in the next couple of days.

pombredanne commented 4 years ago

@savingoyal re:

Windows support would be a complex undertaking (given our liberal use of multiprocessing for starters). We are definitely open to contributions and a good way to kick start the conversation would be to do a POC.

FWIW, all data scientists may not be running a *nix Linux or mac. Having official support for Windows would be a big plus. I wish I could coerce them though but that's a dream. (And FWIW, WSL or Cygwin are their own can of frankenstein worms)

Multiprocessing may not be the main issue IMHO but rather:

keeping Python 2 and 3 compatibility is going to be a major pain on Windows
using signals such as here https://github.com/Netflix/metaflow/blob/c2f70c73782549fdce6c418410ecd9b305199b28/metaflow/plugins/timeout_decorator.py
possibly some assumptions about calling some functions and file system semantics that do not apply or are not available on Windows

For 2. Windows is a pain but I have some POSIX/windows utilities that have been battle tested for timeouts here https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/interrupt.py

For 1. we did that in https://github.com/nexB/scancode-toolkit and I would never do it again which would mean to support Python 3 for Windows. Anything else is a sure way to destroy grey matter very quickly ;)

pablodz commented 4 years ago

And what about news about Windows supporting Linux subsystem?

pombredanne commented 4 years ago

My latest thinking is that Windows support is too mucho pita and with the new WSL2, running on Windows means running on Linux, which is a sane thing. I would close this once some doc is added that WSL2 must be used

erowan commented 2 years ago

I was hoping that a full remote k8s execution of a flow would work on windows but it doesn't. I stubbed out fcntl but hit this polling dependency. I am in a corporate setup so windows is unfortunately my client. I could develop flows locally in docker but it would be nice to be able to execute the flow in k8s from windows.

$ python 00-helloworld/helloworld.py run --with kubernetes:service_account=auser                                
Metaflow 2.5.2.post7+git6b07668 executing HelloFlow for user:U038178                                                
Validating your flow...                                                                                             
    The graph looks good!                                                                                           
Running pylint...                                                                                                   
    Pylint is happy!                                                                                                
Polling is not supported on your operating system (Windows)                                                         
2022-03-17 13:23:24.242 Workflow starting (run-id 1647523400851411):                                                
2022-03-17 13:23:24.628 [1647523400851411/start/1 (pid 34040)] Task is starting.                                    
2022-03-17 13:23:24.629 Workflow failed.                                                                            
2022-03-17 13:23:24.630 Terminating 1 active tasks...                                                               
2022-03-17 13:23:24.631 [1647523400851411/start/1 (pid 34040)] [KILLED BY ORCHESTRATOR]                             
2022-03-17 13:23:24.632 [1647523400851411/start/1 (pid 34040)] [KILLED BY ORCHESTRATOR]                             
2022-03-17 13:23:29.000 Killing 1 remaining tasks after having waited for 5 seconds -- some tasks may not exit clean
2022-03-17 13:23:29.002 Flushing logs...                                                                            
    Internal error                                                                                                  
Traceback (most recent call last):                                                                                  
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 209, in execute                     
    self._launch_workers()                                                                                          
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 564, in _launch_workers             
    self._launch_worker(task)                                                                                       
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 584, in _launch_worker              
    self._poll.add(fd)                                                                                              
AttributeError: 'NoneType' object has no attribute 'add'                                                            

During handling of the above exception, another exception occurred:                                                 

Traceback (most recent call last):                                                                                  
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 1102, in main                           
    start(auto_envvar_prefix="METAFLOW", obj=state)                                                                 
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 829, in __call__         
    return self.main(args, kwargs)                                                                                  
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 782, in main             
    rv = self.invoke(ctx)                                                                                           
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 1259, in invoke          
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                         
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 1066, in invoke          
    return ctx.invoke(self.callback, ctx.params)                                                                    
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 610, in invoke           
    return callback(args, kwargs)                                                                                   
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 666, in wrapper                         
    return func(args, kwargs)                                                                                       
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\decorators.py", line 33, in new_func    
    return f(get_current_context().obj, args, kwargs)                                                               
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 789, in run                             
    runtime.execute()                                                                                               
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 230, in execute                     
    self._killall()                                                                                                 
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 275, in _killall                    
    list(self._poll_workers())                                                                                      
  File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 521, in _poll_workers               
    for event in self._poll.poll(PROGRESS_INTERVAL):                                                                
AttributeError: 'NoneType' object has no attribute 'poll'

I see Argo support might be forthcoming #50 would this be an option for starting flows in k8s from windows or is windows as a client completely out?

Also its not clear to me how the code is packaged? I see decorators for conda & image and reference to the code being tarball'd to S3. Can you explain or point me to the docs that explain the code packaging for k8s. Thanks.

savingoyal commented 2 years ago

@erowan Would you like to try out #992 in your windows environment? Here is the code that packages up your flow.

savingoyal commented 2 months ago

Metaflow works well in WSL2 - closing this issue since that seems like the most reasonable way forward. Many modern ML packages don't play well with windows and have to rely on WSL2, so even if Metaflow somehow worked well within Windows (but outside of WSL2), it's utility will be somewhat limited.

visr commented 2 months ago

Sounds fine to me. I'm not currently using metaflow, but the reason I opened the issue because it wasn't documented that it doesn't work on Windows. Is this now documented? I had a quick look and I don't see anything in https://docs.metaflow.org/getting-started/install, though for R it is: https://docs.metaflow.org/v/r/getting-started/install#windows-support.

Netflix / metaflow

Windows support #60