search

mlcommons / mlcube

MLCube® is a project that reduces friction for machine learning by ensuring that models are easily portable and reproducible.
https://docs.mlcommons.org/mlcube/
Apache License 2.0
154 stars 32 forks source link

SPython incompatible with windows. MLCube fails at import time #292

Open aristizabal95 opened 1 year ago

aristizabal95 commented 1 year ago

Trying to run a docker MLCube on windows fails because of unsupported imports. This happens because spython, the singularity python API, tries to import pwd, which doesn't exist on Windows. Here's the traceback:

$ mlcube run --mlcube=C:\Users\aleja\.medperf\localhost_8000\cubes\1\mlcube.yaml --task=prepare --platform=docker data_path="C:\Users\aleja\.medperf\localhost_8000\demo\137950e4e7b8de3baa4a9982c3ebea6a52bd33d3\mock_chexpert" labels_path="C:\Users\aleja\.medperf\localhost_8000\demo\137950e4e7b8de3baa4a9982c3ebea6a52bd33d3\mock_chexpert" output_path="C:\Users\aleja\.medperf\localhost_8000\data\tmp_26312981032654\data" Ptasks.prepare.parameters.input.data_path.opts="ro" Ptasks.prepare.parameters.input.labels_path.opts="ro"
Traceback (most recent call last):
  File "d:\users\aleja\anaconda3\envs\factored\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "d:\users\aleja\anaconda3\envs\factored\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\Users\aleja\anaconda3\envs\factored\Scripts\mlcube.exe\__main__.py", line 7, in <module>
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\click\core.py", line 1256, in invoke
    Command.invoke(self, ctx)
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\mlcube\__main__.py", line 119, in cli
    _ = SystemSettings().update_installed_runners()
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\mlcube\system_settings.py", line 111, in update_installed_runners
    installed_runners: t.Dict = Platform.get_installed_runners()
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\mlcube\platform.py", line 60, in get_installed_runners
    runner_cls: t.Type[Runner] = get_runner_class()
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\mlcube_singularity\__init__.py", line 40, in get_runner_class
    from mlcube_singularity.singularity_run import SingularityRun
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\mlcube_singularity\singularity_run.py", line 6, in <module>
    from spython.utils.terminal import (
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\spython\utils\__init__.py", line 5, in <module>
    from .terminal import (
  File "d:\users\aleja\anaconda3\envs\factored\lib\site-packages\spython\utils\terminal.py", line 12, in <module>
    import pwd
ModuleNotFoundError: No module named 'pwd'

This issue disappears when uninstall mlcube-singularity. It would be useful to provide some hints to the user if this error happens. Is singularity even supported on windows without WSL? If not, raising an error when installing mlcube-singularity could be another approach

sergey-serebryakov commented 1 year ago

@aristizabal95 Thanks! This is a manifestation of a slightly broader issue associated with how MLCube checks for installed runners and imports them. I'll prioritize this issue.