mshannon-sil
commented
1 week ago After some analysis and discussion, we're planning to move forward with adding an option to use conda instead of Docker. Using conda simplifies user setup compared to Docker and allows us to run clearml agents on platforms where clearml can't spin up Docker containers, like Windows or SLURM.
We're also planning to still use Poetry to manage Python dependencies. When Poetry is called from inside a virtual environment, it installs packages into that virtual environment rather than create a new environment, so it works well with conda environments. There also aren't any strong incentives to switch to conda as our python package manager. Conda packages don't tend to have significant performance differences compared to pip packages, and about 10 of our dependencies can't be installed through a conda channel and would still need to be installed with pip regardless, which increases the chances that we run into environment issues. It's cleaner to use Poetry to install all the python packages, rather than a mix of conda and pip, and to use conda for installing everything else needed for the environment.
ddaspit
We should evaluate whether it may be better to switch from poetry/docker for virtual environment creation and dependency management to conda instead. In theory, this would simplify the setup process for users and allow SILNLP to be more flexible, especially in environments where using docker containers may not be feasible. We would still keep the docker setup for developers that would prefer to use docker.