drivendataorg / cookiecutter-data-science

A logical, reasonably standardized, but flexible project structure for doing and sharing data science work.
https://cookiecutter-data-science.drivendata.org/
MIT License
7.99k stars 2.41k forks source link

add documentation for running make on Windows #333

Closed chrisjkuch closed 3 months ago

mbkranz commented 4 months ago

@chrisjkuch Hi Chris! I'm sinking my teeth into the v2 on Windows. I'm wondering if you have any tips for this? I'm wondering if supporting docker containers would be the way to go/best practice or there are any other "best practices" or recommended ways of doing this? It seems like in the R world, there is some support for recommending docker (see this R course here)

pjbull commented 4 months ago

@mbkranz I was actually just working on some additional documentation for make on Windows.

Docker is one way to do it, but here are some others worth considering:

This brings us to make compatibility on Windows systems, which has been a constant thorn in our sides. Our test suite actually runs on Windows with make. There are multiple ways to install make onto Windows systems. One that has received positive feedback is the version that comes with Git for Windows. When you install git on Windows, the git bash shell comes with it, and handling your data science projects just in this environment is generally reliable. Alternatively, there are versions of make on the Windows package managers, for example chocolately, winget, and scoop. Traditional ways to get the make binaries include cygwin and MinGW. All these options work, but may come with some tradeoffs. Finally, Windows Subsystem for Linux is another great option. It gives a full, non-virtualized linux environment inside your windows machine. It can work well to run all your data science workflows through Ubuntu running on WSL.

Curious if any of those help and work well/easily for you.

mbkranz commented 4 months ago

Thanks for the great suggestions @pjbull . Let me know what you think about the below --

  1. Git for windows
    • I use Git for Windows through cmder emulator and have used Git bash (and some of my colleagues use as well).
    • Unfortunately, we aren't finding the make executable in the Git Bash shell... is there something we are missing by chance? I can't find the executable in any Git directory bin .... image
  2. WSL2
    • Definitely an option! I can definitely go this route for my personal use. However, my task is to try and develop recommendations/example analyses for a broad audienace of researchers/analysts. I'm worried WSL2 would be too much for folks unfamiliar with unix. Have you gotten feedback for using WSL2 with a more junior analyst type persona?
  3. Other options
    • Haven't tried these but thanks for the refs (make be good to just put a make binary executable in my cmder emulator folder. One consideration with these is that it may add an extra piece of software to download. I think git for windows is a MUST for all analysts (no matter what technical skill level). If make is included in git for windows, this would probably be the easiest way to run.
pjbull commented 4 months ago

Yeah, it does not seem like make actually is included with git bash anymore (if it ever was). It looks like there are prebuilt executables with make from the following places, though I can't vouch for them in any way:

jayqi commented 3 months ago

I just saw that there's a package for Make on conda-forge, and it seems to include Windows builds. Has any tried this? https://anaconda.org/conda-forge/make/

Seems to maybe be MinGW's Make? https://github.com/conda-forge/make-feedstock/blob/0236e5e0f90183076c2e867a104fb7843e6bce9a/recipe/meta.yaml#L22C20-L22C27

pjbull commented 3 months ago

I just saw that there's a package for Make on conda-forge, and it seems to include Windows builds. Has any tried this? https://anaconda.org/conda-forge/make/

I just tested this and it works, but with one slightly annoying caveat. Let's say I use conda to manage my virtual environments. Once I activate my project environment, I have to reinstall make into that environment as well since it is just in the base environment:

> conda install -c conda-forge make  # installs into base
> ccds

...  # setup project

> make create_environment
> conda activate project
> make requirements
'make' is not recognized as an internal or external command,
operable program or batch file.
jayqi commented 3 months ago

It'll probably take some thought here to figure out a good workflow to recommend for conda+make.

I think it's plausible that we might be able to recommend a workflow using a global application installer like pixi or condax. We're in a bit of a transitionary period though where pixi is still building out functionality and popularity, and condax is a little unclear its long-term maintenance.

jayqi commented 3 months ago

Closed by #355