Figure out a way to run SLiM-based simulations on Windows

[jielab]

Dear Martin:

I now successfully installed slim on my Windows machine, and was able to run your "Example 3 (Figure 4)" for the majority part.

however, towards the end, it complains Unrecognized command-lline argument: \

But there is no "\" in my command at all...

Is this something having to do with Windows path names?

Thanks! Jie

[bodkan]

I'm glad to hear you've managed to install SLiM on your Windows machine. Unfortunately, as I replied elsewhere, slendr's internals currently assume a unix-like operating system in the background. Specifically, SLiM-based simulations are started on the unix command-line... which, by definition, is not present on the Windows system.

I'll rename this issue to reflect what is actually the problem. As I also mentioned to you elsewhere, I hope to port slendr to Windows over the summer once I have more time to work on this and once I have physical access to Windows machines to do some testing.

[bodkan]

Hey @jielab: How comfortable are you with R development? If you would be interested in accelerating the Windows port, the code that actually makes the SLiM back end of slendr purrr is extremely simple (implemented in the slim() function here). It literally just calls the slim binary on the command line, making sure all paths to required files are passed to the binary correctly (and that the output tree sequence is also saved and loaded to the correct path).

As I said, it will take a while before I can delve into this (earliest during the summer) but if you know your way around R coding and Windows shell (whatever "shell" is in the context of Windows in 2023 -- I ended my Windows journey back when DOS was still a thing 🙃), you should be able to get to a working, fully portable port of slim() fairly easily.

If not, don't worry about this. It will happen at some point, it's just that it was very low priority until know. Unlike msprime() simulations which can be easily run on weak machines even for very large models, slim() simulations of the same scale need large computational resources -- this generally means powerful unix-based computational servers, not Windows machines.

[jielab]

Dear Martin:

I did use R for >10 years, although my level might be only intermediate :- )

So, it would be great if you could throw me the actual R code to implement this.

Best regards, Jie

[bodkan]

Hello again @jielab,

I'm afraid "just throwing you the actual R code to implement this" would basically amount to... well, basically implementing this myself. :)

I will keep this issue open and will post an update once I get to do this so you are aware of any potential progress on this.

I'll be busy with teaching and then go on vacation during the rest of March but I will do my best to find a Windows machine with admin rights (so that I can install all dependencies and have full control over the development environment) somewhere at work once I get back. I can't make any promises on the timeline but unless I'm missing something, I really think porting the slim() function to Windows should be relatively straightforward.

[GKresearch]

Hi @bodkan,

I don't know if you have been working on this issue since last noted, so I apologize if the following information is out-of-date.

I was reviewing this issue and have been working to implement a method to access slim through slendr on Windows. My experience with slendr is pretty nascent and I am not an R expert either, but I took a look at the slim backend, and I think the issue stems from pathing expectations. A lot of my solutions here are likely inelegant as I brute-forced the issue.

I managed to get slim() working on my machine by changing a couple parts of the command. Specifically the following lines in the engines.R script: -line 322: Remove " \\n"
-line 324: else paste0(" \\n -d 'SAMPLES=\"", sampling_path, "\"'") becomes else paste0(" -d \"SAMPLES='", sampling_path, "'\"") -I edited the automatic tempfile() pathing so that instead of producing "\" for delimitation of the path it produces "/". This effects several user-defined variables like the samples and output -Finally, I was having difficulty getting sprintf to work with some of my changes so I switched to paste0, but that can probably be fixed pretty easily, I was just focused on finding a solution to getting slim running.

The primary errors I received during this process are:

• The one mentioned above: Unrecognized command-lline argument: \
• And: ERROR (Eidos_DefineConstantsFromCommandLine): malformed command-line constant definition

I should note, these changes allowed me to get the slim() command to produce output, I don't know how if other issues may come up downstream in the rest of slendr for Windows users.

Sincerely, GK

[bodkan]

Hey @GKresearch,

This is amazing, thanks SO MUCH for poking into this!

This issue is very much still open, and definitely not out-of-date. Unfortunately, I still haven't found a way to work on this on Windows (the only Windows machine I have physical access to is my wife's office laptop and that is hopelessly locked down). That said, the changes you describe are quite similar to the process I went through while implementing the msprime support for Windows a long time ago, when I had a brief window (ha) of opportunity to work with a borrowed Windows machine. So I'm quite confident that you're on the right track!

Would you be interested in opening a pull request with your changes? If you would do this, I could then (relatively) easily take over the changes and test that everything works using my GitHub Actions testing pipeline (which currently mostly excludes Windows).

Actually, if you're not comfortable with git, git workflow, or GitHub pull requests, could you simply upload your modified R/tree-sequences.R file to this GitHub issue (or multiple files, if you had to edit others)? I should be able to take care of the rest -- open the PR, implement the changes and (if you'd be up to it) have you test everything on your computer.

In the meantime, can I ask what you mean by this?

The primary errors I received during this process are:

The one mentioned above: Unrecognized command-lline argument: \ And: ERROR (Eidos_DefineConstantsFromCommandLine): malformed command-line constant definition

Does this mean you saw these errors a) before you started working on these changes, b) during the process of working on these changes, c) you see them even after you implemented this?

Many, MANY, thanks for your help with this! I really appreciate this!

I'm glad you got something working. I'm busy with a conference this week and will be largely travelling on holidays during August, but I'm quite keen to push this forward as soon as I can, hopefully before I start my holidays.

[GKresearch]

Hi @bodkan,

Sure, I will go ahead and set up a new request sometime this week. My experience with git isn't great, but I'm trying to learn, so I will go ahead and try it that way and if there are issues, I'll just upload afterwards. I can't get around to it today, but I'll try to submit it by tomorrow if possible.

Regarding the errors, I simply meant those were the errors I found during the process. After the fixes, these errors no longer appeared.

Enjoy the conference!!!

Sincerely, GK

[bodkan]

Hey, no rush at all. There's almost zero chance I'll manage to look at it this week. So any time after will be perfect.

Regarding the errors, I simply meant those were the errors I found during the process. After the fixes, these errors no longer appeared.

Gotcha, this all makes sense. This suggests that SLiM had probably trouble picking up the paths set up from R through slendr (given some specifics of Windows).

I would have hoped that the paths handled through R will be multiplatform. I seem to remember based on some casual quick checks that -- surprisingly -- \ vs / tended not to matter? But I have never actually looked at this in detail.

Regardless, the unit tests across Linux & macOS I have set up are very extensive, so running them all on Windows will catch all issues.

[bodkan]

Oh, and unless you want to, don't stress too much about git stuff. It can be... frustrating and intimidating. Feel free to play around though. Just be aware that all experts work like this, from time to time, even if they don't admit it. :)

It really is totally cool just uploading the files here though. I would then ping you once I make a PR here myself.

[GKresearch]

I've created a pull request, let me know if it got through!

[bodkan]

Yup, all good! Thanks!

I'll keep this issue open while we continue this conversation over under the new PR.

For future reference, it's PR #142.

[willright28]

Hi @jielab @bodkan,

I wanna ask a small question about how to modify the $PATH that make it point to the path where SLIM installed on windows.

I have installed slim using MYSY2, and the slim.exe could be found on, say, C:/msys64/mingw64/bin/, but the slendr still can not locate the slim even after I added the "PATH="C:/msys64/mingw64/bin/" to Renviron file (named Rcmd_environ on C:\Program Files\R\R-4.1.0\etc).

I have also added "C:/msys64/mingw64/bin/" to the system path of windows but still failed to detect it.

I would like to hearing suggestions from you, many thanks!

[bodkan]

Hi @willright28,

For the time being, you should be able to set the path to the SLiM binary via the function argument slim_path=. See this manual page or run ?slim in your R console for more detail.

However, please note that the slim() functionality is not yet supported on Windows and I'm not entirely sure how well it will work at this moment. I've been working on implementing this constantly for the past week, so far with mixed results. See this PR for discussion. You can track my progress there.

I hope I can get there in the next couple of days but I don't have a Windows machine available and so my progress on this is pretty much constant stumbling in the dark.

[bodkan]

I want to eventually support the $PATH management for slendr/SLiM on Windows as well, but it's been an uphill battle. The slim_path= argument is currently the way to do it.

R and everything shell-related on Windows is like a whole another planet for me.

[willright28]

@bodkan, Thanks a lot for your quick response! I will keep an eye on this project.

[bodkan]

Solved in an epic quest which went down in history of scientific computing as PR #149.

It's very likely that minor issues remain, particularly for SLiM installed through conda. Happy to accept further reports on those here.

That said, if you, a future user, runs into problems with SLiM/slendr on Windows, please try to follow the section 2.3.1 "Installing SLiM with an installer on Windows" in the SLiM manual. Unlike conda installation, this will give you a full-featured SLiM installation including SLiMgui -- it's a little hard to imagine using slendr's spatial SLiM simulation features without SLiMgui. :)