search

ScilifelabDataCentre / genome-portal

This is the repository for the Swedish Reference Genome Portal, a service facilitating access and discovery of genome data of non-model eukaryotic species studied in Sweden
https://genomes.scilifelab.se/
MIT License
0 stars 0 forks source link

makefile errors on MacOS: diagnosis and suggestion of fixes #14

Closed brinkdp closed 4 months ago

brinkdp commented 5 months ago

When working on populating and testing the config.yml files for the different species, I have run into some issues with running the makefile on my MacBook.

These errors are by no means urgent since the project will be deployed on linux eventually. I have also figured out work-arounds in my local instance of the repo, so these errors are not blocking my work. But I thought that I would report my findings and what I have done so far to fix these errors in case someone else want to run the makefile on MacOS.

Error 1: Shell version

The makefile begins by calling for SHELL=/bin/bash, and this is where many of the MacOS issues stems from. The default shell of MacOS is zsh, but it also comes bundled with bash so calls to /bin/bash will execute. The fact that bash is part of the OS made me completely confounded about the errors for the longest time. The problem, as I eventually found out by reading online, the version of bash. Apparently Apple stopped updating their bundled version of bash many years ago when bash changed its license, and changed to zsh. So MacOS still has bash v3.2 (last updated in 2014?). Because of this, there are functions calls in the swedgene makefile that currently does not work properly on MacOS's bash 3.2.

My fix uses the fact that the shell path is declared in a variable in the makefile. Thus I can change the SHELL variable from the command line:

make build SHELL=/bin/zsh

which correctly changes the shell, but eventually runs into an error with cp (see the Error 2 section below).

Another alternative is to use an updated version of bash. However, since /bin/bash/is used by MacOS for some core functionalities, the internet strongly advice against updating that folder as to not break the OS. Instead I downloaded bash 5.2 with the homebrew package manager (brew install bash) and pointed the SHELL variable to that install:

make build SHELL=/opt/homebrew/bin/bash

This is my preferred way of running the makefile, since it uses the same shell the code was developed for. (Another bonus with using bash 5.2 over zsh is that it supports the new fancy formatting in the output :) ).

However, even with bash 5.2, I still get the cp error.

Error 2: different version of cp

The one remaining error that changing the shell on MacOS does not fix has to do with the cp command. Eventually, all of the above fixes will run into this error:

cp: illegal option -- -
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-aclpSsvXx] source_file target_file
       cp [-R [-H | -L | -P]] [-fi | -n] [-aclpSsvXx] source_file ... target_directory
make: *** [install] Error 64

In my naïveté, I thought that changing the shell would also change the versions of the command line tools, but I was wrong. I have now learned that the reason for the error is that MacOS uses BSD utilities instead of GNU utilities. The fix for this, as described online, is either to install GNU coreutils or to use a MacOS-supported command in the code. Since I did not want to permanently modify the makefile, I did the former (brew install coreutils). This enables the correct version of the copy command by invoking gcp instead of cp. When I exchange cp for gcp in the make install function (line: cp --parents -t hugo/static/ $(LOCAL_FILES) $(GFF_INDICES) $(FASTA_INDICES) $(JBROWSE_CONFIGS)) AND run make with SHELL=/opt/homebrew/bin/bash, everything executes nicely!

To avoid having a local altered makefile with cp exchanged for gcp, I modified my shell profile as per the information given by running brew info coreutils. Since zsh is my default shell, I added the line

export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"

to ~/.zshrc

This invokes the GNU utils when I open a new shell, and I can then successfully run make build SHELL=/opt/homebrew/bin/bash!

(A side observation: Interestingly, with the GNU utils in place, make build SHELL=/bin/zsh runs without errors but does not create the config.json file. Running e.g. make DATADIRS=data/clupea_harengus SHELL=/bin/zshgives an error though.

cp: cannot stat 'data/clupea_harengus/config.json': No such file or directory
make: *** [install] Error 1

But I think getting the makefile it to work on zsh is beside the scope of the project. I suggest to should disregard this in favor of the SHELL=/opt/homebrew/bin/ fix. )

Conclusions

MacOS compatibility with the makefile can be achieved by installing homebrew and then doing the following steps:

brew install bash
brew install coreutils
sudo echo -n 'export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"' >> ~/.zshrc

And then always run the makefile and its functions with the SHELL variable fix: make SHELL=/opt/homebrew/bin/bash

While it certainly took me some time to figure all of this out, I like that it is an non-intrusive fix in the sense that does not require any changes to the makefile itself. If you think that my findings are worthwhile, I suggest to update README.mdwith a comment on how to run this on MacOS.

kwentine commented 4 months ago

Thanks for a very clear and detailed write-up Daniel, I did not know about the state of affairs between Bash and MacOS, neither about the different flavors of the standard utilities. To iron out those discrepancies, I will provide a Dockerfile so that make build can be run inside a container.