Closed eyal0 closed 2 years ago
Please provide your GHA yml file or minimal sample to reproduce your issue.
It's probably due to the craziness that I try to make msys behave like bash in my scripts. Getting a single script to work for both unix and windows is a pain.
It would be nice is GITHUB_ENV and PKG_CONFIG_PATH just plain worked instead of all the workarounds needed for github ci.
With path-type: inherit
, those envvars should work.
I'll try it again. I had a very convoluted solution to my problem many months ago that stopped working 2-3 weeks ago. I'll go back to the simply solution and see if it is working for me.
That might be because the default installation path changed (#163). If you were relying on absolute and hardcoded paths, you might need to fix those.
Getting a single script to work for both unix and windows is a pain.
You could check msys2 specific environment variables in the shell script. Like MSYSTEM=mingw64
or OSTYPE=msys
etc.
@Biswa96, see #104 for context.
Yeah, I had a really bad solution. I'm going to try to make it work cleanly. Maybe all that I ever needed was path-inherit? I thought that I had done the first time that I tried and I didn't succeed. But maybe I didn't try it enough or maybe path-inherit works better now?
Either way, I'll give it a go!
There were some changes in the virtual environments provided by GitHub. Previously MSYS2's PATH was added by default, and several packages were preinstalled. Therefore, inheriting that in the clean installation might produce conflicts. Now, in windows-2022
no extra packages are preinstalled, and I'm unsure about having it pre-added to the PATH.
Anyway, I would give it a try again.
@eine I've tried using path-inherit
but I'm not getting the result that I expect.
In my CI, I have a place where I'm printing out the entire environment. Here's what it looks like:
You can see where I'm adding the variables for PKG_CONFIG_PATH, LD_LIBRARY_PATH, and PATH.
Here's where I'm printing the env by running the env
command in msys2:
You can see that the CI is honoring my environment changes. But that's in the environment to windows, not to Unix! It's up to msys to "inherit" that.
Here's the output of env
for the PATH
variable:
I can see my $HOME/.local in there, so that's good! What about PKG_CONFIG_PATH?
Nope. Unlike PATH
, where the variable is copied from the windows environment into the msys environment, PKG_CONFIG_PATH
is not! This is why I had to do that big kludge. It would be nice if PKG_CONFIG_PATH got the same treatment as PATH. I've even settle for having all the windows environment variables copied over into new names so that I could at least use them inside of msys and do it myself!
For now, I have no solution. I could add source $GITHUB_ENV
to every single rule in the CI but that's pretty ugly! I tried to add that into the shell but it's really difficult, too, because GitHub has expectations about how that shell command will look.
What can I do to fix this?
With
path-type: inherit
, those envvars should work.
They don't. :-( Do you have a CI output that I can look at which proves otherwise?
You can see here where PATH gets special handling: https://github.com/msys2/MSYS2-packages/blob/915946a637e1f2b7e26e32782f3af322009293db/filesystem/profile#L28-L45
PKG_CONFIG_PATH does not: https://github.com/msys2/MSYS2-packages/blob/915946a637e1f2b7e26e32782f3af322009293db/filesystem/profile#L53
@eine Okay, I found a solution to my problem. I'll describe it here and the different things that I tried. Hopefully it will save someone time in the future. If you don't have time to read it, here's the trick: Use ~/.bash_profile
instead of $GITHUB_ENV
for everything.
We want to use a single GitHub CI for all builds, macos, Windows, and Ubuntu. This makes our CI smaller and more manageable. macos and unix are pretty easy because they are mostly compatible but Windows is a little trickier, especially about the environment variables! The issue is that if you put something like this in your CI:
- name: Set env variable FOO to bar
run: |
echo "FOO=bar" >> $GITHUB_ENV
It will set the variable in the environments for macos and for ubuntu, but it won't work the same for Windows. That's because it is setting the environment variable in the windows environment, which is, like the powershell. But you want the environment variable set inside of msys2.
The solution should be to use path-type: inherit
. You can read about how it works. If you look at the code for it here and here, you'll see that inherit only affects PATH
and not other things that you might set, like PKG_CONFIG_PATH
. That's a bummer because we'd like those to work, too.
One solution would be to instead write to a different file, maybe called ~/my_env
and then source it at each command. So you would need to modify every single command in your CI file to have a source
line at the top. Like this:
- run: |
source ~/.my_env
do_the_rest.sh
That's annoying! We don't want to have to do that.
GitHub will let you change the shell command for your platform. So you might try something like this:
defaults:
run:
shell: source ~/my_env; /bin/bash {0}
Now it should source your file before running the shell each time. I tried that but it didn't work. GitHub was complaining that source
wasn't an executable file on the disk. That's true, source is a command that bash know about, it's not a file that can be executed.
We know that bash will always run .bashrc
before it starts. So we could just put the lines that we want in .bashrc
. However, one problem: .bashrc
is only run for interactive shells. This is described in the man page for bash:
When an interactive shell that is not a login shell is started, bash reads and executes commands from /etc/bash.bashrc and ~/.bashrc, if these files exist. This may be inhibited by us‐
ing the --norc option. The --rcfile file option will force bash to read and execute commands from file instead of /etc/bash.bashrc and ~/.bashrc.
So we need to make bash interactive. So we need to modify the shell to be interactive.
defaults:
run:
shell: source ~/my_env; /bin/bash -i {0}
This causes other problems, though, because if the shell is interactive then some of the stuff that you do in your CI will behave differently and your tests will fail in a weird way. So this is no good.
~/.bash_login
is run only for login shells. And we can fake that by running bash
with a -l
. So that's a good technique for us!
So in the end, here's what the CI script could looks like:
jobs:
my_job:
name: ${{ matrix.os }}_job
strategy:
matrix:
os: [ubuntu, macos, windows]
include:
- os: windows
shell: msys2 {0}
- os: ubuntu
shell: '/usr/bin/bash -l -e -o pipefail {0}'
- os: macos
shell: '/bin/bash -l -e -o pipefail {0}'
runs-on: ${{ matrix.os }}-latest
defaults:
run:
shell: ${{ matrix.shell }}
steps:
- name: Setup paths and env
run: |
rm -f ~/.bash_profile
echo "export FOO=bar" >> ~/.bash_profile
What this does is:
os
is one of ubuntu
, windows
, and macos
. So matrix
, if we converted it to json, would be 3 jobs, each with a different value of os
. Here's what the job matrix looks like in yaml:
- os: windows
- os: ubuntu
- os: macos
shell
variable to all elements of the matrix with os
that matches. This way, if you have an even more complex matrix with lots of rows you don't have to attach the shell to each one. This is what we have now in yaml for our matrix.
- os: windows
shell: msys2 {0}
- os: ubuntu
shell: '/usr/bin/bash -l -e -o pipefail {0}'
- os: macos
shell: '/bin/bash -l -e -o pipefail {0}'
~/.bash_profile
. It starts as all comments except for a line that causes it to exit when we are not interactive. Any lines that we place after that won't be executed because we are not interactive, so we need to remove that. Might as well just remove it all, there is nothing useful in there. (If you want to check for yourself, put cat .bash_profile
into a CI command and see it in the logs.)~/.bash_profile
. I used export
but I'm not totally sure if that is necessary. No harm in it anyway.That's it! Now a single CI will work for windows and ubuntu and macos. The pcb2gcode project is using this successfully with a fairly complicated build for all three platforms and they all use the same steps, more or less.
This is a new error that I'm getting in GitHub CI. It started about 2 weeks ago. Seems like cygpath might be provided by git for windows, which I have never explicitly installed... Maybe GitHub changed something about their environment and cygpath is no longer in there?