The "/project/" path isn't there anymore for Linux in version 1.5.3

[hoffmang9]

Upgraded from 1.5.1 to 1.5.3 in a branch here: https://github.com/Chia-Network/bls-signatures/tree/up-cibuildwheel

In 1.5.1 we would install cmake into /project/cmake-3.17.3-Linux-x86_64 and build various dependencies from there as we add /project/cmake-3.17.3-Linux-x86_64 to the front of PATH: https://github.com/Chia-Network/bls-signatures/runs/880546025?check_suite_focus=true#step:10:299

In 1.5.3 cmake gets installed into //cmake-3.17.3-Linux-x86_64 so the PATH we preset doesn't work and the build fails. https://github.com/Chia-Network/bls-signatures/runs/887691441?check_suite_focus=true#step:10:284

Operative workflow here: https://github.com/Chia-Network/bls-signatures/blob/up-cibuildwheel/.github/workflows/build.yml

I can work around but seems like either an issue that needs a fix or some mention in documentation. I'm sure it came out of the move to docker - which otherwise looks cool.

[hoffmang9]

Confirmed the same behavior with version 1.5.4 as well.

[YannickJadoul]

While you're at it: how easy is it to try 1.5.2? It would be nice to pinpoint the exact release :-)

[hoffmang9]

Trivial - give me 5 minutes.

[YannickJadoul]

Thanks. I assume it's going to be #386, so that would mean 1.5.2 should still be OK.

[hoffmang9]

Indeed 1.5.2 works just fine - https://github.com/Chia-Network/bls-signatures/runs/887756210?check_suite_focus=true#step:10:297

[YannickJadoul]

OK, then I think that the problem is that CIBW_BEFORE_BUILD etc isn't executed inside of /project or with /project as cwd. Could you check this, @joerick?

I guess that would mean that passing --prefix=/project would solve your problem, @hoffmang9? Running pwd would confirm whether that would work as workaround.

But this seems like a regression (or at least an unintended change), so let me see if I can make a PR for this.

[hoffmang9]

Queued up to push was a quick echo $PATH and pwd for as soon as the link above completed.

[hoffmang9]

Two interesting things.

First pwd is / so definitely not /project has it's historically been (or we've been luckily misusing...)

Second, our trick to find the cmake path -

CIBW_ENVIRONMENT_LINUX: "PATH=/project/cmake-3.17.3-Linux-`uname -m`/bin:$PATH"

no longer works in 1.5.4. Not critical but interesting.

[YannickJadoul]

Second, our trick to find the cmake path -

CIBW_ENVIRONMENT_LINUX: "PATH=/project/cmake-3.17.3-Linux-`uname -m`/bin:$PATH"

no longer works in 1.5.4. Not critical but interesting.

Well, that's the same issue, or do you mean x86_64 isn't interpolated anymore? Again, we just changed some things (#408 and #403), so that's also possible. Do you have a link, perhaps?

[joerick]

OK, then I think that the problem is that CIBW_BEFORE_BUILD etc isn't executed inside of /project or with /project as pwd. Could you check this, @joerick?

Erm, wow, okay! That's a pretty big regression! My apologies. I can't deal with it now, and I probably won't have time tomorrow either, so I'm thinking the best thing might be to pull 1.5.4 and 1.5.3 from PyPI for now, until we can look at these regressions properly. Would that be okay with you @YannickJadoul ?

[joerick]

Two interesting things.

First pwd is / so definitely not /project has it's historically been (or we've been luckily misusing...)

No, cwd should be /project , as you had assumed :) that's a bug

Second, our trick to find the cmake path -

CIBW_ENVIRONMENT_LINUX: "PATH=/project/cmake-3.17.3-Linux-`uname -m`/bin:$PATH"

no longer works in 1.5.4. Not critical but interesting.

Also sounds like a bug!

[YannickJadoul]

Would that be okay with you @YannickJadoul ?

I'll leave version management up to you, actually. It's not thát broken, because we pass the absolute path to pip wheel. Not sure it's worth pulling things of PyPI for, but again... up to you.

I was just looking, and it should be as easy as adding a few times cwd='/project' (or similar) to the docker.call calls. And a few tests. If this is intended (just saw your next comment pop up, so it was intended), I can quickly make a PR, still?

[YannickJadoul]

If this is intended (just saw your next comment pop up, so it was intended)

Also, CIBW_BEFORE_ALL wás not executed from /project before. Was that intended, or should it also be run from /project for consistency?

[Czaki]

@hoffmang9 After quick look to code I see one comment: # don't build i686 targets, can't seem to find cmake for these You can get cmake from https://pypi.org/project/cmake/#files Then You need to play a little with PATH but should work.

Second think is tat you should CIBW_BEFORE_BUILD for external dependences (like cmake).

@YannickJadoul @joerick looking on this project {project} substitution should also be applied to environment.

I think that this line contains bug

https://github.com/joerick/cibuildwheel/blob/c0cb43800b6740677e373889705b61b17b879a8d/cibuildwheel/docker_container.py#L119

It should be: chdir = f'cd {cwd}' if cwd else 'cd /project'

[joerick]

FYI I've yanked those versions.

I was just looking, and it should be as easy as adding a few times cwd='/project' (or similar) to the docker.call calls. And a few tests. If this is intended (just saw your next comment pop up, so it was intended), I can quickly make a PR, still?

Please do!

And BEFORE_ALL should be from /project, too.

[YannickJadoul]

Second think is tat you should CIBW_BEFORE_BUILD for external dependences (like cmake).

He is doing that, I saw. CIBW_BEFORE_ALL could be nicer, even, doing it only once?

I think that this line contains bug

https://github.com/joerick/cibuildwheel/blob/c0cb43800b6740677e373889705b61b17b879a8d/cibuildwheel/docker_container.py#L119

No, this should be out of docker_container.py. Nothing else in there refers to the specific situation we use it for, so this should probably also be in linux.py.

[hoffmang9]

@hoffmang9 After quick look to code I see one comment: # don't build i686 targets, can't seem to find cmake for these You can get cmake from https://pypi.org/project/cmake/#files Then You need to play a little with PATH but should work.

That comment is ancient and wrong. We only build on x86_64 for uint128 reasons...

Second think is tat you should CIBW_BEFORE_BUILD for external dependences (like cmake).

Only linux needs cmake installed so we're using CIBW_BEFORE_BUILD_LINUX. Building before everything would blow up MacOS and Windows.

[hoffmang9]

Now that does lead me to mention the strong desire to be able to cache a phase of builds before going on to cp38 - but that's an enhancement for another issue... Installing cmake and compiling gmp and sodium twice - is no fun...

[Czaki]

@YannickJadoul @hoffmang9 My bad. Yes. Instead using CIBW_BEFORE_BUILD_LINUX you should use CIBW_BEFORE_ALL_LINUX

[hoffmang9]

The thanks I have for the suggestion goes way beyond just a heart emoji!

[YannickJadoul]

@hoffmang9 Do you have a link to a build where CIBW_ENVIRONMENT_LINUX: "PATH=/project/cmake-3.17.3-Linux-`uname -m`/bin:$PATH" goes wrong?

[hoffmang9]

https://github.com/Chia-Network/bls-signatures/runs/887777849?check_suite_focus=true#step:10:280

path and pwd - https://github.com/Chia-Network/bls-signatures/runs/887777849?check_suite_focus=true#step:10:316

That check in's workflow - https://github.com/Chia-Network/bls-signatures/blob/672d8ef513b4331b3ece3c515553ceeddb3fd58c/.github/workflows/build.yml#L76

[YannickJadoul]

Modulo potentially a few stupid mistakes left, and some review, #410 and #411 should fix your issues, @hoffmang9! Both were reasonably severe (especially the second one), so thanks a lot for reporting! :-)

[hoffmang9]

I just tested with @YannickJadoul's fork and everything works correctly for our repo. https://github.com/Chia-Network/bls-signatures/blob/601995cbb0e850302560ead29dd526a7194a9138/.github/workflows/build.yml#L41

CMake going into /project: https://github.com/Chia-Network/bls-signatures/runs/888519945?check_suite_focus=true#step:10:283

Path includes /project and uname results of

CIBW_ENVIRONMENT_LINUX: "PATH=/project/cmake-3.17.3-Linux-`uname -m`/bin:$PATH"

https://github.com/Chia-Network/bls-signatures/runs/888519945?check_suite_focus=true#step:10:324

[YannickJadoul]

Thanks for checking, @hoffmang9 :-)

[joerick]

Alright, @YannickJadoul's fantastic fixes are released, as v1.5.5. It'd be lovely if you could give it a try @hoffmang9, when you get a chance :)

[hoffmang9]

...Itchy trigger finger. Answers in 15 minutes.

[hoffmang9]

Got stuck on a Zoom - works for me!

https://github.com/Chia-Network/bls-signatures/runs/899694313?check_suite_focus=true#step:10:292

[joerick]

marvellous. Thank you for your help debugging these issues!

[hoffmang9]

My pleasure and feel free to ping me in the future as we give cibuildwheel a workout across 4 repos and 4 build platforms.

[YannickJadoul]

All turning green and happy here, as well :-)

Thanks a lot, @hoffmang9, for reporting and helping to debug!