prebuild for github actions

Fuzzyma commented 2 years ago

Atm, whenever the dependencies are installed, github actions needs to recompile re2 every time which takes a lot of time. Would it be possible to supply a build for the architecture github actions are using?

samatcolumn commented 2 years ago

Big +1 to this issue! Building re2 is about 45s of our 4 minute CI jobs.

uhop commented 2 years ago

It actually requires a change in https://github.com/uhop/install-artifact-from-github

It will bump a major version in that project.

Fuzzyma commented 2 years ago

@uhop well that repo is under your control as well. Cant you just do that? :)

uhop commented 2 years ago

I left that comment above so I remember to implement this feature. ;-)

uhop commented 2 years ago

The code looks right. I expected to see some Github Actions-specific checks, but it has none.

It looks like I need to see an output of your CI job to figure out why it bails out from getting a precompiled version.

Fuzzyma commented 2 years ago

I think this is the relevant part:

2022-06-12T06:42:11.2744485Z ➤ YN0000: ┌ Link step
2022-06-12T06:42:11.2745449Z ##[group]Link step
2022-06-12T06:42:36.1530936Z ➤ YN0007: │ core-js@npm:3.6.5 must be built because it never has been before or the last one failed
2022-06-12T06:42:36.1549476Z ➤ YN0007: │ core-js@npm:3.22.8 must be built because it never has been before or the last one failed
2022-06-12T06:42:36.1555732Z ➤ YN0007: │ protobufjs@npm:6.11.3 must be built because it never has been before or the last one failed
2022-06-12T06:42:36.1559585Z ➤ YN0007: │ vue-demi@npm:0.12.5 [2e4c4] must be built because it never has been before or the last one failed
2022-06-12T06:42:36.1628519Z ➤ YN0007: │ cypress@npm:9.7.0 must be built because it never has been before or the last one failed
2022-06-12T06:42:36.1633627Z ➤ YN0007: │ esbuild@npm:0.14.42 must be built because it never has been before or the last one failed
2022-06-12T06:42:36.1636953Z ➤ YN0007: │ protobufjs@npm:6.11.2 must be built because it never has been before or the last one failed
2022-06-12T06:42:36.1640717Z ➤ YN0007: │ es5-ext@npm:0.10.61 must be built because it never has been before or the last one failed
2022-06-12T06:42:36.1651000Z ➤ YN0007: │ re2@npm:1.17.4 must be built because it never has been before or the last one failed
2022-06-12T06:42:36.1656807Z ➤ YN0007: │ vue-demi@npm:0.13.1 [52ecd] must be built because it never has been before or the last one failed
2022-06-12T06:42:36.1658795Z ➤ YN0007: │ @sentry/cli@npm:2.1.0 must be built because it never has been before or the last one failed
2022-06-12T06:42:37.2716611Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT No github repository was identified.
2022-06-12T06:42:37.2720846Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT Building locally ...
2022-06-12T06:42:37.4915818Z ➤ YN0000: │ @sentry/cli@npm:2.1.0 STDERR info sentry-cli Downloading from https://downloads.sentry-cdn.com/sentry-cli/2.1.0/sentry-cli-Linux-x86_64
2022-06-12T06:42:37.7680853Z ➤ YN0000: │ cypress@npm:9.7.0 STDOUT Note: Skipping binary installation: Environment variable CYPRESS_INSTALL_BINARY = 0.
2022-06-12T06:42:37.7692024Z ➤ YN0000: │ cypress@npm:9.7.0 STDOUT 
2022-06-12T06:42:38.0056360Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT 
2022-06-12T06:42:38.0057576Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT > re2@1.17.4 rebuild
2022-06-12T06:42:38.0058875Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT > node-gyp rebuild
2022-06-12T06:42:38.0059814Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT 
2022-06-12T06:42:38.0493659Z ➤ YN0000: │ @sentry/cli@npm:2.1.0 STDERR info Checksum validation passed. 
2022-06-12T06:42:38.0895045Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info it worked if it ends with ok
2022-06-12T06:42:38.0921677Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info using node-gyp@8.4.1
2022-06-12T06:42:38.0922956Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info using node@16.15.1 | linux | x64
2022-06-12T06:42:38.1824436Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info find Python using Python version 3.8.10 found at "/usr/bin/python3"
2022-06-12T06:42:38.3049701Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp http GET https://nodejs.org/download/release/v16.15.1/node-v16.15.1-headers.tar.gz
2022-06-12T06:42:38.4194756Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp http 200 https://nodejs.org/download/release/v16.15.1/node-v16.15.1-headers.tar.gz
2022-06-12T06:42:38.7743119Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp http GET https://nodejs.org/download/release/v16.15.1/SHASUMS256.txt
2022-06-12T06:42:38.8058321Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp http 200 https://nodejs.org/download/release/v16.15.1/SHASUMS256.txt
2022-06-12T06:42:38.8147835Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn /usr/bin/python3
2022-06-12T06:42:38.8154714Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args [
2022-06-12T06:42:38.8156126Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '/home/runner/work/packmatic/packmatic/node_modules/re2/node_modules/node-gyp/gyp/gyp_main.py',
2022-06-12T06:42:38.8157430Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   'binding.gyp',
2022-06-12T06:42:38.8199678Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-f',
2022-06-12T06:42:38.8200633Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   'make',
2022-06-12T06:42:38.8201452Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-I',
2022-06-12T06:42:38.8202445Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '/home/runner/work/packmatic/packmatic/node_modules/re2/build/config.gypi',
2022-06-12T06:42:38.8203344Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-I',
2022-06-12T06:42:38.8204359Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '/home/runner/work/packmatic/packmatic/node_modules/re2/node_modules/node-gyp/addon.gypi',
2022-06-12T06:42:38.8205541Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-I',
2022-06-12T06:42:38.8206479Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '/home/runner/.cache/node-gyp/16.15.1/include/node/common.gypi',
2022-06-12T06:42:38.8207408Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-Dlibrary=shared_library',
2022-06-12T06:42:38.8208272Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-Dvisibility=default',
2022-06-12T06:42:38.8209198Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-Dnode_root_dir=/home/runner/.cache/node-gyp/16.15.1',
2022-06-12T06:42:38.8210260Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-Dnode_gyp_dir=/home/runner/work/packmatic/packmatic/node_modules/re2/node_modules/node-gyp',
2022-06-12T06:42:38.8211361Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-Dnode_lib_file=/home/runner/.cache/node-gyp/16.15.1/<(target_arch)/node.lib',
2022-06-12T06:42:38.8212499Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-Dmodule_root_dir=/home/runner/work/packmatic/packmatic/node_modules/re2',
2022-06-12T06:42:38.8213415Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-Dnode_engine=v8',
2022-06-12T06:42:38.8214267Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '--depth=.',
2022-06-12T06:42:38.8215110Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '--no-parallel',
2022-06-12T06:42:38.8215980Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '--generator-output',
2022-06-12T06:42:38.8216811Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   'build',
2022-06-12T06:42:38.8217634Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args   '-Goutput_dir=.'
2022-06-12T06:42:38.8218439Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args ]
2022-06-12T06:42:38.9821851Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn make
2022-06-12T06:42:38.9822892Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
2022-06-12T06:42:38.9844909Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT make: Entering directory '/home/runner/work/packmatic/packmatic/node_modules/re2/build'
2022-06-12T06:42:38.9861407Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/lib/addon.o
2022-06-12T06:42:40.7584406Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/lib/new.o
2022-06-12T06:42:42.6070867Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/lib/exec.o
2022-06-12T06:42:43.7909482Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/lib/test.o
2022-06-12T06:42:44.7528536Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/lib/match.o
2022-06-12T06:42:45.9909364Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/lib/replace.o
2022-06-12T06:42:47.9426480Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/lib/search.o
2022-06-12T06:42:48.8933048Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/lib/split.o
2022-06-12T06:42:50.0466008Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/lib/to_string.o
2022-06-12T06:42:51.0048288Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/lib/accessors.o
2022-06-12T06:42:52.1814372Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/lib/util.o
2022-06-12T06:42:53.2581345Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/bitstate.o
2022-06-12T06:42:54.2301772Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/compile.o
2022-06-12T06:42:55.9296658Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/dfa.o
2022-06-12T06:42:59.0072004Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/filtered_re2.o
2022-06-12T06:42:59.8921317Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/mimics_pcre.o
2022-06-12T06:43:00.6639656Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/nfa.o
2022-06-12T06:43:02.0434605Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/onepass.o
2022-06-12T06:43:03.1275648Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/parse.o
2022-06-12T06:43:05.2566607Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/perl_groups.o
2022-06-12T06:43:05.2912852Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/prefilter.o
2022-06-12T06:43:06.6830677Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/prefilter_tree.o
2022-06-12T06:43:08.8590203Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/prog.o
2022-06-12T06:43:10.9790336Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/re2.o
2022-06-12T06:43:12.9777503Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/regexp.o
2022-06-12T06:43:15.2384666Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/set.o
2022-06-12T06:43:16.7095597Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/simplify.o
2022-06-12T06:43:17.8659639Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/stringpiece.o
2022-06-12T06:43:18.2923708Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/tostring.o
2022-06-12T06:43:19.0938113Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/unicode_casefold.o
2022-06-12T06:43:19.1309714Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/re2/unicode_groups.o
2022-06-12T06:43:19.2318407Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/util/pcre.o
2022-06-12T06:43:20.2338366Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/util/rune.o
2022-06-12T06:43:20.3113136Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   CXX(target) Release/obj.target/re2/vendor/util/strutil.o
2022-06-12T06:43:20.6989556Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   SOLINK_MODULE(target) Release/obj.target/re2.node
2022-06-12T06:43:20.9283782Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT   COPY Release/re2.node
2022-06-12T06:43:20.9326080Z ➤ YN0000: │ re2@npm:1.17.4 STDOUT make: Leaving directory '/home/runner/work/packmatic/packmatic/node_modules/re2/build'
2022-06-12T06:43:20.9355382Z ➤ YN0000: │ re2@npm:1.17.4 STDERR gyp info ok 
2022-06-12T06:43:20.9546199Z ##[endgroup]
2022-06-12T06:43:20.9547270Z ➤ YN0000: └ Completed in 1m 10s

Hope it helps!

uhop commented 2 years ago

This is the decision line:

re2@npm:1.17.4 STDOUT No github repository was identified.

It corresponds to this line: https://github.com/uhop/install-artifact-from-github/blob/master/bin/install-from-cache.js#L189

Effectively, this function fails to identify a GitHub repository: https://github.com/uhop/install-artifact-from-github/blob/master/bin/install-from-cache.js#L64

It does so purely on environment variables set by NPM. It is known that not all package managers set them or any other equivalent package manager-specific variables. See https://github.com/uhop/node-re2/issues/92 for examples.

Do you use NPM or anything else? I want to create a minimal repro case so I can debug.

Fuzzyma commented 2 years ago

I am using yarn (newest version)

uhop commented 2 years ago

It looks like your problem is related to https://github.com/uhop/node-re2/issues/69

IIRC we (developers of binary modules) have no support from Yarn: https://github.com/uhop/node-re2/issues/69#issuecomment-738190562

There is a bigger issue not solved by developers of non-NPM packages — the dependency layout. Apparently, they introduced "advanced" layouts to speed up their managers, which breaks where modules go relying on a custom module resolution. Obviously, for binary modules, we need to know where to put them, if we use any precompilation technique.

I actually filed a feature request for Yarn:

My ticket: https://github.com/yarnpkg/berry/issues/2436
- I am willing to implement it if there is a will to accept it.
Another relevant discussion: https://github.com/yarnpkg/berry/issues/1659

Recently some guy provided a plugin for Yarn, which adds necessary environment variables. I didn't try it, but it can solve your problem: https://github.com/Ayc0/yarn-plugin-envs

I suggest you to try it and share your results with me in case it takes me some time to do it.

uhop commented 2 years ago

I opened a PR: https://github.com/yarnpkg/berry/pull/4546

samatcolumn commented 2 years ago

@uhop if you open a bounty for this issue I would like to contribute to it! This has a lot of value to me and I'd be happy to pay if that would bump it up your priority list.

uhop commented 2 years ago

@samatcolumn — thank you for your support. At the moment the proverbial ball is on the yarn's side and a few hours ago we had some movements: both developers I am working with on this feature molded my code as they see fit. I hope it means that it will be merged into the main branch soon. Fingers crossed! As soon as it happens re2 will start using prebuilds automatically.

uhop commented 2 years ago

…and the PR is merged. As soon as new yarn is published, try to update it and it should work.

samatcolumn commented 2 years ago

@uhop whhhaaaaatttt!!! That's amazing news.

Can you explain this a little more:

As soon as new yarn is published, try to update it and it should work.

I don't use yarn at all. I just use some packages (notably firebase-tools) which depend on re2 and npm install-ing those packages on GitHub Actions is very slow due to the re2 rebuild. When should I expect to see a speedup and what actions do I need to take?

Fuzzyma commented 2 years ago

Haha finally!! Poking people until they do stuff sometimes works 😂. Thanks so much for fixing this!!

uhop commented 2 years ago

I don't use yarn at all. I just use some packages (notably firebase-tools) which depend on re2 and npm install-ing those packages on GitHub Actions is very slow due to the re2 rebuild. When should I expect to see a speedup and what actions do I need to take?

Sorry, my mistake. Most people complaining about that are yarn users.

I suspect that situation is like that:

When I publish a version I pre-build it for all supported versions of Node at that moment on Linux, Mac, and Windows.
- I don't go back to old versions to make builds for new Node versions, which were not available at the time of publishing.
firebase-tools likely uses an old version of node-re2.
- I went to see its package.json but it doesn't refer to re2 directly. It must be a dependency of a dependency.
Building re2 with new Node versions is possible in many cases but it is likely to trigger a rebuild because there is no pre-build version.

Practical suggestions:

Try to use an older version of Node, which had a pre-build for that specific re2 version.
- You can find the re2 version in the package-lock.json file.
Try to lobby the authors of the package that refers to re2 to upgrade the dependencies.
- I try very hard not to break users' code when making new versions. So it is likely will go super smooth.

Potentially I can have a process to add more pre-built modules to existing versions. I have no idea how to do it from the top of my head:

There are some parameters I have to pass to Google Actions:
- I have to pass, which version(s) to update.
- Optionally I have to pass what Node versions to use for that.
I have to trigger it somehow.

If you have any suggestions or a PR (even better!), I am all ears. Feel free to open another ticket for that.

uhop commented 2 years ago

@Fuzzyma — please confirm that it works for you.

Fuzzyma commented 2 years ago

@uhop re2 is only a dependency of a dependency for me as well. So I wait for the yarn release and will report back :)

uhop commented 2 years ago

Potentially I can have a process to add more pre-built modules to existing versions. I have no idea how to do it from the top of my head:

There are some parameters I have to pass to Google Actions:

I have to pass, which version(s) to update.

Optionally I have to pass what Node versions to use for that.

I have to trigger it somehow.

Note to me: it all could be doable — https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow

uhop / node-re2

prebuild for github actions #125