Closed benbenz closed 2 years ago
Hi,
thanks for the analysis.
I am sorry but I am still struggling, how to incorporate the git submodules in npm for the different libraries correctly.
After looking into the npm install docs, it seems to be necessary to use:
npm install --save git+https://github.com/fails-components/webtransport
for the installation, since this should should also fetch the submodules. (I do not have a unix cmd at hand, it still fails under windows, but would be nice if you can try it and post the error log).
This was for v1.
For v2, please do instead:
cd webtransport
git submodule update -i -r
npm run install
I have no removed the git submodule update, since it did not work in v1. But I am puzzled about the message, that it does not find build.ninja. Since this should be generated by cmake.js in build/build.ninja, can you check this and the content of the build directory?
Ok v1 is still broken, it seems, that npm strips the whole .git stuff before installing, in this way also the submodules are gone..., this need some additional work.
Hi,
No worries!
With the new URL, it seems it goes further, but stops when it compiles the webtransport project:
npm ERR! code 1
npm ERR! git dep preparation failed
npm ERR! command /usr/local/bin/node /usr/local/lib/node_modules/npm/bin/npm-cli.js install --force --cache=/Users/benoit/.npm --prefer-offline=false --prefer-online=false --offline=false --no-progress --no-save --no-audit --include=dev --include=peer --include=optional --no-package-lock-only --no-dry-run
npm ERR! > @fails-components/webtransport@0.0.0 install
npm ERR! > cmake-js build
npm ERR!
npm ERR! [
npm ERR! '/usr/local/bin/node',
npm ERR! '/Users/benoit/.npm/_cacache/tmp/git-cloneaKQhsZ/node_modules/.bin/cmake-js',
npm ERR! 'build'
npm ERR! ]
npm ERR! Not searching for unused variables given on the command line.
npm ERR! -- The C compiler identification is AppleClang 10.0.0.10001145
npm ERR! -- The CXX compiler identification is AppleClang 10.0.0.10001145
npm ERR! -- Detecting C compiler ABI info
npm ERR! -- Detecting C compiler ABI info - done
npm ERR! -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
npm ERR! -- Detecting C compile features
npm ERR! -- Detecting C compile features - done
npm ERR! -- Detecting CXX compiler ABI info
npm ERR! -- Detecting CXX compiler ABI info - done
npm ERR! -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
npm ERR! -- Detecting CXX compile features
npm ERR! -- Detecting CXX compile features - done
npm ERR! -- Found Protobuf: /usr/local/lib/libprotobuf.dylib (found version "3.19.4")
npm ERR! -- Configuring incomplete, errors occurred!
npm ERR! See also "/Users/benoit/.npm/_cacache/tmp/git-cloneaKQhsZ/build/CMakeFiles/CMakeOutput.log".
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! info TOOL Using Ninja generator, because ninja is available.
npm ERR! info CMD CONFIGURE
npm ERR! info RUN [
npm ERR! info RUN 'cmake',
npm ERR! info RUN '/Users/benoit/.npm/_cacache/tmp/git-cloneaKQhsZ',
npm ERR! info RUN '--no-warn-unused-cli',
npm ERR! info RUN '-G',
npm ERR! info RUN 'Ninja',
npm ERR! info RUN '-DCMAKE_JS_VERSION=6.3.0',
npm ERR! info RUN '-DCMAKE_BUILD_TYPE=Release',
npm ERR! info RUN '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/Users/benoit/.npm/_cacache/tmp/git-cloneaKQhsZ/build/Release',
npm ERR! info RUN '-DCMAKE_JS_INC=/Users/benoit/.cmake-js/node-x64/v16.14.0/include/node;/Users/benoit/.npm/_cacache/tmp/git-cloneaKQhsZ/node_modules/nan',
npm ERR! info RUN '-DCMAKE_JS_SRC=',
npm ERR! info RUN '-DNODE_RUNTIME=node',
npm ERR! info RUN '-DNODE_RUNTIMEVERSION=16.14.0',
npm ERR! info RUN '-DNODE_ARCH=x64',
npm ERR! info RUN '-DCMAKE_CXX_FLAGS=-D_DARWIN_USE_64_BIT_INODE=1 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DBUILDING_NODE_EXTENSION',
npm ERR! info RUN '-DCMAKE_SHARED_LINKER_FLAGS=-undefined dynamic_lookup'
npm ERR! info RUN ]
npm ERR! CMake Error at CMakeLists.txt:13 (add_subdirectory):
npm ERR! add_subdirectory given source "third_party/abseil-cpp" which is not an
npm ERR! existing directory.
npm ERR!
npm ERR!
npm ERR! CMake Error at CMakeLists.txt:16 (add_subdirectory):
npm ERR! add_subdirectory given source "third_party/boringssl/src" which is not an
npm ERR! existing directory.
npm ERR!
npm ERR!
npm ERR! CMake Error at CMakeLists.txt:19 (add_subdirectory):
npm ERR! add_subdirectory given source "third_party/zlib" which is not an existing
npm ERR! directory.
npm ERR!
npm ERR!
npm ERR! ERR! OMG Process terminated: 1
npm ERR! npm ERR! code 1
npm ERR! npm ERR! path /Users/benoit/.npm/_cacache/tmp/git-cloneaKQhsZ
npm ERR! npm ERR! command failed
npm ERR! npm ERR! command sh -c cmake-js build
npm ERR!
npm ERR! npm ERR! A complete log of this run can be found in:
npm ERR! npm ERR! /Users/benoit/.npm/_logs/2022-03-08T16_08_41_942Z-debug-0.log
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/benoit/.npm/_logs/2022-03-08T16_06_15_291Z-debug-0.log
Details in debug.log
```
0 verbose cli [
0 verbose cli '/usr/local/bin/node',
0 verbose cli '/usr/local/lib/node_modules/npm/bin/npm-cli.js',
0 verbose cli 'install',
0 verbose cli '--force',
0 verbose cli '--cache=/Users/benoit/.npm',
0 verbose cli '--prefer-offline=false',
0 verbose cli '--prefer-online=false',
0 verbose cli '--offline=false',
0 verbose cli '--no-progress',
0 verbose cli '--no-save',
0 verbose cli '--no-audit',
0 verbose cli '--include=dev',
0 verbose cli '--include=peer',
0 verbose cli '--include=optional',
0 verbose cli '--no-package-lock-only',
0 verbose cli '--no-dry-run'
0 verbose cli ]
1 info using npm@8.3.1
2 info using node@v16.14.0
3 timing npm:load:whichnode Completed in 1ms
4 timing config:load:defaults Completed in 3ms
5 timing config:load:file:/usr/local/lib/node_modules/npm/npmrc Completed in 1ms
6 timing config:load:builtin Completed in 1ms
7 timing config:load:cli Completed in 4ms
8 timing config:load:env Completed in 1ms
9 timing config:load:file:/Users/benoit/.npm/_cacache/tmp/git-cloneaKQhsZ/.npmrc Completed in 1ms
10 timing config:load:project Completed in 2ms
11 timing config:load:file:/Users/benoit/.npmrc Completed in 1ms
12 timing config:load:user Completed in 1ms
13 timing config:load:file:/usr/local/etc/npmrc Completed in 1ms
14 timing config:load:global Completed in 1ms
15 timing config:load:validate Completed in 1ms
16 timing config:load:credentials Completed in 1ms
17 timing config:load:setEnvs Completed in 1ms
18 timing config:load Completed in 18ms
19 timing npm:load:configload Completed in 19ms
20 timing npm:load:setTitle Completed in 26ms
21 timing config:load:flatten Completed in 5ms
22 timing npm:load:display Completed in 6ms
23 verbose logfile /Users/benoit/.npm/_logs/2022-03-08T16_08_41_942Z-debug-0.log
24 timing npm:load:logFile Completed in 19ms
25 timing npm:load:timers Completed in 0ms
26 timing npm:load:configScope Completed in 0ms
27 warn using --force Recommended protections disabled.
28 timing npm:load Completed in 74ms
29 timing arborist:ctor Completed in 1ms
30 silly logfile start cleaning logs, removing 1 files
31 timing idealTree:init Completed in 303ms
32 timing idealTree:userRequests Completed in 1ms
33 silly idealTree buildDeps
34 timing idealTree:#root Completed in 0ms
35 timing idealTree:buildDeps Completed in 2ms
36 timing idealTree:fixDepFlags Completed in 0ms
37 timing idealTree Completed in 307ms
38 timing reify:loadTrees Completed in 312ms
39 timing reify:diffTrees Completed in 13ms
40 silly reify moves {}
41 timing reify:retireShallow Completed in 1ms
42 timing reify:createSparse Completed in 100ms
43 timing reify:loadBundles Completed in 0ms
44 timing reifyNode:node_modules/setimmediate Completed in 3216ms
45 timing reifyNode:node_modules/natural-compare Completed in 3219ms
46 timing reifyNode:node_modules/is-iojs Completed in 3221ms
47 timing reifyNode:node_modules/invert-kv Completed in 3225ms
48 timing reifyNode:node_modules/wrap-ansi Completed in 3277ms
49 timing reifyNode:node_modules/wrappy Completed in 3278ms
50 timing reifyNode:node_modules/universalify Completed in 3296ms
51 timing reifyNode:node_modules/to-fast-properties Completed in 3297ms
52 timing reifyNode:node_modules/strip-bom Completed in 3308ms
53 timing reifyNode:node_modules/strip-json-comments Completed in 3309ms
54 timing reifyNode:node_modules/strip-ansi Completed in 3309ms
55 timing reifyNode:node_modules/string-width Completed in 3309ms
56 timing reifyNode:node_modules/shebang-command Completed in 3317ms
57 timing reifyNode:node_modules/resolve-from Completed in 3317ms
58 timing reifyNode:node_modules/slice-ansi Completed in 3319ms
59 timing reifyNode:node_modules/require-from-string Completed in 3319ms
60 timing reifyNode:node_modules/process-nextick-args Completed in 3347ms
61 timing reifyNode:node_modules/path-is-absolute Completed in 3348ms
62 timing reifyNode:node_modules/parent-module Completed in 3348ms
63 timing reifyNode:node_modules/path-exists Completed in 3348ms
64 timing reifyNode:node_modules/os-locale Completed in 3348ms
65 timing reifyNode:node_modules/p-locate Completed in 3349ms
66 timing reifyNode:node_modules/path-parse Completed in 3351ms
67 timing reifyNode:node_modules/p-try Completed in 3350ms
68 timing reifyNode:node_modules/number-is-nan Completed in 3350ms
69 timing reifyNode:node_modules/p-limit Completed in 3352ms
70 timing reifyNode:node_modules/once Completed in 3353ms
71 timing reifyNode:node_modules/ms Completed in 3353ms
72 timing reifyNode:node_modules/locate-path Completed in 3352ms
73 timing reifyNode:node_modules/is-extglob Completed in 3384ms
74 timing reifyNode:node_modules/is-fullwidth-code-point Completed in 3396ms
75 timing reifyNode:node_modules/ini Completed in 3410ms
76 timing reifyNode:node_modules/has-unicode Completed in 3411ms
77 timing reifyNode:node_modules/imurmurhash Completed in 3411ms
78 timing reifyNode:node_modules/inflight Completed in 3412ms
79 timing reifyNode:node_modules/find-up Completed in 3424ms
80 timing reifyNode:node_modules/fast-levenshtein Completed in 3425ms
81 timing reifyNode:node_modules/fs-minipass Completed in 3437ms
82 timing reifyNode:node_modules/eslint-plugin-import/node_modules/ms Completed in 3438ms
83 timing reifyNode:node_modules/eslint-import-resolver-node Completed in 3446ms
84 timing reifyNode:node_modules/duplexer2 Completed in 3498ms
85 timing reifyNode:node_modules/color-name Completed in 3499ms
86 timing reifyNode:node_modules/convert-source-map Completed in 3500ms
87 timing reifyNode:node_modules/code-point-at Completed in 3499ms
88 timing reifyNode:node_modules/camelcase Completed in 3500ms
89 timing reifyNode:node_modules/chownr Completed in 3507ms
90 timing reifyNode:node_modules/bindings Completed in 3511ms
91 timing reifyNode:node_modules/brace-expansion Completed in 3512ms
92 timing reifyNode:node_modules/decamelize Completed in 3513ms
93 timing reifyNode:node_modules/buffer-shims Completed in 3512ms
94 timing reifyNode:node_modules/ansi-regex Completed in 3519ms
95 timing reifyNode:node_modules/@types/json5 Completed in 3520ms
96 timing reifyNode:node_modules/@babel/highlight/node_modules/has-flag Completed in 3530ms
97 timing reifyNode:node_modules/@babel/highlight/node_modules/ansi-styles Completed in 3532ms
98 timing reifyNode:node_modules/@babel/highlight/node_modules/escape-string-regexp Completed in 3548ms
99 timing reifyNode:node_modules/word-wrap Completed in 3567ms
100 timing reifyNode:node_modules/y18n Completed in 3576ms
101 timing reifyNode:node_modules/window-size Completed in 3578ms
102 timing reifyNode:node_modules/table/node_modules/string-width Completed in 3590ms
103 timing reifyNode:node_modules/table/node_modules/strip-ansi Completed in 3591ms
104 timing reifyNode:node_modules/table/node_modules/ansi-regex Completed in 3591ms
105 timing reifyNode:node_modules/table/node_modules/is-fullwidth-code-point Completed in 3592ms
106 timing reifyNode:node_modules/string_decoder Completed in 3592ms
107 timing reifyNode:node_modules/slice-ansi/node_modules/is-fullwidth-code-point Completed in 3593ms
108 timing reifyNode:node_modules/splitargs Completed in 3594ms
109 timing reifyNode:node_modules/supports-color Completed in 3596ms
110 timing reifyNode:node_modules/shebang-regex Completed in 3604ms
111 timing reifyNode:node_modules/path-key Completed in 3604ms
112 timing reifyNode:node_modules/minizlib Completed in 3606ms
113 timing reifyNode:node_modules/minipass Completed in 3614ms
114 timing reifyNode:node_modules/lru-cache Completed in 3614ms
115 timing reifyNode:node_modules/lodash.padend Completed in 3614ms
116 timing reifyNode:node_modules/lodash.pad Completed in 3616ms
117 timing reifyNode:node_modules/lodash.padstart Completed in 3616ms
118 timing reifyNode:node_modules/lodash.truncate Completed in 3618ms
119 timing reifyNode:node_modules/lcid Completed in 3618ms
120 timing reifyNode:node_modules/is-glob Completed in 3634ms
121 timing reifyNode:node_modules/inherits Completed in 3635ms
122 timing reifyNode:node_modules/import-fresh Completed in 3634ms
123 timing reifyNode:node_modules/has-flag Completed in 3635ms
124 timing reifyNode:node_modules/glob-parent Completed in 3679ms
125 timing reifyNode:node_modules/eslint/node_modules/strip-ansi Completed in 3680ms
126 timing reifyNode:node_modules/eslint/node_modules/ansi-regex Completed in 3681ms
127 timing reifyNode:node_modules/escape-string-regexp Completed in 3682ms
128 timing reifyNode:node_modules/eslint/node_modules/strip-json-comments Completed in 3683ms
129 timing reifyNode:node_modules/core-util-is Completed in 3692ms
130 timing reifyNode:node_modules/callsites Completed in 3692ms
131 timing reifyNode:node_modules/cliui Completed in 3694ms
132 timing reifyNode:node_modules/astral-regex Completed in 3705ms
133 timing reifyNode:node_modules/@eslint/eslintrc/node_modules/strip-json-comments Completed in 3708ms
134 timing reifyNode:node_modules/@babel/traverse/node_modules/@babel/code-frame Completed in 3708ms
135 timing reifyNode:node_modules/@babel/template/node_modules/@babel/code-frame Completed in 3709ms
136 timing reifyNode:node_modules/@babel/highlight/node_modules/supports-color Completed in 3717ms
137 timing reifyNode:node_modules/@babel/highlight Completed in 3720ms
138 timing reifyNode:node_modules/@babel/helper-split-export-declaration Completed in 3721ms
139 timing reifyNode:node_modules/@babel/helper-hoist-variables Completed in 3722ms
140 timing reifyNode:node_modules/@babel/helper-function-name Completed in 3722ms
141 timing reifyNode:node_modules/@babel/helper-get-function-arity Completed in 3723ms
142 timing reifyNode:node_modules/@babel/helper-simple-access Completed in 3723ms
143 timing reifyNode:node_modules/@babel/helper-environment-visitor Completed in 3723ms
144 timing reifyNode:node_modules/@babel/core/node_modules/@babel/code-frame Completed in 3724ms
145 timing reifyNode:node_modules/util-deprecate Completed in 3749ms
146 timing reifyNode:node_modules/yallist Completed in 3752ms
147 timing reifyNode:node_modules/unzipper/node_modules/process-nextick-args Completed in 3768ms
148 timing reifyNode:node_modules/rimraf Completed in 3771ms
149 timing reifyNode:node_modules/lru-cache/node_modules/yallist Completed in 3782ms
150 timing reifyNode:node_modules/minimatch Completed in 3783ms
151 timing reifyNode:node_modules/isarray Completed in 3790ms
152 timing reifyNode:node_modules/jsonfile Completed in 3794ms
153 timing reifyNode:node_modules/js-tokens Completed in 3804ms
154 timing reifyNode:node_modules/fs.realpath Completed in 3843ms
155 timing reifyNode:node_modules/esrecurse Completed in 3857ms
156 timing reifyNode:node_modules/ansi-styles Completed in 4707ms
157 timing reifyNode:node_modules/balanced-match Completed in 4728ms
158 timing reifyNode:node_modules/@babel/code-frame Completed in 4871ms
159 timing reifyNode:node_modules/which Completed in 4909ms
160 timing reifyNode:node_modules/v8-compile-cache Completed in 4910ms
161 timing reifyNode:node_modules/url-join Completed in 4931ms
162 timing reifyNode:node_modules/safe-buffer Completed in 4951ms
163 timing reifyNode:node_modules/picocolors Completed in 4963ms
164 timing reifyNode:node_modules/mkdirp Completed in 4969ms
165 timing reifyNode:node_modules/has Completed in 5028ms
166 timing reifyNode:node_modules/file-entry-cache Completed in 5889ms
167 timing reifyNode:node_modules/eslint-plugin-prettier Completed in 6012ms
168 timing reifyNode:node_modules/duplexer2/node_modules/string_decoder Completed in 6061ms
169 timing reifyNode:node_modules/deep-extend Completed in 6063ms
170 timing reifyNode:node_modules/cross-spawn/node_modules/which Completed in 6064ms
171 timing reifyNode:node_modules/are-we-there-yet/node_modules/string_decoder Completed in 6064ms
172 timing reifyNode:node_modules/@humanwhocodes/config-array Completed in 6065ms
173 timing reifyNode:node_modules/@babel/traverse/node_modules/globals Completed in 6066ms
174 timing reifyNode:node_modules/@babel/highlight/node_modules/color-name Completed in 6066ms
175 timing reifyNode:node_modules/unzipper/node_modules/isarray Completed in 6085ms
176 timing reifyNode:node_modules/tar/node_modules/safe-buffer Completed in 6084ms
177 timing reifyNode:node_modules/type-check Completed in 6087ms
178 timing reifyNode:node_modules/punycode Completed in 6092ms
179 timing reifyNode:node_modules/lodash.merge Completed in 6093ms
180 timing reifyNode:node_modules/listenercount Completed in 6093ms
181 timing reifyNode:node_modules/flat-cache/node_modules/rimraf Completed in 6144ms
182 timing reifyNode:node_modules/eslint-config-standard Completed in 6153ms
183 timing reifyNode:node_modules/duplexer2/node_modules/isarray Completed in 6158ms
184 timing reifyNode:node_modules/are-we-there-yet/node_modules/isarray Completed in 6175ms
185 timing reifyNode:node_modules/acorn-jsx Completed in 6176ms
186 timing reifyNode:node_modules/@babel/helper-validator-option Completed in 6195ms
187 timing reifyNode:node_modules/progress Completed in 6225ms
188 timing reifyNode:node_modules/levn Completed in 6225ms
189 timing reifyNode:node_modules/follow-redirects Completed in 6258ms
190 timing reifyNode:node_modules/eslint-visitor-keys Completed in 6268ms
191 timing reifyNode:node_modules/eslint-plugin-standard Completed in 6279ms
192 timing reifyNode:node_modules/espree/node_modules/eslint-visitor-keys Completed in 6282ms
193 timing reifyNode:node_modules/eslint-utils/node_modules/eslint-visitor-keys Completed in 6282ms
194 timing reifyNode:node_modules/debug Completed in 6322ms
195 timing reifyNode:node_modules/delegates Completed in 6325ms
196 timing reifyNode:node_modules/color-convert Completed in 6325ms
197 timing reifyNode:node_modules/concat-map Completed in 6326ms
198 timing reifyNode:node_modules/buffers Completed in 6326ms
199 timing reifyNode:node_modules/ansi-colors Completed in 6326ms
200 timing reifyNode:node_modules/@babel/helper-validator-identifier Completed in 6329ms
201 timing reifyNode:node_modules/@babel/highlight/node_modules/color-convert Completed in 6329ms
202 timing reifyNode:node_modules/memory-stream Completed in 6375ms
203 timing reifyNode:node_modules/isexe Completed in 6391ms
204 timing reifyNode:node_modules/graceful-fs Completed in 6394ms
205 timing reifyNode:node_modules/globals Completed in 6397ms
206 timing reifyNode:node_modules/fast-diff Completed in 6399ms
207 timing reifyNode:node_modules/estraverse Completed in 6400ms
208 timing reifyNode:node_modules/esrecurse/node_modules/estraverse Completed in 6401ms
209 timing reifyNode:node_modules/esquery/node_modules/estraverse Completed in 6401ms
210 timing reifyNode:node_modules/chainsaw Completed in 6402ms
211 timing reifyNode:node_modules/@babel/highlight/node_modules/chalk Completed in 6402ms
212 timing reifyNode:node_modules/@babel/helper-module-imports Completed in 6500ms
213 timing reifyNode:node_modules/semver Completed in 6549ms
214 timing reifyNode:node_modules/optionator Completed in 6548ms
215 timing reifyNode:node_modules/ignore Completed in 6546ms
216 timing reifyNode:node_modules/functional-red-black-tree Completed in 6546ms
217 timing reifyNode:node_modules/glob Completed in 6546ms
218 timing reifyNode:node_modules/gensync Completed in 6546ms
219 timing reifyNode:node_modules/file-uri-to-path Completed in 6547ms
220 timing reifyNode:node_modules/flat-cache Completed in 6547ms
221 timing reifyNode:node_modules/escalade Completed in 6548ms
222 timing reifyNode:node_modules/deep-is Completed in 6553ms
223 timing reifyNode:node_modules/chalk Completed in 6555ms
224 timing reifyNode:node_modules/buffer-indexof-polyfill Completed in 6557ms
225 timing reifyNode:node_modules/are-we-there-yet Completed in 6557ms
226 timing reifyNode:node_modules/supports-preserve-symlinks-flag Completed in 6576ms
227 timing reifyNode:node_modules/npmlog Completed in 6574ms
228 timing reifyNode:node_modules/node-releases Completed in 6574ms
229 timing reifyNode:node_modules/jsesc Completed in 6572ms
230 timing reifyNode:node_modules/is-weakref Completed in 6573ms
231 timing reifyNode:node_modules/is-shared-array-buffer Completed in 6574ms
232 timing reifyNode:node_modules/gauge Completed in 6571ms
233 timing reifyNode:node_modules/eslint-plugin-node/node_modules/ignore Completed in 6572ms
234 timing reifyNode:node_modules/emoji-regex Completed in 6575ms
235 timing reifyNode:node_modules/cross-spawn Completed in 6576ms
236 timing reifyNode:node_modules/json-schema-traverse Completed in 6587ms
237 timing reifyNode:node_modules/is-regex Completed in 6588ms
238 timing reifyNode:node_modules/is-core-module Completed in 6587ms
239 timing reifyNode:node_modules/eslint-plugin-node/node_modules/semver Completed in 6585ms
240 timing reifyNode:node_modules/eslint-import-resolver-node/node_modules/debug Completed in 6626ms
241 timing reifyNode:node_modules/eslint-module-utils/node_modules/debug Completed in 6627ms
242 timing reifyNode:node_modules/define-properties Completed in 6628ms
243 timing reifyNode:node_modules/@babel/helper-compilation-targets/node_modules/semver Completed in 6654ms
244 timing reifyNode:node_modules/@babel/helper-module-transforms Completed in 6656ms
245 timing reifyNode:node_modules/@babel/eslint-parser/node_modules/semver Completed in 6657ms
246 timing reifyNode:node_modules/@babel/core/node_modules/semver Completed in 6657ms
247 timing reifyNode:node_modules/side-channel Completed in 6674ms
248 timing reifyNode:node_modules/is-string Completed in 6672ms
249 timing reifyNode:node_modules/is-callable Completed in 6672ms
250 timing reifyNode:node_modules/is-bigint Completed in 6672ms
251 timing reifyNode:node_modules/has-bigints Completed in 6671ms
252 timing reifyNode:node_modules/fast-deep-equal Completed in 6671ms
253 timing reifyNode:node_modules/esutils Completed in 6671ms
254 timing reifyNode:node_modules/@jridgewell/sourcemap-codec Completed in 6671ms
255 timing reifyNode:node_modules/@babel/template Completed in 6695ms
256 timing reifyNode:node_modules/@jridgewell/resolve-uri Completed in 6695ms
257 timing reifyNode:node_modules/@babel/helper-compilation-targets Completed in 6695ms
258 timing reifyNode:node_modules/unbox-primitive Completed in 6712ms
259 timing reifyNode:node_modules/which-boxed-primitive Completed in 6720ms
260 timing reifyNode:node_modules/prelude-ls Completed in 6722ms
261 timing reifyNode:node_modules/prettier-linter-helpers Completed in 6723ms
262 timing reifyNode:node_modules/is-symbol Completed in 6720ms
263 timing reifyNode:node_modules/is-number-object Completed in 6752ms
264 timing reifyNode:node_modules/is-negative-zero Completed in 6754ms
265 timing reifyNode:node_modules/is-date-object Completed in 6754ms
266 timing reifyNode:node_modules/is-boolean-object Completed in 6756ms
267 timing reifyNode:node_modules/internal-slot Completed in 6757ms
268 timing reifyNode:node_modules/function-bind Completed in 6755ms
269 timing reifyNode:node_modules/get-symbol-description Completed in 6755ms
270 timing reifyNode:node_modules/eslint-config-prettier Completed in 6762ms
271 timing reifyNode:node_modules/rc Completed in 6795ms
272 timing reifyNode:node_modules/object-keys Completed in 6797ms
273 timing reifyNode:node_modules/get-intrinsic Completed in 6797ms
274 timing reifyNode:node_modules/call-bind Completed in 6815ms
275 timing reifyNode:node_modules/unzipper Completed in 6844ms
276 timing reifyNode:node_modules/has-tostringtag Completed in 6877ms
277 timing reifyNode:node_modules/browserslist Completed in 6894ms
278 timing reifyNode:node_modules/@jridgewell/trace-mapping Completed in 6910ms
279 timing reifyNode:node_modules/text-table Completed in 6942ms
280 timing reifyNode:node_modules/string.prototype.trimend Completed in 6945ms
281 timing reifyNode:node_modules/string.prototype.trimstart Completed in 6946ms
282 timing reifyNode:node_modules/espree Completed in 6951ms
283 timing reifyNode:node_modules/eslint-plugin-import/node_modules/doctrine Completed in 6969ms
284 timing reifyNode:node_modules/doctrine Completed in 6979ms
285 timing reifyNode:node_modules/has-symbols Completed in 7024ms
286 timing reifyNode:node_modules/ansi Completed in 7053ms
287 timing reifyNode:node_modules/table/node_modules/json-schema-traverse Completed in 7079ms
288 timing reifyNode:node_modules/readable-stream Completed in 7078ms
289 timing reifyNode:node_modules/json-stable-stringify-without-jsonify Completed in 7075ms
290 timing reifyNode:node_modules/big-integer Completed in 7101ms
291 timing reifyNode:node_modules/@babel/compat-data Completed in 7113ms
292 timing reifyNode:node_modules/eslint-module-utils Completed in 7153ms
293 timing reifyNode:node_modules/fast-json-stable-stringify Completed in 7254ms
294 timing reifyNode:node_modules/eslint-plugin-import/node_modules/debug Completed in 7253ms
295 timing reifyNode:node_modules/array.prototype.flat Completed in 7253ms
296 timing reifyNode:node_modules/@humanwhocodes/object-schema Completed in 7253ms
297 timing reifyNode:node_modules/object.values Completed in 7269ms
298 timing reifyNode:node_modules/eslint-scope Completed in 7262ms
299 timing reifyNode:node_modules/sprintf-js Completed in 7292ms
300 timing reifyNode:node_modules/json5 Completed in 7290ms
301 timing reifyNode:node_modules/es-to-primitive Completed in 7296ms
302 timing reifyNode:node_modules/electron-to-chromium Completed in 7312ms
303 timing reifyNode:node_modules/array-includes Completed in 7312ms
304 timing reifyNode:node_modules/minimist Completed in 7338ms
305 timing reifyNode:node_modules/@ampproject/remapping Completed in 7338ms
306 timing reifyNode:node_modules/regexpp Completed in 7382ms
307 timing reifyNode:node_modules/@babel/helpers Completed in 7430ms
308 timing reifyNode:node_modules/esprima Completed in 7452ms
309 timing reifyNode:node_modules/flatted Completed in 7454ms
310 timing reifyNode:node_modules/eslint-utils Completed in 7485ms
311 timing reifyNode:node_modules/@babel/eslint-parser Completed in 7494ms
312 timing reifyNode:node_modules/traverse Completed in 7511ms
313 timing reifyNode:node_modules/eslint-plugin-promise Completed in 7514ms
314 timing reifyNode:node_modules/object.assign Completed in 7538ms
315 timing reifyNode:node_modules/fstream Completed in 7533ms
316 timing reifyNode:node_modules/unzipper/node_modules/readable-stream Completed in 7596ms
317 timing reifyNode:node_modules/cmake-js Completed in 7629ms
318 timing reifyNode:node_modules/duplexer2/node_modules/readable-stream Completed in 7697ms
319 timing reifyNode:node_modules/are-we-there-yet/node_modules/readable-stream Completed in 7697ms
320 timing reifyNode:node_modules/@babel/generator Completed in 7731ms
321 timing reifyNode:node_modules/tar Completed in 7757ms
322 timing reifyNode:node_modules/@babel/core/node_modules/json5 Completed in 7747ms
323 timing reifyNode:node_modules/yargs Completed in 7812ms
324 timing reifyNode:node_modules/argparse Completed in 7845ms
325 timing reifyNode:node_modules/node-addon-api Completed in 7857ms
326 timing reifyNode:node_modules/object-inspect Completed in 7862ms
327 timing reifyNode:node_modules/@eslint/eslintrc Completed in 7859ms
328 timing reifyNode:node_modules/binary Completed in 7876ms
329 timing reifyNode:node_modules/tsconfig-paths Completed in 7902ms
330 timing reifyNode:node_modules/type-fest Completed in 7964ms
331 timing reifyNode:node_modules/eslint/node_modules/semver Completed in 8086ms
332 timing reifyNode:node_modules/@babel/traverse Completed in 8179ms
333 timing reifyNode:node_modules/enquirer Completed in 8196ms
334 timing reifyNode:node_modules/source-map Completed in 8258ms
335 timing reifyNode:node_modules/js-yaml Completed in 8303ms
336 timing reifyNode:node_modules/esquery Completed in 8308ms
337 timing reifyNode:node_modules/axios Completed in 8342ms
338 timing reifyNode:node_modules/uri-js Completed in 8451ms
339 timing reifyNode:node_modules/nan Completed in 8457ms
340 timing reifyNode:node_modules/acorn Completed in 8459ms
341 timing reifyNode:node_modules/fs-extra Completed in 8523ms
342 timing reifyNode:node_modules/bluebird Completed in 8523ms
343 timing reifyNode:node_modules/@babel/parser Completed in 8659ms
344 timing reifyNode:node_modules/eslint-plugin-node Completed in 8706ms
345 timing reifyNode:node_modules/@babel/core Completed in 8721ms
346 timing reifyNode:node_modules/eslint-plugin-es Completed in 8726ms
347 timing reifyNode:node_modules/table Completed in 8871ms
348 timing reifyNode:node_modules/ajv Completed in 9043ms
349 timing reifyNode:node_modules/eslint-plugin-import Completed in 9318ms
350 timing reifyNode:node_modules/resolve Completed in 9368ms
351 timing reifyNode:node_modules/@babel/types Completed in 9460ms
352 timing reifyNode:node_modules/table/node_modules/ajv Completed in 10665ms
353 timing reifyNode:node_modules/eslint Completed in 11036ms
354 timing reifyNode:node_modules/caniuse-lite Completed in 12042ms
355 timing reifyNode:node_modules/lodash Completed in 12254ms
356 timing reifyNode:node_modules/es-abstract Completed in 12265ms
357 timing reifyNode:node_modules/prettier Completed in 12281ms
358 timing reify:unpack Completed in 12287ms
359 timing reify:unretire Completed in 0ms
360 timing build:queue Completed in 13ms
361 timing build:link:node_modules/@babel/parser Completed in 18ms
362 timing build:link:node_modules/@babel/core/node_modules/json5 Completed in 15ms
363 timing build:link:node_modules/cross-spawn/node_modules/which Completed in 15ms
364 timing build:link:node_modules/@babel/eslint-parser/node_modules/semver Completed in 15ms
365 timing build:link:node_modules/flat-cache/node_modules/rimraf Completed in 15ms
366 timing build:link:node_modules/eslint/node_modules/semver Completed in 15ms
367 timing build:link:node_modules/@babel/helper-compilation-targets/node_modules/semver Completed in 15ms
368 timing build:link:node_modules/acorn Completed in 18ms
369 timing build:link:node_modules/eslint-plugin-node/node_modules/semver Completed in 15ms
370 timing build:link:node_modules/cmake-js Completed in 18ms
371 timing build:link:node_modules/eslint Completed in 18ms
372 timing build:link:node_modules/eslint-config-prettier Completed in 18ms
373 timing build:link:node_modules/browserslist Completed in 18ms
374 timing build:link:node_modules/esprima Completed in 17ms
375 timing build:link:node_modules/js-yaml Completed in 17ms
376 timing build:link:node_modules/jsesc Completed in 17ms
377 timing build:link:node_modules/json5 Completed in 18ms
378 timing build:link:node_modules/rc Completed in 17ms
379 timing build:link:node_modules/mkdirp Completed in 18ms
380 timing build:link:node_modules/rimraf Completed in 17ms
381 timing build:link:node_modules/prettier Completed in 18ms
382 timing build:link:node_modules/semver Completed in 17ms
383 timing build:link:node_modules/which Completed in 17ms
384 timing build:link:node_modules/@babel/core/node_modules/semver Completed in 16ms
385 timing build:link:node_modules/resolve Completed in 17ms
386 timing build:link:node_modules/window-size Completed in 17ms
387 timing build:link Completed in 20ms
388 timing build:deps Completed in 34ms
389 timing build Completed in 34ms
390 timing reify:build Completed in 35ms
391 timing reify:trash Completed in 0ms
392 timing reify Completed in 12803ms
393 timing command:install Completed in 19076ms
394 verbose stack Error: command failed
394 verbose stack at ChildProcess.
This is indeed puzzling. I'm probably doing something wrong? Following your new commands, I get the following output:
> @fails-components/webtransport@0.0.0 install
> cmake-js build
[ '/usr/local/bin/node', '/usr/local/bin/cmake-js', 'build' ]
info TOOL Using Ninja generator, because ninja is available.
info CMD CONFIGURE
info RUN [
info RUN 'cmake',
info RUN '/Users/benoit/dev/webtransport',
info RUN '--no-warn-unused-cli',
info RUN '-G',
info RUN 'Ninja',
info RUN '-DCMAKE_JS_VERSION=6.3.0',
info RUN '-DCMAKE_BUILD_TYPE=Release',
info RUN '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/Users/benoit/dev/webtransport/build/Release',
info RUN '-DCMAKE_JS_INC=/Users/benoit/.cmake-js/node-x64/v16.14.0/include/node',
info RUN '-DCMAKE_JS_SRC=',
info RUN '-DNODE_RUNTIME=node',
info RUN '-DNODE_RUNTIMEVERSION=16.14.0',
info RUN '-DNODE_ARCH=x64',
info RUN '-DCMAKE_CXX_FLAGS=-D_DARWIN_USE_64_BIT_INODE=1 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DBUILDING_NODE_EXTENSION',
info RUN '-DCMAKE_SHARED_LINKER_FLAGS=-undefined dynamic_lookup'
info RUN ]
Not searching for unused variables given on the command line.
-- The C compiler identification is AppleClang 10.0.0.10001145
-- The CXX compiler identification is AppleClang 10.0.0.10001145
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found Perl: /usr/bin/perl (found version "5.18.2")
-- The ASM compiler identification is AppleClang
-- Found assembler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
CMake Deprecation Warning at third_party/zlib/CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - failed
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Renaming
-- /Users/benoit/dev/webtransport/third_party/zlib/zconf.h
-- to 'zconf.h.included' because this file is included with zlib
-- but CMake generates it automatically in the build directory.
-- Found Protobuf: /usr/local/lib/libprotobuf.dylib (found version "3.19.4")
node:internal/modules/cjs/loader:936
throw err;
^
Error: Cannot find module 'node-addon-api'
Require stack:
- /Users/benoit/dev/webtransport/[eval]
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
at Function.Module._load (node:internal/modules/cjs/loader:778:27)
at Module.require (node:internal/modules/cjs/loader:1005:19)
at require (node:internal/modules/cjs/helpers:102:18)
at [eval]:1:1
at Script.runInThisContext (node:vm:129:12)
at Object.runInThisContext (node:vm:305:38)
at node:internal/process/execution:75:19
at [eval]-wrapper:6:22
at evalScript (node:internal/process/execution:74:60) {
code: 'MODULE_NOT_FOUND',
requireStack: [ '/Users/benoit/dev/webtransport/[eval]' ]
}
CMake Error at CMakeLists.txt:825 (string):
string sub-command REPLACE requires at least four arguments.
CMake Error at CMakeLists.txt:826 (string):
string sub-command REPLACE requires at least four arguments.
-- Configuring incomplete, errors occurred!
See also "/Users/benoit/dev/webtransport/build/CMakeFiles/CMakeOutput.log".
See also "/Users/benoit/dev/webtransport/build/CMakeFiles/CMakeError.log".
ERR! OMG Process terminated: 1
So I ran a simple npm install
to install the node_modules, and I got:
> @fails-components/webtransport@0.0.0 install
> cmake-js build
[
'/usr/local/bin/node',
'/Users/benoit/dev/webtransport/node_modules/.bin/cmake-js',
'build'
]
info TOOL Using Ninja generator, because ninja is available.
info CMD BUILD
info RUN [
info RUN 'cmake',
info RUN '--build',
info RUN '/Users/benoit/dev/webtransport/build',
info RUN '--config',
info RUN 'Release'
info RUN ]
ninja: error: loading 'build.ninja': No such file or directory
ERR! OMG Process terminated: 1
npm ERR! code 1
npm ERR! path /Users/benoit/dev/webtransport
npm ERR! command failed
npm ERR! command sh -c cmake-js build
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/benoit/.npm/_logs/2022-03-08T16_20_59_557Z-debug-0.log
The build directory is like this:
Thank you for your help!
With v1 it is puzzling. The problem is that the libs are git submodules. The new npm command should also fetch the submodules, but for some reasons CMake say the diretories of the three submodules
third_party/abseil-cpp,
third_party/boringssl/srcand
third_party/zlib` are empty.
While trying to debug, I believe I had it once that it worked today, I will try to find the right commit hash (but I may be mistaken). npm should have handled the submodules, I can not do anything, since the git stuff is already gone in the preinstall script.
For v2, you are right npm run install
is wrong npm install
is right.
Here is my build directory:
But I think in CMakeError.log is the clue:
Change Dir: /Users/benoit/dev/webtransport/build/CMakeFiles/CMakeTmp ```
I can only guess that one of the libraries contain a test for off64_t type and it is not present on Mac OS X. But I have also no clue, which lib and which its cmake file it is. You can google about this error I found for example also OpenCV with this issue... and some google products long time ago....
Any way if it does not help, you may consider running it in a docker linux container as I do it under windows, I assume that the application is finally running under linux anyway.
Ok regarding v1, there seems to be a problem in npm since version 7 (https://github.com/npm/cli/issues/2774) that breaks submodule support in npm. Maybe you can revert back to npm 6, or pursue v2 further.
I have to think, if I start releasing prerelease packages, but actually I do not like this as long as the packages is not really tested.
Regarding v2, you may look into the CMakelists.txt file, there you may want to comment add_subdirectory
and see if the error concernung off64_t goes away, if any of them is excluded. Of course other errors will pop up, but in this way we can identify if any of these libs causes CMake to do this test. Protobuf may be another cause.
hi again,
v2 seems to work a little further .... but the main problem is the code relies on the epoll implementation in quiche which is not supported in OSX :( The dockerization of the server could be an option indeed! But I have a pretty tight session interaction with an existing server that I need to maintain.
"Working" steps:
git clone --recurse-submodules https://github.com/fails-components/webtransport.git
cd webtransport
npm install # this will throw an error but it will install the node_modules
cmake-js compile
cmake-js compile
will throw errors on all epoll
related stuff... (noticeably sys/epoll.h
not found). Ex:
/Users/benoit/dev/webtransport/third_party/quiche/epoll_server/simple_epoll_server.h:38:10: fatal error: 'sys/epoll.h' file not found
#include <sys/epoll.h>
^~~~~~~~~~~~~
It is possible to comment the simple_epoll_server out of the CMakeList.txt file. But the http3Server seems to be deeply rooted in EPoll.
I had to use clang v13 to avoid some compilation errors with quiche (I used the v13.0.1 binary distribution for OSX on llvm website). The flag CXXFLAGS="--std=c++17" fixed some issues but others remained.
I havent seen the off64_t error again but will look into your comments if this happens again.
Thanks again for your help !
Ok regarding v1, there seems to be a problem in npm since version 7 (npm/cli#2774) that breaks submodule support in npm. Maybe you can revert back to npm 6, or pursue v2 further.
Thanks this is good to know.
Hi
hi again,
v2 seems to work a little further .... but the main problem is the code relies on the epoll implementation in quiche which is not supported in OSX :( The dockerization of the server could be an option indeed! But I have a pretty tight session interaction with an existing server that I need to maintain.
Ok, and epoll is not available on OS X? Then there is probably not an easy way to fix this, of course, one could try to write and replacement (kqueue) , if a similar mechanism exists, so something that wakes up, if some thing happens to a fd. But for me without a mac, it has zero priority. And it probably needs some time. And the tight session interaction does this mean your server is on OS X...? I thought most server will eventually run on linux.
"Working" steps:
git clone --recurse-submodules https://github.com/fails-components/webtransport.git cd webtransport npm install # this will throw an error but it will install the node_modules cmake-js compile
cmake-js compile
will throw errors on allepoll
related stuff... (noticeablysys/epoll.h
not found). Ex:/Users/benoit/dev/webtransport/third_party/quiche/epoll_server/simple_epoll_server.h:38:10: fatal error: 'sys/epoll.h' file not found #include <sys/epoll.h> ^~~~~~~~~~~~~
It is possible to comment the simple_epoll_server out of the CMakeList.txt file. But the http3Server seems to be deeply rooted in EPoll.
Yes the whole architecture of libquiche seems to be event driven by the epollserver, but of course it should be possible to be replaced.
Note1
I had to use clang v13 to avoid some compilation errors with quiche (I used the v13.0.1 binary distribution for OSX on llvm website). The flag CXXFLAGS="--std=c++17" fixed some issues but others remained.
that means
set(CXX_STANDARD 17)
in cmakelists.txt does not set it to c++17 ?Note2
I havent seen the off64_t error again but will look into your comments if this happens again.
Thanks again for your help !
Hi,
Understood :) I forked the project and will try to adapt the server to libuv, libev or kQueue. I've read Nodejs is using libuv? It would be logical to stay with this maybe? I don't have any experience with this but I will let you know if I finished something that is somewhat running....
Thanks again !
that means
set(CXX_STANDARD 17)
in cmakelists.txt does not set it to c++17 ?
Hmm it could have been that I ran sequential compiles and the first errors were skipped, only to come back again at the "end". I would have to look more into this.... I could be wrong.
Hi,
Understood :) I forked the project and will try to adapt the server to libuv, libev or kQueue. I've read Nodejs is using libuv? It would be logical to stay with this maybe? I don't have any experience with this but I will let you know if I finished something that is somewhat running....
Thanks again !
I do not think that you have to do so much changes. Please look into: https://github.com/fails-components/webtransport/blob/master/platform/net/quic/platform/impl/quic_epoll_impl.h Here you can insert some preprocessor stuff to branch to other epoll objects for MacOS. These object you can then also place in the platform impl dirs of the package. I think can just copy the epoll server code from libquiche and change the few system calls to kqueue, rename it and that's it. (If it is compatible...) Then some changes to the cmakefile... and that's it. I do not think there are many other platform dependent things lurking, most is STL. Edit: I just saw that libuv is using epoll and kqueue in the background, so yes an epoll server with libuv would be great, the file where the changes happen remain the same....
I do not think that you have to do so much changes. Please look into: https://github.com/fails-components/webtransport/blob/master/platform/net/quic/platform/impl/quic_epoll_impl.h Here you can insert some preprocessor stuff to branch to other epoll objects for MacOS. Hopefully yes ! Thank you for the pointer. I'm having a little bit of an issue with the compilation of quiche at the moment but if that is solved, I hope to be able to adapt to a libuv implementation.
Note: I understood the reason for the build.ninja not found error: when the configure script errors before starting to compile the files, on the next run, it will try to go to the next step and not find the build.ninja file. You have to remove the build directory (after solving the original issue) so that it builds properly.
that means
set(CXX_STANDARD 17)
in cmakelists.txt does not set it to c++17 ?
So it seems the flag was not taken into account on Mac. I had to add this line in CMakeList.txt:
set_property(TARGET gquiche PROPERTY CXX_STANDARD 17)
Note: I understood the reason for the build.ninja not found error: when the configure script errors before starting to compile the files, on the next run, it will try to go to the next step and not find the build.ninja file. You have to remove the build directory (after solving the original issue) so that it builds properly.
You can just run npm run rebuild
, it cleans all old files.
that means
set(CXX_STANDARD 17)
in cmakelists.txt does not set it to c++17 ?So it seems the flag was not taken into account on Mac. I had to add this line in CMakeList.txt:
set_property(TARGET gquiche PROPERTY CXX_STANDARD 17)
Compiles also fine under linux, I have commited it.
Great. Glad I could help. I've actually added to the webtransport project/target as well.
Great. Glad I could help. I've actually added to the webtransport project/target as well.
Ok, I have added this as well.
Hi ! I think the code is working on Mac OSX. There were some small issues with the CMake file on mac (some even dependent on the cmake version) and needing some rewriting of the simple_server with libuv. But I think I am seeing the right messages here on Chrome. It also works "as this" with no special options and certbot certificates. Let me know if you want me to detail here the main changes? Again, thanks so much for this work and your contribution !!
Hi! Thanks, I have used your code as inspiration for replacing the epoll server entirely with libuv (actually you do not have to build it, it is readily available for a node c++ addon). The corresponding commit is https://github.com/fails-components/webtransport/commit/52a462c473f11f1f6a8c7dd151911038d55fac16 .
I tried to find all changes, I hope I did not miss any. But in the CMake file I left out: ``` target_include_directories(gquiche PUBLIC /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk /usr/local/include) # on older CMake -isysroot removes /usr/local/include ```` I think this was debugging code. Please tell me if I missed something.
Hello. Great, thank you. Looking at the changes I think this looks great.
CMake:
One thing though: it would be nice to add the libuv submodule. I have now installed libuv on my mac but I know it was not there before, so it would be nice to add this submodule as well, dont you think? This would avoid an extra step of installation for people checking out the repository. Is it always installed on Linux?
I will give a fresh install a try shortly and let you know.
Hi,
Hello. Great, thank you. Looking at the changes I think this looks great.
CMake:
- I will try again without the SDK path but I remember I had an error at some point which pushed me to include this. But I havent tried again without and it could have been related to something else. I will let you know.
But anyway, if it is needed, hard coding the path is not an option, since it may change. So if it is missing, a CMake variable or mechnaism, that will determine the path should be used, otherwise it will break if the path changes or is different for different versions.
- Same for the /usr/local/include. On another machine I had to include this because clang (or maybe CMake) was removing this directory silently when the -isysroot option was activated. It wouldn't hurt to leave this here anyhow.
Here it is the same, we have to use a mechanism, which adds the missing part of the object automatically during configure, similar to the stuff for example for protobuf.
One thing though: it would be nice to add the libuv submodule. I have now installed libuv on my mac but I know it was not there before, so it would be nice to add this submodule as well, dont you think? This would avoid an extra step of installation for people checking out the repository. Is it always installed on Linux?
No it is not always there for linux. But it is an integral part of node itself (and is also used by nan internally) and it is important to use the same lib in this case (for the other libs in the submodules it is important, that we use a different http3 capable version), so that not problems with mixing two libs occurs. But I had to add:
target_include_directories(gquiche
....
PUBLIC ${CMAKE_JS_INC}
)
so that all parts used for nan (and libuv is one of them) are already present when building lib quiche.
Nice! 👯♂️ I will clone and report asap
Hello! I tried a fresh build and I encountered just one issue with the protobuf library not found. This was related to the /usr/local/include directory being added to the gquiche library.
This fixed the problem, around line 807 in CMakeLists.txt:
target_include_directories(gquiche PUBLIC /usr/local/include)
target_link_directories(gquiche PUBLIC /usr/local/lib)
While investigating I found out that the -isysroot option added by CMake on OSX may disable the lookout for this directory: here
I think there is a fix with that NLOPT_INCLUDE_DIR
thing, but it seemed okay to add those directories as they are pretty much created on any OSX distribution and this is the default path where protobuf installs when running a make install
But I agree its not very clean :/
Other than that, the test ran without any issue once this modification was brought to CMakeLists :)
Actually, I realize CMakeLists.txt has a section to find the Protobuf install. It should be possible to re-use ${Protobuf_LIBRARIES}
and ${Protobuf_INCLUDE_DIRS}
: here
This seems to work: line 22:
set(Protobuf_USE_STATIC_LIBS ON)
~line 795:
target_link_libraries(gquiche
absl::base
absl::int128
absl::strings
absl::str_format
absl::flat_hash_map
absl::any
absl::synchronization
${Protobuf_LIBRARIES}
ssl crypto)
if(APPLE)
target_compile_definitions (gquiche PRIVATE __APPLE_USE_RFC_3542)
target_include_directories(gquiche PUBLIC ${Protobuf_INCLUDE_DIRS})
endif()
The reason why I added set(Protobuf_USE_STATIC_LIBS ON)
is because the test was hanging mid-way without it. I suspect the Apple Security stuff to lock the dependent protobuf DYLIB. But it was not showing it for some reason. Using the option made the test run entirely.
EDIT: still seems to be hanging sometimes.
This becomes unrelated to this thread but, are you seeing some of the following issues?
1) sometimes the test is not completing properly: it hangs at the datagram stage (I realized this could be the case after some refreshes in the forked version as well)
2) segfault on a hard refresh from the client side (Chrome) (not happening on the forked version)
3) TypeError: this.streams.delete is not a function
error on webtransport.js:238
(not happening on forked version)
This becomes unrelated to this thread but, are you seeing some of the following issues?
1. sometimes the test is not completing properly: it hangs at the datagram stage (I realized this could be the case after some refreshes in the forked version as well)
I did not see this, or I did not pay attention. I will have a lock at the weekend.
2. segfault on a hard refresh from the client side (Chrome) (not happening on the forked version)
I have seen this as well, but not after I made some additional changes/fixes, however I forget to push. I will look in the evening or tomorrow morning to push it.
3. `TypeError: this.streams.delete is not a function`error on `webtransport.js:238` (not happening on forked version)
Yes, I found this and fixed it, but I forgot to push. (Hopefully this is the reason...)
Now, the forgotten fixes are commited and pushed. Hopefully this fixes the segfault. But anyway there might be still issues related to garbage collection and object deletion. So if anything odd happens please post it.
${Protobuf_INCLUDE_DIRS}
Actually, I realize CMakeLists.txt has a section to find the Protobuf install. It should be possible to re-use
${Protobuf_LIBRARIES}
and${Protobuf_INCLUDE_DIRS}
: hereThis seems to work: line 22:
set(Protobuf_USE_STATIC_LIBS ON)
~line 795:
target_link_libraries(gquiche absl::base absl::int128 absl::strings absl::str_format absl::flat_hash_map absl::any absl::synchronization ${Protobuf_LIBRARIES} ssl crypto) if(APPLE) target_compile_definitions (gquiche PRIVATE __APPLE_USE_RFC_3542) target_include_directories(gquiche PUBLIC ${Protobuf_INCLUDE_DIRS}) endif()
~The reason why I added
set(Protobuf_USE_STATIC_LIBS ON)
is because the test was hanging mid-way without it. I suspect the Apple Security stuff to lock the dependent protobuf DYLIB. But it was not showing it for some reason. Using the option made the test run entirely.~EDIT: still seems to be hanging sometimes.
I have added these chnages as well. Static linking is fine, in this way binaries will also work on systems with out protobuf installed, which may be beneficial. I think no we got the mac os x case.
Hi I ran a fresh clone. Everything built as expected and the test ran fully! Closing the issue. I noticed some hangs on some refreshes with Chrome, but no segfault, no webtransport.js error and some of the next refreshes could recover fully.
I noticed some hangs on some refreshes with Chrome, but no segfault, no webtransport.js error and some of the next refreshes could recover fully.
One final note, this was probably caused by faulty example code for the client side, if buffers got merged, which is allowed according to spec. This is now fixed.
Oh that's great ! Thanks. I should start with webtransport next week. I'll make sure to let you know if I notice anything.
Test is working consistently on osx as well :)
I know the issue is closed, but library quiche changed the eventloop default implementation, which should now cover mac os x as well. Would you test the following branch on mac os x before I merge it with master: https://github.com/fails-components/webtransport/tree/eventloop ? Thanks!
No problem: will check the branch today or tomorrow and let you know.
I tried with the eventloop
branch and I get a handshake error in the browser. I dont see any error on the server side.
Also, I don't get this error with the master
branch.
Connection failed. WebTransportError: Opening handshake failed.
Connection closed abruptly
I tried running Chrome with the "ignore errors" options, without luck (it seems the ignore-certificate-errors-spki-list
argument is not recognized by chrome v102 though). I have been using a certbot certificate which has been working fine so far (and still works in the master
branch).
I guess there is an event sequence that is not fully completed while negotiating? Any idea where I should add breakpoints to figure this out?
Well, opening handshake is probably just a sign, that the event loop got stuck.
There is in terms of certifcates no difference between the branches.
You can also run npm test
and see if this is working.
Regarding the break points in httpeventloop.h the WakeUpHelper's ReadPipeCallback' OnSocketEvent, has a read and it can be an issue, if the read is blocking and the setNonBlocking did work on Mac....
Otherwise the OnSocketEvent of server and client can be a good place... and of course the eventloop object of the libary.
Ok. I'll try to check those breakpoints out. test is giving the following output:
start generating self signed certificate
start Http3Server and startup echo tests
start GlobalEventLoop
createGlobalEventLoop
server started now wait 2 seconds
now startup client
wait for client to be ready
tests took too long, probably hanging
This means it is hanging in the polling, so I suspect really that the unblocking of the fd is not working on mac...
Still trying to investigate on this ... Im having trouble breaking on the breakpoints above but I am also encountering an issue with my app. Will report as soon as this is figured out.
Maybe use a printf....., also yesterday someone suggested me, that may be other portable libs are also an option.
Yes will do that. You mean for Quiche ? or libuv equivalent? So quiche has definitively moved to this new eventloop paradigm?
A libuv equivalent, quiche is currently moving towards multiple options for eventloop, there is also an option without additional event loop. I got the info yesterday not sure, what I will do.
Ha Great. A lot of action seems to be going on around this event mechanism! Great to know Quiche is offering options.
hi Marten, Im coming back with not much more info sadly. Ive added printf's where they belong, at:
printf("Http3Server::CreateUDPSocketAndListen") ;
printf("Http3Server::startServerInt") ;
printf("Http3Server::OnSocketEvent") ;
printf("WakeUpHelper::OnSocketEvent\n");
Nothing prints. I've also added breakpoints there and it is not passing through. Could there be some syntax that changed (in WebTransport) in the JS and the client is disconnecting... ? I can only break in the creation of the Http3Server (ran 2 times likely because of the 2 channels in my app).
When in VS Code in debug mode, I get an ECONNRESET Error printed (sometimes a variant is 'socket hang'). But I dont get it when running the same code through the command line. This inconsistence is also a little weird to me - it could have to do with my app. I couldn't find the text that outputs this message either. The server is running very slow on this machine in debug mode, and it could timeout .. maybe. But I thought I should also mention this.
Would you have another entry point I could check ? I would have thought that startServerInt
or createUDPSocketAndListen
would do it !
Hi, ok interesting... different to what I thought..
hi Marten, Im coming back with not much more info sadly. Ive added printf's where they belong, at:
printf("Http3Server::CreateUDPSocketAndListen") ; printf("Http3Server::startServerInt") ; That is really weird, if you do not reach this point nothing can start. Please check, Wake() on the Eventloop and also if Schedule is called from the event loop. These trigger the execution of the other tasks, which are not called. And may be also the creating of the pipe, if anything fails there. printf("Http3Server::OnSocketEvent") ; printf("WakeUpHelper::OnSocketEvent\n");
Nothing prints. I've also added breakpoints there and it is not passing through. Could there be some syntax that changed (in WebTransport) in the JS and the client is disconnecting... ? I can only break in the creation of the Http3Server (ran 2 times likely because of the 2 channels in my app). No, this is identical to the main branch, it must be the event.
When in VS Code in debug mode, I get an ECONNRESET Error printed (sometimes a variant is 'socket hang'). But I dont get it when running the same code through the command line. This inconsistence is also a little weird to me - it could have to do with my app. I couldn't find the text that outputs this message either. The server is running very slow on this machine in debug mode, and it could timeout .. maybe. But I thought I should also mention this. I do not think it is connected.
Would you have another entry point I could check ? I would have thought that
startServerInt
orcreateUDPSocketAndListen
would do it !
You are right, these should do, so scheduling this task is somewhat broken. Anyway if it turns out to be too complicated, we might also wait for quiche to use different mechanism and I rewrite it again.
Marten
Ok, quiche has now a libevent based eventloop, I have switched to the new code in the branch: https://github.com/fails-components/webtransport/tree/eventloop can you test it on MacOS, please ?
Also a test on the windows branch (Windows support is almost there...) of a MacOS build would be nice: https://github.com/fails-components/webtransport/tree/windows I had to reorganize large parts of the CMakefile and include more libs.....
Hello, Will test both branch and let you know !
For libevent
, after installing openssl (with conda in an openssl
env) and running export OPENSSL_ROOT_DIR=/Users/benoit/opt/miniconda3/envs/openssl
, I could build the branch. But I got this error:
[ 65%] Building CXX object CMakeFiles/gquiche.dir/third_party/quiche/quiche/quic/bindings/quic_libevent.cc.o
In file included from /Users/benoit/dev/webtransport/third_party/quiche/quiche/quic/bindings/quic_libevent.cc:5:
/Users/benoit/dev/webtransport/third_party/quiche/quiche/quic/bindings/quic_libevent.h:11:10: fatal error: 'event2/event.h' file not found
#include "event2/event.h"
^~~~~~~~~~~~~~~~
1 error generated.
I guess Quiche is not finding libevent ... I'm not ultra familiar with Bazel. Do you know where/how I can add the libevent dependency for quiche?
EDIT Nevermind, I forgot quiche standard build was bypassed through CMake. I have added PUBLIC third_party/libevent/include
line 807 of CMakeLists.txt and this did the trick.
I then had to run the following in thirdparty/libevent
, in order to create the event2/include/event-config.h
file which was missing and causing another build error:
autoconf
./configure
make
I also had to add the following line in CMakeList around line 807:
PUBLIC third_party
This is not really correct/conventional, but this avoided this error:
[ 64%] Building CXX object CMakeFiles/gquiche.dir/third_party/quiche/quiche/quic/core/crypto/quic_crypto_server_config.cc.o
In file included from /Users/benoit/dev/webtransport/third_party/quiche/quiche/quic/core/crypto/quic_crypto_server_config.cc:5:
In file included from /Users/benoit/dev/webtransport/third_party/quiche/quiche/quic/core/crypto/quic_crypto_server_config.h:25:
/Users/benoit/dev/webtransport/third_party/quiche/quiche/quic/core/proto/cached_network_parameters_proto.h:8:10: fatal error: 'quiche/quic/core/proto/cached_network_parameters.pb.h' file not found
#include "quiche/quic/core/proto/cached_network_parameters.pb.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/gquiche.dir/third_party/quiche/quiche/quic/core/crypto/quic_crypto_server_config.cc.o] Error 1
make[1]: *** [CMakeFiles/gquiche.dir/all] Error 2
make: *** [all] Error 2
Quiche was able to build at this point. And the TEST SEEMS TO BE OKAY ! :D :D
Yes, it is not finding the headers....
My guess is that you did not fetch the submodules with git submodule update --recursive
(maybe also with --init) and that the submodule with libevent is not present. (For the windows build I had to turn some additional dependencies into submodules (not available with the needed runtime library), which increases build time on linux)...
Hello,
Thanks for making this repository!
I'm having trouble installing/running the repository. When following the install directions (v1), it seems git is not finding a valid git repository. When trying to install after downloading the repository (v2), the submodules are properly installed but the build is failing because build.ninja is missing ... ? I know I do not have perl6 installed yet but it doesn't seem to be related to this quite yet and I wanted to try with the default perl for now...
OS: Mac OSX 10.13.6 (High Sierra) Node: Node v16.14.0 NPM: v8.3.1 protoc: libprotoc 3.19.4 ninja: 1.10.2 Go: go1.17.8 darwin/amd64 cmake-js: 6.3.0 cmake: 3.23.0-rc2 clang: Apple LLVM version 10.0.0 (clang-1000.11.45.2) perl: v5.18.2
STEPS v1:
ERROR v1:
STEPS v2:
ERROR v2: