MatrixAI / Polykey-CLI

Polykey CLI - Open Source Decentralized Secret Sharing System for Zero Trust Workflows
https://polykey.com
GNU General Public License v3.0
6 stars 3 forks source link

Fix Failing Build Tests on Windows & MacOS #14

Open emmacasolin opened 2 years ago

emmacasolin commented 2 years ago

Specification

This is not about integration testing, this is about our unit tests

Not all of our tests are passing on Windows/Mac, both in the CI/CD and on matrix-win-1/matrix-mac-1. Some of these are due to obvious platform differences (e.g. the usage of / vs \), but others may be more difficult to debug.

Failing tests on windows:

Failing tests on Mac (all timeouts, even when only a single test is run):

Additional context

Tasks

  1. Group tests by similar failures
  2. ...
  3. ...
emmacasolin commented 2 years ago

Full output from failing tests:

``` Summary of all failing tests FAIL tests/bin/vaults/vaults.test.ts (90.027 s) ● CLI vaults › should clone and pull a vault expect(received).toBe(expected) // Object.is equality Expected: 0 Received: 255 268 | 269 | let result = await testBinUtils.pkStdio([...command], {}, dataDir); > 270 | expect(result.exitCode).toBe(0); | ^ 271 | 272 | const clonedVaultId = await polykeyAgent.vaultManager.getVaultId( 273 | vaultName, at Object. (tests/bin/vaults/vaults.test.ts:270:31) FAIL tests/bin/agent/start.test.ts (170.883 s) ● start › start in foreground spawn ts-node ENOENT ● start › start in foreground thrown: undefined 28 | }); 29 | }); > 30 | test( | ^ 31 | 'start in foreground', 32 | async () => { 33 | const password = 'abc123'; at tests/bin/agent/start.test.ts:30:3 at Object. (tests/bin/agent/start.test.ts:16:1) ● start › start in background spawn ts-node ENOENT ● start › start in background thrown: undefined 99 | global.defaultTimeout * 2, 100 | ); > 101 | test( | ^ 102 | 'start in background', 103 | async () => { 104 | const password = 'abc123'; at tests/bin/agent/start.test.ts:101:3 at Object. (tests/bin/agent/start.test.ts:16:1) ● start › concurrent starts results in 1 success spawn ts-node ENOENT ● start › concurrent starts results in 1 success spawn ts-node ENOENT ● start › concurrent starts results in 1 success thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 200 | global.defaultTimeout * 2, 201 | ); > 202 | test( | ^ 203 | 'concurrent starts results in 1 success', 204 | async () => { 205 | const password = 'abc123'; at tests/bin/agent/start.test.ts:202:3 at Object. (tests/bin/agent/start.test.ts:16:1) ● start › concurrent with bootstrap results in 1 success spawn ts-node ENOENT ● start › concurrent with bootstrap results in 1 success spawn ts-node ENOENT ● start › concurrent with bootstrap results in 1 success thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 298 | global.defaultTimeout * 2, 299 | ); > 300 | test( | ^ 301 | 'concurrent with bootstrap results in 1 success', 302 | async () => { 303 | const password = 'abc123'; at tests/bin/agent/start.test.ts:300:3 at Object. (tests/bin/agent/start.test.ts:16:1) ● start › start with existing state spawn ts-node ENOENT ● start › start with existing state thrown: undefined 390 | global.defaultTimeout * 2, 391 | ); > 392 | test( | ^ 393 | 'start with existing state', 394 | async () => { 395 | const password = 'abc123'; at tests/bin/agent/start.test.ts:392:3 at Object. (tests/bin/agent/start.test.ts:16:1) ● start › start when interrupted, requires fresh on next start spawn ts-node ENOENT ● start › start when interrupted, requires fresh on next start thrown: undefined 470 | global.defaultTimeout * 2, 471 | ); > 472 | test( | ^ 473 | 'start when interrupted, requires fresh on next start', 474 | async () => { 475 | const password = 'password'; at tests/bin/agent/start.test.ts:472:3 at Object. (tests/bin/agent/start.test.ts:16:1) ● start › start from recovery code spawn ts-node ENOENT ● start › start from recovery code thrown: undefined 580 | global.defaultTimeout * 2, 581 | ); > 582 | test( | ^ 583 | 'start from recovery code', 584 | async () => { 585 | const password1 = 'abc123'; at tests/bin/agent/start.test.ts:582:3 at Object. (tests/bin/agent/start.test.ts:16:1) ● start › start with network configuration spawn ts-node ENOENT ● start › start with network configuration thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 714 | global.defaultTimeout * 3, 715 | ); > 716 | test( | ^ 717 | 'start with network configuration', 718 | async () => { 719 | const status = new Status({ at tests/bin/agent/start.test.ts:716:3 at Object. (tests/bin/agent/start.test.ts:16:1) FAIL tests/bin/secrets/secrets.test.ts (41.16 s) ● CLI secrets › commandNewDir › should make a directory ErrorEncryptedFSError: ENOENT: no such file or directory, dir1\MySecret1 67 | for (const dirent of dirents) { 68 | const res = path.join(dir, dirent.toString()); > 69 | const stat = await fs.promises.stat(res); | ^ 70 | if (stat.isDirectory()) { 71 | yield* readdirRecursively(fs, res); 72 | } else if (stat.isFile()) { at node_modules/encryptedfs/src/EncryptedFS.ts:2932:15 at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12) at readdirRecursively (src/vaults/utils.ts:69:18) at Object.readdirRecursively (src/vaults/utils.ts:71:7) at src/vaults/VaultOps.ts:256:22 at src/vaults/VaultInternal.ts:427:14 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at Object.listSecrets (src/vaults/VaultOps.ts:254:10) at tests/bin/secrets/secrets.test.ts:169:22 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) ● CLI secrets › commandNewDirSecret › should add a directory of secrets expect(received).toStrictEqual(expected) // deep equality - Expected - 3 + Received + 3 Array [ - "secrets/secret-1", - "secrets/secret-2", - "secrets/secret-3", + "secrets\\secret-1", + "secrets\\secret-2", + "secrets\\secret-3", ] 269 | await polykeyAgent.vaultManager.withVaults([vaultId], async (vault) => { 270 | const list = await vaultOps.listSecrets(vault); > 271 | expect(list.sort()).toStrictEqual([ | ^ 272 | 'secrets/secret-1', 273 | 'secrets/secret-2', 274 | 'secrets/secret-3', at tests/bin/secrets/secrets.test.ts:271:29 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at constructor_.withVaults (src/vaults/VaultManager.ts:971:12) at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at Object. (tests/bin/secrets/secrets.test.ts:269:7) FAIL tests/bin/bootstrap.test.ts (54.965 s) ● bootstrap › concurrent bootstrapping results in 1 success spawn ts-node ENOENT ● bootstrap › concurrent bootstrapping results in 1 success spawn ts-node ENOENT ● bootstrap › concurrent bootstrapping results in 1 success thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 105 | global.defaultTimeout * 2, 106 | ); > 107 | test( | ^ 108 | 'concurrent bootstrapping results in 1 success', 109 | async () => { 110 | const password = 'password'; at tests/bin/bootstrap.test.ts:107:3 at Object. (tests/bin/bootstrap.test.ts:10:1) at runMicrotasks () ● bootstrap › bootstrap when interrupted, requires fresh on next bootstrap spawn ts-node ENOENT ● bootstrap › bootstrap when interrupted, requires fresh on next bootstrap thrown: undefined 186 | global.defaultTimeout * 2, 187 | ); > 188 | test( | ^ 189 | 'bootstrap when interrupted, requires fresh on next bootstrap', 190 | async () => { 191 | const password = 'password'; at tests/bin/bootstrap.test.ts:188:3 at Object. (tests/bin/bootstrap.test.ts:10:1) at runMicrotasks () FAIL tests/bin/agent/stop.test.ts (70.078 s) ● stop › stopping is idempotent during concurrent calls and STOPPING or DEAD status expect(received).toBe(expected) // Object.is equality Expected: 0 Received: "ENOENT" 152 | expect(agentStop1.exitCode).toBe(0); 153 | } else { > 154 | expect(agentStop1.exitCode).toBe(0); | ^ 155 | expect(agentStop2.exitCode).toBe(0); 156 | } 157 | expect(agentStop3.exitCode).toBe(0); at Object. (tests/bin/agent/stop.test.ts:154:37) ● stop › stopping starting agent results in error spawn ts-node ENOENT ● stop › stopping starting agent results in error thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 160 | global.defaultTimeout * 2, 161 | ); > 162 | test( | ^ 163 | 'stopping starting agent results in error', 164 | async () => { 165 | const password = 'abc123'; at tests/bin/agent/stop.test.ts:162:3 at Object. (tests/bin/agent/stop.test.ts:12:1) FAIL tests/bin/utils.test.ts ● bin/utils › errors in human and json format expect(received).toBe(expected) // Object.is equality - Expected - 1 + Received + 1 - {"type":"TypeError","data":{"message":"some error","stack":"TypeError: some error\n at Object. (C:\Users\Emma Casolin\Projects\js-polykey\tests\bin\utils.test.ts:84:27)\n at Promise.then.completed (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\utils.js:333:28)\n at new Promise ()\n at callAsyncCircusFn (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\utils.js:259:10)\n at _callCircusTest (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\run.js:277:40)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at _runTest (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\run.js:209:3)\n at _runTestsForDescribeBlock (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\run.js:97:9)\n at _runTestsForDescribeBlock (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\run.js:91:9)\n at run (C:\Users\Emma Casolin\Projects\js-polykey\node_modules\jest-circus\build\run.js:31:3)"}} + {"type":"TypeError","data":{"message":"some error","stack":"TypeError: some error\n at Object. (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\tests\\bin\\utils.test.ts:84:27)\n at Promise.then.completed (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\utils.js:333:28)\n at new Promise ()\n at callAsyncCircusFn (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\utils.js:259:10)\n at _callCircusTest (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\run.js:277:40)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at _runTest (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\run.js:209:3)\n at _runTestsForDescribeBlock (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\run.js:97:9)\n at _runTestsForDescribeBlock (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\run.js:91:9)\n at run (C:\\Users\\Emma Casolin\\Projects\\js-polykey\\node_modules\\jest-circus\\build\\run.js:31:3)"}} ↵ 170 | expect( 171 | binUtils.outputFormatter({ type: 'json', data: standardError }), > 172 | ).toBe( | ^ 173 | `{"type":"${standardError.name}","data":{"message":"${ 174 | standardError.message 175 | }","stack":"${standardError.stack?.replaceAll('\n', '\\n')}"}}\n`, at Object. (tests/bin/utils.test.ts:172:7) FAIL tests/bin/agent/status.test.ts (44.001 s) ● status › status on STARTING, STOPPING, DEAD agent spawn ts-node ENOENT ● status › status on STARTING, STOPPING, DEAD agent thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 25 | }); 26 | }); > 27 | test( | ^ 28 | 'status on STARTING, STOPPING, DEAD agent', 29 | async () => { 30 | // This test must create its own agent process at tests/bin/agent/status.test.ts:27:3 at Object. (tests/bin/agent/status.test.ts:11:1) at runMicrotasks () Test Suites: 7 failed, 25 passed, 32 total Tests: 17 failed, 1 todo, 95 passed, 113 total Snapshots: 0 total Time: 362.339 s Ran all test suites matching /tests\\bin/i. GLOBAL TEARDOWN Destroying Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-niloUA > jest "tests/keys/keymanager" "--maxWorkers=50%" Determining test suites to run... GLOBAL SETUP Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-BuA71v FAIL tests/keys/KeyManager.test.ts (63.31 s) KeyManager √ KeyManager readiness (920 ms) √ constructs root key pair, root cert, root certs and db key (898 ms) √ creates a recovery code and can recover from the same code (26689 ms) √ create deterministic keypair with recovery code (9273 ms) √ uses WorkerManager for generating root key pair (478 ms) √ encrypting and decrypting with root key (491 ms) √ uses WorkerManager for encryption and decryption with root key (575 ms) √ encrypting beyond maximum size (467 ms) √ signing and verifying with root key (509 ms) √ uses WorkerManager for signing and verifying with root key (523 ms) √ can change root key password (1860 ms) √ can reset root certificate (2491 ms) × can reset root key pair (3008 ms) × can renew root key pair (3030 ms) √ order of certificate chain should be leaf to root (5422 ms) dbKey √ Creates a key when started. (509 ms) √ Throws an exception when it fails to parse the key. (913 ms) √ key remains unchanged when resetting keys. (1346 ms) √ key remains unchanged when renewing keys. (1350 ms) ● KeyManager › can reset root key pair EBUSY: resource busy or locked, unlink 'C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-8sVU03\db\000005.ldb' ● KeyManager › can renew root key pair EBUSY: resource busy or locked, unlink 'C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-dJ4AxX\db\000005.ldb' Test Suites: 1 failed, 1 total Tests: 2 failed, 17 passed, 19 total Snapshots: 0 total Time: 63.548 s Ran all test suites matching /tests\\keys\\keymanager/i. PS C:\Users\Emma Casolin\Projects\js-polykey> npm run test tests/network/proxy -- --maxWorkers=50% > @matrixai/polykey@1.0.0 test > jest "tests/network/proxy" "--maxWorkers=50%" Determining test suites to run... GLOBAL SETUP Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-WkqxaW ERROR:Proxy CONNECT bad request:Failed CONNECT to 127.0.0.1:80 - ErrorProxyConnectAuth ERROR:Proxy CONNECT bad request:Failed CONNECT to 0.0.0.0:80 - ErrorProxyConnectInvalidUrl ERROR:Proxy CONNECT bad request:Failed CONNECT - ErrorProxyConnectMissingNodeId ERROR:Proxy CONNECT bad request:Failed CONNECT - ErrorProxyConnectInvalidUrl ERROR:Proxy connection timeout:Failed CONNECT to 127.0.0.1:61837 - ErrorConnectionStartTimeout ERROR:Proxy connection reset:Failed CONNECT to 127.0.0.1:61839 - ErrorConnectionStart: UTP_ECONNRESET ERROR:Proxy missing certificates:Failed CONNECT to 127.0.0.1:61846 - ErrorConnectionStart: 5604:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:c:\ws\deps\openssl\openssl\ssl\record\rec_layer_s3.c:1546:SSL alert number 40 ERROR:Proxy invalid node id:Failed CONNECT to 127.0.0.1:61850 - ErrorCertChainUnclaimed: Node ID is not claimed by any certificate WARN:ConnectionForward 127.0.0.1:61864:Forward Error: ErrorConnectionTimeout WARN:ConnectionForward 127.0.0.1:61866:Forward Error: ErrorConnectionTimeout ERROR:Proxy port 0:Failed CONNECT to 127.0.0.1:0 - ErrorConnectionStartTimeout WARN:ConnectionReverse 127.0.0.1:61886:Reverse Error: ErrorConnectionTimeout WARN:Proxy test:Failed connection from 127.0.0.1:61886 - ErrorConnectionComposeTimeout WARN:Proxy test:Failed connection from 127.0.0.1:61888 - ErrorCertChainEmpty: No certificates available to verify WARN:ConnectionReverse 127.0.0.1:61888:Reverse Error: ErrorConnectionTimeout WARN:Proxy test:Failed connection from 127.0.0.1:61892 - ErrorConnectionNotRunning WARN:ConnectionReverse 127.0.0.1:61892:Reverse Error: ErrorConnectionEndTimeout FAIL tests/network/Proxy.test.ts (75.495 s) Proxy √ proxy readiness (76 ms) √ HTTP CONNECT bad request failures to the forward proxy (46 ms) × connection to port 0 fails (20005 ms) √ connection start timeout due to hanging remote (4187 ms) √ connection reset due to ending remote (4604 ms) √ open connection fails due to missing certificates (31 ms) √ HTTP CONNECT fails due to missing certificates (26 ms) √ open connection fails due to invalid node id (209 ms) √ HTTP CONNECT fails due to invalid node id (66 ms) √ open connection success - forward initiates end (70 ms) √ open connection success - reverse initiates end (84 ms) √ HTTP CONNECT success - forward initiates end (80 ms) √ HTTP CONNECT success - reverse initiates end (174 ms) √ HTTP CONNECT success - client initiates end (233 ms) √ HTTP CONNECT success by opening connection first (84 ms) √ open connection keepalive timeout (1086 ms) √ HTTP CONNECT keepalive timeout (1178 ms) √ stopping the proxy with open forward connections (89 ms) √ open connection to multiple servers (90 ms) × open connection to port 0 fails (20007 ms) √ open connection timeout due to lack of ready signal (3016 ms) √ open connection success (8 ms) √ open connection to multiple clients (15 ms) × closed connection due to ending server (7 ms) √ connect timeout due to hanging client (3162 ms) √ connect fails due to missing client certificates (2126 ms) √ connect success (113 ms) √ stopping the proxy with open reverse connections (1083 ms) √ connectionEstablishedCallback is called when a ReverseConnection is established (59 ms) ● Proxy › connection to port 0 fails thrown: "Exceeded timeout of 20000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 242 | await proxy.stop(); 243 | }); > 244 | test('connection to port 0 fails', async () => { | ^ 245 | const proxy = new Proxy({ 246 | authToken, 247 | logger: logger.getChild('Proxy port 0'), at tests/network/Proxy.test.ts:244:3 at Object. (tests/network/Proxy.test.ts:108:1) ● Proxy › open connection to port 0 fails thrown: "Exceeded timeout of 20000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 2316 | await proxy.stop(); 2317 | }); > 2318 | test('open connection to port 0 fails', async () => { | ^ 2319 | const proxy = new Proxy({ 2320 | logger: logger.getChild('Proxy port 0'), 2321 | authToken: '', at tests/network/Proxy.test.ts:2318:3 at Object. (tests/network/Proxy.test.ts:108:1) ● Proxy › closed connection due to ending server expect(received).toBe(expected) // Object.is equality Expected: 1 Received: 0 2563 | const utpSocketPort = utpSocket.address().port; 2564 | await proxy.openConnectionReverse(localHost, utpSocketPort as Port); > 2565 | expect(proxy.getConnectionReverseCount()).toBe(1); | ^ 2566 | await expect(serverConnP).resolves.toBeUndefined(); 2567 | // The server receives the end confirmation for graceful exit 2568 | await expect(serverConnEndP).resolves.toBeUndefined(); at Object. (tests/network/Proxy.test.ts:2565:47) Test Suites: 1 failed, 1 total Tests: 3 failed, 26 passed, 29 total Snapshots: 0 total Time: 75.722 s, estimated 87 s Ran all test suites matching /tests\\network\\proxy/i. GLOBAL TEARDOWN Destroying Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-WkqxaW Jest did not exit one second after the test run has completed. This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with `--detectOpenHandles` to troubleshoot this issue. PS C:\Users\Emma Casolin\Projects\js-polykey> npm run test tests/nodes/nodeconnection.test -- --maxWorkers=50% > @matrixai/polykey@1.0.0 test > jest "tests/nodes/nodeconnection.test" "--maxWorkers=50%" Determining test suites to run... GLOBAL SETUP Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-kK3jf3 WARN:ConnectionReverse 127.0.0.1:64741:Server Error: ErrorConnectionEndTimeout WARN:ConnectionReverse 127.0.0.1:64741:Reverse Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:64742:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:64744:Client Error: ErrorConnectionEndTimeout WARN:ConnectionReverse 127.0.0.1:64745:Server Error: ErrorConnectionEndTimeout ERROR:NodeConnection test:Failed CONNECT to 128.0.0.1:12345 - ErrorConnectionStart: Send failed (status: -4062) ERROR:grpc:Failed to connect to 128.0.0.1:12345/?nodeId=vmudpqa10ulc0538eud5ae4ia913ct15sv0bhth6v85hdlgtuhea0 through proxy 127.0.0.1:52189 ERROR:grpc:Failed to connect to proxy 127.0.0.1:52189 with error connect ECONNREFUSED 127.0.0.1:52189 ERROR:grpc:Failed to connect to proxy 127.0.0.1:52189 with error connect ECONNREFUSED 127.0.0.1:52189 WARN:ConnectionForward 127.0.0.1:59508:Client Error: ErrorConnectionEndTimeout WARN:ConnectionReverse 127.0.0.1:59173:Server Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:59174:Client Error: ErrorConnectionEndTimeout WARN:ConnectionReverse 127.0.0.1:59175:Server Error: ErrorConnectionEndTimeout ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:59178 - ErrorCertChainUnclaimed: Node ID is not claimed by any certificate ERROR:grpc:Failed to connect to 127.0.0.1:59178/?nodeId=vd1g01vfel7usqkn7bijvvd8cfte7q4h0e2jfnj8rgmn66ih8f8ng through proxy 127.0.0.1:52207 ERROR:grpc:Failed to connect to proxy 127.0.0.1:52207 with error connect ECONNREFUSED 127.0.0.1:52207 ERROR:grpc:Failed to connect to proxy 127.0.0.1:52207 with error connect ECONNREFUSED 127.0.0.1:52207 WARN:NodeConnection test:Failed connection from 127.0.0.1:59180 - ErrorConnectionNotRunning ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:59181 - ErrorConnectionStart: Client network socket disconnected before secure TLS co ERROR:grpc:Failed to connect to 127.0.0.1:59181/?nodeId=vdd6vphgd4n9lmff3h0t2p5r5s27r594r1qidp26a1e56rm798b7g through proxy 127.0.0.1:52216 ERROR:grpc:Failed to connect to proxy 127.0.0.1:52216 with error connect ECONNREFUSED 127.0.0.1:52216 ERROR:grpc:Failed to connect to proxy 127.0.0.1:52216 with error connect ECONNREFUSED 127.0.0.1:52216 WARN:ConnectionForward 127.0.0.1:58219:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:51189:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:51191:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:51193:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:51195:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:51200:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:51202:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:51204:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:51206:Client Error: ErrorConnectionEndTimeout ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:51211 - ErrorCertChainUnclaimed: Node ID is not claimed by any certificate ERROR:grpc:Failed to connect to 127.0.0.1:51211/?nodeId=viala3ulemjdt7pfchefu9khp4pub301c1dli3i5unitjp6k4h2c0 through proxy 127.0.0.1:52313 WARN:ConnectionReverse 127.0.0.1:51212:Server Error: ErrorConnectionEndTimeout WARN:NodeConnection test:Failed connection from 127.0.0.1:51212 - ErrorConnectionNotRunning ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:51211 - ErrorConnectionStart: Client network socket disconnected before secure TLS co ERROR:grpc:Failed to connect to 127.0.0.1:51211/?nodeId=viala3ulemjdt7pfchefu9khp4pub301c1dli3i5unitjp6k4h2c0 through proxy 127.0.0.1:52313 ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:51211 - ErrorCertChainUnclaimed: Node ID is not claimed by any certificate ERROR:grpc:Failed to connect to 127.0.0.1:51211/?nodeId=viala3ulemjdt7pfchefu9khp4pub301c1dli3i5unitjp6k4h2c0 through proxy 127.0.0.1:52313 WARN:ConnectionReverse 127.0.0.1:51212:Server Error: ErrorConnectionEndTimeout WARN:NodeConnection test:Failed connection from 127.0.0.1:51212 - ErrorConnectionNotRunning ERROR:NodeConnection test:Failed CONNECT to 127.0.0.1:51211 - ErrorConnectionStart: Client network socket disconnected before secure TLS co ERROR:grpc:Failed to connect to 127.0.0.1:51211/?nodeId=v6ksrp0othl670be87gcngk7gqcvcnnd8ckvkbma9o26un1lbq0o0 through proxy 127.0.0.1:52313 ERROR:grpc:Failed to connect to proxy 127.0.0.1:52313 with error connect ECONNREFUSED 127.0.0.1:52313 WARN:ConnectionForward 127.0.0.1:51211:Client Error: ErrorConnectionEndTimeout ERROR:grpc:Failed to connect to proxy 127.0.0.1:52313 with error connect ECONNREFUSED 127.0.0.1:52313 ERROR:grpc:Failed to connect to proxy 127.0.0.1:52313 with error connect ECONNREFUSED 127.0.0.1:52313 WARN:ConnectionForward 127.0.0.1:51213:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:61089:Client Error: ErrorConnectionEndTimeout FAIL tests/nodes/NodeConnection.test.ts (380.382 s) NodeConnection test √ session readiness (4927 ms) √ connects to its target (via direct connection) (4649 ms) √ connects to its target but proxies connect first (3998 ms) √ grpcCall after connection drops (4101 ms) √ fails to connect to target (times out) (3278 ms) √ getRootCertChain (4929 ms) √ getExpectedPublicKey (3748 ms) √ should call `killSelf if connection is closed based on bad certificate (3216 ms) √ should call `killSelf if connection is closed before TLS is established (4122 ms) √ should call `killSelf if the Agent is stopped. (5684 ms) × should call `killSelf and throw if the server exit's during testUnaryFail (43061 ms) × should call `killSelf and throw if the server kill's during testUnaryFail (43529 ms) × should call `killSelf and throw if the server sigkill's during testUnaryFail (42579 ms) × should call `killSelf and throw if the server exit's during testStreamFail (43072 ms) × should call `killSelf and throw if the server kill's during testStreamFail (42783 ms) × should call `killSelf and throw if the server sigkill's during testStreamFail (43681 ms) √ existing connection handles a resetRootKeyPair on sending side (5289 ms) √ existing connection handles a renewRootKeyPair on sending side (6081 ms) √ existing connection handles a resetRootCert on sending side (4667 ms) √ existing connection handles a resetRootKeyPair on receiving side (5398 ms) √ existing connection handles a renewRootKeyPair on receiving side (5735 ms) √ existing connection handles a resetRootCert on receiving side (4061 ms) √ new connection handles a resetRootKeyPair on sending side (5549 ms) √ new connection handles a renewRootKeyPair on sending side (5000 ms) √ new connection handles a resetRootCert on sending side (4152 ms) √ new connection handles a resetRootKeyPair on receiving side (9190 ms) √ new connection handles a renewRootKeyPair on receiving side (5578 ms) √ new connection handles a resetRootCert on receiving side (6292 ms) ● NodeConnection test › should call `killSelf and throw if the server exit's during testUnaryFail spawn ts-node ENOENT ● NodeConnection test › should call `killSelf and throw if the server exit's during testUnaryFail thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 717 | }); 718 | const options = ['exit', 'kill', 'sigkill']; > 719 | test.each(options)( | ^ 720 | "should call `killSelf and throw if the server %s's during testUnaryFail", 721 | async (option) => { 722 | let nodeConnection: at node_modules/jest-each/build/bind.js:45:11 at Array.forEach () at tests/nodes/NodeConnection.test.ts:719:21 at Object. (tests/nodes/NodeConnection.test.ts:66:1) ● NodeConnection test › should call `killSelf and throw if the server kill's during testUnaryFail spawn ts-node ENOENT ● NodeConnection test › should call `killSelf and throw if the server kill's during testUnaryFail thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 717 | }); 718 | const options = ['exit', 'kill', 'sigkill']; > 719 | test.each(options)( | ^ 720 | "should call `killSelf and throw if the server %s's during testUnaryFail", 721 | async (option) => { 722 | let nodeConnection: at node_modules/jest-each/build/bind.js:45:11 at Array.forEach () at tests/nodes/NodeConnection.test.ts:719:21 at Object. (tests/nodes/NodeConnection.test.ts:66:1) ● NodeConnection test › should call `killSelf and throw if the server sigkill's during testUnaryFail spawn ts-node ENOENT ● NodeConnection test › should call `killSelf and throw if the server sigkill's during testUnaryFail thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 717 | }); 718 | const options = ['exit', 'kill', 'sigkill']; > 719 | test.each(options)( | ^ 720 | "should call `killSelf and throw if the server %s's during testUnaryFail", 721 | async (option) => { 722 | let nodeConnection: at node_modules/jest-each/build/bind.js:45:11 at Array.forEach () at tests/nodes/NodeConnection.test.ts:719:21 at Object. (tests/nodes/NodeConnection.test.ts:66:1) ● NodeConnection test › should call `killSelf and throw if the server exit's during testStreamFail spawn ts-node ENOENT ● NodeConnection test › should call `killSelf and throw if the server exit's during testStreamFail thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 787 | global.defaultTimeout * 2, 788 | ); > 789 | test.each(options)( | ^ 790 | "should call `killSelf and throw if the server %s's during testStreamFail", 791 | async (option) => { 792 | let nodeConnection: at node_modules/jest-each/build/bind.js:45:11 at Array.forEach () at tests/nodes/NodeConnection.test.ts:789:21 at Object. (tests/nodes/NodeConnection.test.ts:66:1) ● NodeConnection test › should call `killSelf and throw if the server kill's during testStreamFail spawn ts-node ENOENT ● NodeConnection test › should call `killSelf and throw if the server kill's during testStreamFail thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 787 | global.defaultTimeout * 2, 788 | ); > 789 | test.each(options)( | ^ 790 | "should call `killSelf and throw if the server %s's during testStreamFail", 791 | async (option) => { 792 | let nodeConnection: at node_modules/jest-each/build/bind.js:45:11 at Array.forEach () at tests/nodes/NodeConnection.test.ts:789:21 at Object. (tests/nodes/NodeConnection.test.ts:66:1) ● NodeConnection test › should call `killSelf and throw if the server sigkill's during testStreamFail spawn ts-node ENOENT ● NodeConnection test › should call `killSelf and throw if the server sigkill's during testStreamFail thrown: "Exceeded timeout of 40000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test." 787 | global.defaultTimeout * 2, 788 | ); > 789 | test.each(options)( | ^ 790 | "should call `killSelf and throw if the server %s's during testStreamFail", 791 | async (option) => { 792 | let nodeConnection: at node_modules/jest-each/build/bind.js:45:11 .ts:66:1) Test Suites: 1 failed, 1 total Tests: 6 failed, 22 passed, 28 total Snapshots: 0 total Time: 380.547 s, estimated 407 s Ran all test suites matching /tests\\nodes\\nodeconnection.test/i. GLOBAL TEARDOWN Destroying Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-kK3jf3 PS C:\Users\Emma Casolin\Projects\js-polykey> npm run test tests/vaults -- --maxWorkers=50% > @matrixai/polykey@1.0.0 test > jest "tests/vaults" "--maxWorkers=50%" Determining test suites to run... GLOBAL SETUP Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-A3ophi ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout ERROR:GRPCClientAgentService:vaultsGitInfoGet:ErrorGitUndefinedRefs: Ref HEAD cannot be found WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:64239:Client Error: ErrorConnectionEndTimeout WARN:ConnectionForward 127.0.0.1:57148:Client Error: ErrorConnectionEndTimeout FAIL tests/vaults/VaultInternal.test.ts (190.854 s) VaultInternal √ VaultInternal readiness (3186 ms) √ is type correct (2259 ms) √ creating state on disk (2050 ms) √ accessing a change (4033 ms) √ maintains data across VaultInternal instances (5703 ms) √ can change to the current commit (3871 ms) √ can change commits and preserve the log with no intermediate vault mutation (9033 ms) × does not allow changing to an unrecognised commit (11534 ms) √ can change to the latest commit (10696 ms) √ adjusts HEAD after vault mutation, discarding forward and preserving backwards history (10862 ms) √ write operation allowed (2761 ms) √ read operation allowed (3382 ms) √ concurrent write operations prevented (8170 ms) √ commit added if mutation in writeF (3089 ms) √ no commit added if no mutation in write (2326 ms) √ commit message contains all actions made in the commit (7572 ms) √ no mutation to vault when part of a commit operation fails (4227 ms) √ concurrent read operations allowed (3402 ms) √ no commit after read (3459 ms) √ only exposes limited commands of VaultInternal (2862 ms) √ cannot commit when the remote field is set (1882 ms) × cannot checkout old commits after branching commit (11442 ms) × can recover from dirty state (7216 ms) × clean errant commits recovering from dirty state (8184 ms) √ commit added if mutation in writeG (3719 ms) √ no commit added if no mutation in writeG (2172 ms) √ no mutation to vault when part of a commit operation fails in writeG (2560 ms) √ no commit after readG (3666 ms) × garbage collection (10793 ms) √ writeF respects read and write locking (2381 ms) √ writeG respects read and write locking (2610 ms) √ readF respects write locking (1573 ms) √ readG respects write locking (1656 ms) √ readF allows concurrent reads (1556 ms) √ readG allows concurrent reads (1673 ms) √ can create with CreateVaultInternal (2919 ms) × can create an existing vault with CreateVaultInternal (3390 ms) √ stop is idempotent (1545 ms) √ destroy is idempotent (1492 ms) ● VaultInternal › does not allow changing to an unrecognised commit expect(received).rejects.toThrow() Received promise resolved instead of rejected Resolved to value: undefined 220 | await efs.close(fd); 221 | }); > 222 | await expect(vault.version(fourthCommit)).rejects.toThrow( | ^ 223 | vaultsErrors.ErrorVaultReferenceMissing, 224 | ); 225 | }); at expect (node_modules/expect/build/index.js:128:15) at Object. (tests/vaults/VaultInternal.test.ts:222:11) ● VaultInternal › cannot checkout old commits after branching commit expect(received).rejects.toThrow() Received promise resolved instead of rejected Resolved to value: undefined 529 | await efs.writeFile('test4', 'testdata4'); 530 | }); > 531 | await expect(() => { | ^ 532 | return vault.version(thirdCommit); 533 | }).rejects.toThrow(); 534 | await expect(() => { at expect (node_modules/expect/build/index.js:128:15) at Object. (tests/vaults/VaultInternal.test.ts:531:13) ● VaultInternal › can recover from dirty state ErrorEncryptedFSError: ENOENT: no such file or directory, zBmuurZGqWiEge3Fnt47NGg\.git\objects 1080 | // Walking all objects 1081 | const objectPath = path.join(this.vaultGitDir, 'objects'); > 1082 | const buckets = (await this.efs.readdir(objectPath)).filter((item) => { | ^ 1083 | return item !== 'info' && item !== 'pack'; 1084 | }); 1085 | for (const bucket of buckets) { at node_modules/encryptedfs/src/EncryptedFS.ts:2321:15 at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12) at constructor_.garbageCollectGitObjects (src/vaults/VaultInternal.ts:1082:22) at constructor_.setupGit (src/vaults/VaultInternal.ts:744:9) at constructor_.start_ (src/vaults/VaultInternal.ts:303:5) at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at constructor_.start (src/vaults/VaultInternal.ts:269:14) at node_modules/@matrixai/async-init/src/CreateDestroyStartStop.ts:111:24 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at Object. (tests/vaults/VaultInternal.test.ts:561:5) ● VaultInternal › clean errant commits recovering from dirty state ErrorEncryptedFSError: ENOENT: no such file or directory, zNiUfngckMkoinYUm43jku7\.git\objects 1080 | // Walking all objects 1081 | const objectPath = path.join(this.vaultGitDir, 'objects'); > 1082 | const buckets = (await this.efs.readdir(objectPath)).filter((item) => { | ^ 1083 | return item !== 'info' && item !== 'pack'; 1084 | }); 1085 | for (const bucket of buckets) { at node_modules/encryptedfs/src/EncryptedFS.ts:2321:15 at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12) at constructor_.garbageCollectGitObjects (src/vaults/VaultInternal.ts:1082:22) at constructor_.setupGit (src/vaults/VaultInternal.ts:744:9) at constructor_.start_ (src/vaults/VaultInternal.ts:303:5) at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at constructor_.start (src/vaults/VaultInternal.ts:269:14) at node_modules/@matrixai/async-init/src/CreateDestroyStartStop.ts:111:24 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at Object. (tests/vaults/VaultInternal.test.ts:621:5) ● VaultInternal › garbage collection ErrorEncryptedFSError: ENOENT: no such file or directory, z5jkvSDcS3sJsjn9CaWeGcK\.git\objects 1080 | // Walking all objects 1081 | const objectPath = path.join(this.vaultGitDir, 'objects'); > 1082 | const buckets = (await this.efs.readdir(objectPath)).filter((item) => { | ^ 1083 | return item !== 'info' && item !== 'pack'; 1084 | }); 1085 | for (const bucket of buckets) { at node_modules/encryptedfs/src/EncryptedFS.ts:2321:15 at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12) at constructor_.garbageCollectGitObjects (src/vaults/VaultInternal.ts:1082:22) at Object. (tests/vaults/VaultInternal.test.ts:724:7) ● VaultInternal › can create an existing vault with CreateVaultInternal expect(received).toHaveLength(expected) Expected length: 2 Received length: 6 Received array: ["zPB64HxhKtEGWXBi7pjFbuz", "zPB64HxhKtEGWXBi7pjFbuz\\.git", "zPB64HxhKtEGWXBi7pjFbuz\\data", "zXjcSjDBt7ifWjZ9RdrKp1r", "zXjcSjDBt7ifWjZ9RdrKp1r\\.git", "zXjcSjDBt7ifWjZ9RdrKp1r\\data"] 928 | vaultsUtils.encodeVaultId(vaultId1), 929 | ); > 930 | expect(await efs.readdir('.')).toHaveLength(2); | ^ 931 | } finally { 932 | await vault1?.stop(); 933 | await vault1?.destroy(); at Object. (tests/vaults/VaultInternal.test.ts:930:38) FAIL tests/vaults/VaultManager.test.ts (197.042 s) VaultManager √ VaultManager readiness (287 ms) √ is type correct (79 ms) √ can create many vaults and open a vault (23274 ms) √ can rename a vault (2733 ms) √ can delete a vault (1432 ms) √ can list vaults (3214 ms) √ able to read and load existing metadata (14462 ms) √ cannot concurrently create vaults with the same name (1337 ms) √ can concurrently rename the same vault (1571 ms) √ can concurrently open and rename the same vault (1350 ms) √ can save the commit state of a vault (2878 ms) √ Do actions on a vault using `withVault` (5366 ms) √ handleScanVaults should list all vaults with permissions (4413 ms) √ ScanVaults should get all vaults with permissions from remote node (8795 ms) √ stopping respects locks (2402 ms) √ destroying respects locks (2436 ms) √ withVault respects locks (2532 ms) √ Creation adds a vault (1365 ms) √ Concurrently creating vault with same name only creates 1 vault (1397 ms) √ vaults persist (1748 ms) √ vaults can be removed from map (1628 ms) √ stopping vaultManager empties map and stops all vaults (3098 ms) √ destroying vaultManager destroys all data (1948 ms) √ withVaults should throw if vaultId doesn't exist (58 ms) √ generateVaultId handles vault conflicts (2085 ms) With remote agents × clone vaults from a remote keynode using a vault name (6495 ms) × clone vaults from a remote keynode using a vault name with no history (4797 ms) √ fails to clone non existing vault (4147 ms) × clone and pull vaults using a vault id (6223 ms) √ should reject cloning when permissions are not set (4435 ms) × should reject Pulling when permissions are not set (5079 ms) × can pull a cloned vault (6172 ms) × manage pulling from different remotes (7080 ms) √ able to recover metadata after complex operations (16020 ms) × throw when trying to commit to a cloned vault (6939 ms) √ test pulling a vault that isn't remote (4135 ms) × pullVault respects locking (5953 ms) ● VaultManager › With remote agents › clone vaults from a remote keynode using a vault name ErrorPolykeyRemote: Ref HEAD cannot be found 201 | if (key === 'UNKNOWN' && errorData != null) { 202 | const error: Error = JSON.parse(errorData, reviver); > 203 | const remoteError = new grpcErrors.ErrorPolykeyRemote( | ^ 204 | metadata, 205 | error.message, 206 | { at toError (src/grpc/utils/utils.ts:203:29) at gf (src/grpc/utils/utils.ts:453:13) at constructor_.request (src/vaults/VaultInternal.ts:783:22) at src/vaults/VaultInternal.ts:164:55 at src/nodes/NodeConnectionManager.ts:204:16 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12) at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36) at src/vaults/VaultManager.ts:627:23 ● VaultManager › With remote agents › clone vaults from a remote keynode using a vault name with no history ErrorPolykeyRemote: Ref HEAD cannot be found 201 | if (key === 'UNKNOWN' && errorData != null) { 202 | const error: Error = JSON.parse(errorData, reviver); > 203 | const remoteError = new grpcErrors.ErrorPolykeyRemote( | ^ 204 | metadata, 205 | error.message, 206 | { at toError (src/grpc/utils/utils.ts:203:29) at gf (src/grpc/utils/utils.ts:453:13) at constructor_.request (src/vaults/VaultInternal.ts:783:22) at src/vaults/VaultInternal.ts:164:55 at src/nodes/NodeConnectionManager.ts:204:16 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12) at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36) at src/vaults/VaultManager.ts:627:23 ● VaultManager › With remote agents › clone and pull vaults using a vault id ErrorPolykeyRemote: Ref HEAD cannot be found 201 | if (key === 'UNKNOWN' && errorData != null) { 202 | const error: Error = JSON.parse(errorData, reviver); > 203 | const remoteError = new grpcErrors.ErrorPolykeyRemote( | ^ 204 | metadata, 205 | error.message, 206 | { at toError (src/grpc/utils/utils.ts:203:29) at gf (src/grpc/utils/utils.ts:453:13) at runMicrotasks () at constructor_.request (src/vaults/VaultInternal.ts:783:22) at src/vaults/VaultInternal.ts:164:55 at src/nodes/NodeConnectionManager.ts:204:16 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12) at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36) ● VaultManager › With remote agents › should reject Pulling when permissions are not set ErrorPolykeyRemote: Ref HEAD cannot be found 201 | if (key === 'UNKNOWN' && errorData != null) { 202 | const error: Error = JSON.parse(errorData, reviver); > 203 | const remoteError = new grpcErrors.ErrorPolykeyRemote( | ^ 204 | metadata, 205 | error.message, 206 | { at toError (src/grpc/utils/utils.ts:203:29) at gf (src/grpc/utils/utils.ts:453:13) at runMicrotasks () at constructor_.request (src/vaults/VaultInternal.ts:783:22) at src/vaults/VaultInternal.ts:164:55 at src/nodes/NodeConnectionManager.ts:204:16 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12) at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36) ● VaultManager › With remote agents › can pull a cloned vault ErrorPolykeyRemote: Ref HEAD cannot be found 201 | if (key === 'UNKNOWN' && errorData != null) { 202 | const error: Error = JSON.parse(errorData, reviver); > 203 | const remoteError = new grpcErrors.ErrorPolykeyRemote( | ^ 204 | metadata, 205 | error.message, 206 | { at toError (src/grpc/utils/utils.ts:203:29) at gf (src/grpc/utils/utils.ts:453:13) at runMicrotasks () at constructor_.request (src/vaults/VaultInternal.ts:783:22) at src/vaults/VaultInternal.ts:164:55 at src/nodes/NodeConnectionManager.ts:204:16 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12) at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36) ● VaultManager › With remote agents › manage pulling from different remotes ErrorPolykeyRemote: Ref HEAD cannot be found 201 | if (key === 'UNKNOWN' && errorData != null) { 202 | const error: Error = JSON.parse(errorData, reviver); > 203 | const remoteError = new grpcErrors.ErrorPolykeyRemote( | ^ 204 | metadata, 205 | error.message, 206 | { at toError (src/grpc/utils/utils.ts:203:29) at gf (src/grpc/utils/utils.ts:453:13) at runMicrotasks () at constructor_.request (src/vaults/VaultInternal.ts:783:22) at src/vaults/VaultInternal.ts:164:55 at src/nodes/NodeConnectionManager.ts:204:16 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12) at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36) ● VaultManager › With remote agents › throw when trying to commit to a cloned vault ErrorPolykeyRemote: Ref HEAD cannot be found 201 | if (key === 'UNKNOWN' && errorData != null) { 202 | const error: Error = JSON.parse(errorData, reviver); > 203 | const remoteError = new grpcErrors.ErrorPolykeyRemote( | ^ 204 | metadata, 205 | error.message, 206 | { at toError (src/grpc/utils/utils.ts:203:29) at gf (src/grpc/utils/utils.ts:453:13) at runMicrotasks () at constructor_.request (src/vaults/VaultInternal.ts:783:22) at src/vaults/VaultInternal.ts:164:55 at src/nodes/NodeConnectionManager.ts:204:16 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12) at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36) ● VaultManager › With remote agents › pullVault respects locking ErrorPolykeyRemote: Ref HEAD cannot be found 201 | if (key === 'UNKNOWN' && errorData != null) { 202 | const error: Error = JSON.parse(errorData, reviver); > 203 | const remoteError = new grpcErrors.ErrorPolykeyRemote( | ^ 204 | metadata, 205 | error.message, 206 | { at toError (src/grpc/utils/utils.ts:203:29) at gf (src/grpc/utils/utils.ts:453:13) at runMicrotasks () at constructor_.request (src/vaults/VaultInternal.ts:783:22) at src/vaults/VaultInternal.ts:164:55 at src/nodes/NodeConnectionManager.ts:204:16 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at constructor_.withConnF (src/nodes/NodeConnectionManager.ts:196:12) at Function.cloneVaultInternal (src/vaults/VaultInternal.ts:160:36) FAIL tests/vaults/utils.test.ts Vaults utils × EFS can be read recursively (393 ms) √ fs can be read recursively (7 ms) √ decodeNodeId does not throw an error (3 ms) ● Vaults utils › EFS can be read recursively EBUSY: resource busy or locked, unlink 'C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-8gaaEG\LOCK' FAIL tests/vaults/VaultOps.test.ts (172.971 s) VaultOps √ adding a secret (3108 ms) √ adding a secret and getting it (3070 ms) × able to make directories (3745 ms) √ adding and committing a secret 10 times (37850 ms) √ updating secret content (3456 ms) √ updating secret content within a directory (4506 ms) √ updating a secret 10 times (11808 ms) × deleting a secret (7323 ms) × deleting a secret within a directory (1778 ms) √ deleting a secret 10 times (18010 ms) √ renaming a secret (3594 ms) × renaming a secret within a directory (4516 ms) × listing secrets (7718 ms) √ listing secret directories (4938 ms) × adding hidden files and directories (4922 ms) × updating and deleting hidden files and directories (12287 ms) × adding a directory of 1 secret (2530 ms) √ adding a directory with subdirectories and files (4681 ms) √ testing the errors handling of adding secret directories (6501 ms) × adding a directory of 100 secrets with some secrets already existing (23817 ms) ● VaultOps › able to make directories expect(received).toContain(expected) // indexOf Expected value: "dir-3" Received array: ["dir-1", "dir-2", "dir-3\\dir-4", "dir-3\\dir-4\\secret-1"] 124 | expect(dir).toContain('dir-1'); 125 | expect(dir).toContain('dir-2'); > 126 | expect(dir).toContain('dir-3'); | ^ 127 | 128 | expect(await efs.readdir('dir-3')).toContain('dir-4'); 129 | expect(await efs.readdir(path.join('dir-3', 'dir-4'))).toContain( at tests/vaults/VaultOps.test.ts:126:19 at src/vaults/VaultInternal.ts:427:14 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at Object. (tests/vaults/VaultOps.test.ts:122:5) ● VaultOps › deleting a secret expect(received).rejects.toThrow() Received promise resolved instead of rejected Resolved to value: undefined 204 | ); 205 | await vaultOps.deleteSecret(vault, 'secret-1'); > 206 | await expect(() => vaultOps.deleteSecret(vault, 'dir-1')).rejects.toThrow(); | ^ 207 | await vaultOps.deleteSecret(vault, path.join('dir-1', 'secret-2')); 208 | await vaultOps.deleteSecret(vault, 'dir-1'); 209 | await expect(vault.readF((efs) => efs.readdir('.'))).resolves.not.toContain( at expect (node_modules/expect/build/index.js:128:15) at Object. (tests/vaults/VaultOps.test.ts:206:11) ● VaultOps › deleting a secret within a directory expect(received).rejects.toThrow() Received promise resolved instead of rejected Resolved to value: undefined 212 | }); 213 | test('deleting a secret within a directory', async () => { > 214 | await expect(() => | ^ 215 | vaultOps.mkdir(vault, path.join('dir-1', 'dir-2')), 216 | ).rejects.toThrow(errors.ErrorVaultsRecursive); 217 | await vaultOps.mkdir(vault, path.join('dir-1', 'dir-2'), { at expect (node_modules/expect/build/index.js:128:15) at Object. (tests/vaults/VaultOps.test.ts:214:11) ● VaultOps › renaming a secret within a directory expect(received).resolves.toContain(expected) // indexOf Expected value: "secret-change" Received array: [] 268 | path.join(dirPath, 'secret-change'), 269 | ); > 270 | await expect(vault.readF((efs) => efs.readdir(dirPath))).resolves.toContain( | ^ 271 | `secret-change`, 272 | ); 273 | }); at Object.toContain (node_modules/expect/build/index.js:194:22) at Object. (tests/vaults/VaultOps.test.ts:270:71) ● VaultOps › listing secrets expect(received).toStrictEqual(expected) // deep equality - Expected - 1 + Received + 1 Array [ - "dir1/dir2/secret-3", + "dir1\\dir2\\secret-3", "secret-1", "secret-2", ] 281 | 'secret-content', 282 | ); > 283 | expect((await vaultOps.listSecrets(vault)).sort()).toStrictEqual( | ^ 284 | ['secret-1', 'secret-2', 'dir1/dir2/secret-3'].sort(), 285 | ); 286 | }); at Object. (tests/vaults/VaultOps.test.ts:283:56) ● VaultOps › adding hidden files and directories ErrorEncryptedFSError: ENOENT: no such file or directory, .hiddenDir\.hiddenInSecret 67 | for (const dirent of dirents) { 68 | const res = path.join(dir, dirent.toString()); > 69 | const stat = await fs.promises.stat(res); | ^ 70 | if (stat.isDirectory()) { 71 | yield* readdirRecursively(fs, res); 72 | } else if (stat.isFile()) { at node_modules/encryptedfs/src/EncryptedFS.ts:2932:15 at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12) at readdirRecursively (src/vaults/utils.ts:69:18) at Object.readdirRecursively (src/vaults/utils.ts:71:7) at src/vaults/VaultOps.ts:256:22 at src/vaults/VaultInternal.ts:427:14 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at Object.listSecrets (src/vaults/VaultOps.ts:254:10) at Object. (tests/vaults/VaultOps.test.ts:328:18) ● VaultOps › updating and deleting hidden files and directories ErrorEncryptedFSError: ENOENT: no such file or directory, .hidingDir\.hiddenInSecret 67 | for (const dirent of dirents) { 68 | const res = path.join(dir, dirent.toString()); > 69 | const stat = await fs.promises.stat(res); | ^ 70 | if (stat.isDirectory()) { 71 | yield* readdirRecursively(fs, res); 72 | } else if (stat.isFile()) { at node_modules/encryptedfs/src/EncryptedFS.ts:2932:15 at Object.maybeCallback (node_modules/encryptedfs/src/utils.ts:405:12) at readdirRecursively (src/vaults/utils.ts:69:18) at Object.readdirRecursively (src/vaults/utils.ts:71:7) at src/vaults/VaultOps.ts:256:22 at src/vaults/VaultInternal.ts:427:14 at withF (node_modules/@matrixai/resources/src/utils.ts:24:12) at Object.listSecrets (src/vaults/VaultOps.ts:254:10) at Object. (tests/vaults/VaultOps.test.ts:351:18) ● VaultOps › adding a directory of 1 secret expect(received).resolves.toContain(expected) // indexOf Expected value: "secret" Received array: [] 380 | await expect( 381 | vault.readF((efs) => efs.readdir(secretDirName)), > 382 | ).resolves.toContain('secret'); | ^ 383 | 384 | await fs.promises.rm(secretDir, { 385 | force: true, at Object.toContain (node_modules/expect/build/index.js:194:22) at Object. (tests/vaults/VaultOps.test.ts:382:16) ● VaultOps › adding a directory of 100 secrets with some secrets already existing expect(received).resolves.toContain(expected) // indexOf Expected value: "secret 0" Received array: [] 512 | await expect( 513 | vault.readF((efs) => efs.readdir(secretDirName)), > 514 | ).resolves.toContain('secret ' + j.toString()); | ^ 515 | } 516 | expect( 517 | ( at Object.toContain (node_modules/expect/build/index.js:194:22) at Object. (tests/vaults/VaultOps.test.ts:514:20) Test Suites: 4 failed, 4 total Tests: 24 failed, 75 passed, 99 total Snapshots: 0 total Time: 364.67 s Ran all test suites matching /tests\\vaults/i. GLOBAL TEARDOWN Destroying Global Data Dir: C:\Users\EMMACA~1\AppData\Local\Temp\polykey-test-global-A3ophi ```
emmacasolin commented 2 years ago

Fixing the failures mentioned in this issue needs to be done before working on the integration tests for these platforms, since there are likely underlying issues we aren't aware of and that would make it difficult to work on integration tests. The failures can be grouped into a few different categories:

  1. ENOENT - We seem to frequently see ENOENT errors when attempting to spawn a child process on Windows. We also sometimes see ENOENT being returned from operations using the EFS (also on Windows) and when attempting to write to files (e.g. keys) on Mac - potentially related to this https://github.com/MatrixAI/js-db/pull/38#issuecomment-1183229572
  2. Pathnames - This one is a very simple issue, and it may even be the underlying cause of a lot of other failures on Windows. This refers to Windows using \ to separate pathnames rather than /
  3. Ref HEAD cannot be found - This could potentially be related to the issue with path names on Windows using \ instead of /, however we often see this error when performing vaults operations on remote agents
  4. Permissions - An example of this is Error: EPERM: operation not permitted, unlink 'C:\Users\GITLAB~1\AppData\Local\Temp\polykey-test-m4rbmK\token' in the sessions tests (Windows)
  5. EBUSY - There seem to be some locking issues on Windows, for example when writing to keypairs, e.g. EBUSY: resource busy or locked, unlink 'C:\Users\GITLAB~1\AppData\Local\Temp\polykey-test-4f3f1G\db\000005.ldb'
  6. Timeouts - Almost all of the failures on Mac are timeouts, however, some of these timeouts persist even when only a single test is run, so I think something else is going on here that causes these timeouts.
CMCDragonkai commented 2 years ago

I've found additional problems on windows with respect to process.stdout.write and console programs.

There are 3 dimensions to dealing with Windows for console programs.

  1. Asynchronous vs synchronous - https://nodejs.org/api/process.html#a-note-on-process-io
  2. Encoding - we want to standardise on UTF-8
  3. Line Endings - posix uses LF, windows uses CRLF, we can use os.EOL to auto switch

I've found that powershell and cmd also behave differently.

I've written this test script:

const os = require('os');
const process = require('process');
const fs = require('fs');
const { Buffer } = require ('buffer');

async function main(argv = process.argv) {
  argv = argv.slice(2);
  if (argv[0] === '0') {
    console.log('Hello 盘');
  } else if (argv[0] === '1') {
    process.stdout.write(`HELLO${os.EOL}盘${os.EOL}`);
  } else if (argv[0] === '2') {
    // LF
    process.stdout.write('HELLO\n盘\n');
  } else if (argv[0] === '3') {
    // CRLF
    process.stdout.write('HELLO\r\n盘\r\n');
  } else if (argv[0] === '4') {
    // WITH setting the utf-8 encoding
    process.stdout.setDefaultEncoding('utf-8');
    // process.stdout.setEncoding('utf-8');
    process.stdout.write('HELLO\n盘\n');
  } else if (argv[0] === '5') {
    // WITH setting the utf-8 encoding
    process.stdout.setDefaultEncoding('utf-8');
    process.stdout.write('HELLO\r\n盘\r\n');
  } else if (argv[0] === '6') {
    // Writing buffers directly
    process.stdout.write(Buffer.from('hello\n盘\n', 'utf-8'));
  } else if (argv[0] === '7') {
    // Writing buffers directly
    process.stdout.write(Buffer.from('hello\r\n盘\r\n', 'utf-8'));
  } else if (argv[0] === '8') {
    // Setting encoding AND writing buffers
    process.stdout.setDefaultEncoding('utf-8');
    // process.stdout.setEncoding('utf-8');
    process.stdout.write(Buffer.from('hello\n盘\n', 'utf-8'));
  } else if (argv[0] === '9') {
    // Setting encoding AND writing buffers
    process.stdout.setDefaultEncoding('utf-8');
    process.stdout.write(Buffer.from('hello\r\n盘\r\n', 'utf-8'));
  } else if (argv[0] === '10') {
    process.stdout.write(
      Buffer.from('hello\n盘\n', 'utf-8'),
      'utf-8'
    );
  } else if (argv[0] === '11') {
    // Writing with fs
    fs.writeFileSync(1, 'hello\n盘\n');
  } else if (argv[0] === '12') {
    // Writing with fs
    fs.writeFileSync(1, 'hello\r\n盘\r\n');
  } else if (argv[0] === '13') {
    // Writing with fs and specifying encoding
    fs.writeFileSync(1, 'hello\n盘\n', 'utf-8');
  } else if (argv[0] === '14') {
    // Writing with fs and specifying encoding
    fs.writeFileSync(1, 'hello\r\n盘\r\n', 'utf-8');
  } else if (argv[0] === '15') {
    // Writing with fs and buffers
    fs.writeFileSync(1, Buffer.from('hello\n盘\n', 'utf-8'));
  } else if (argv[0] === '16') {
    // Writing with fs and buffers
    fs.writeFileSync(1, Buffer.from('hello\r\n盘\r\n', 'utf-8'));
  } else if (argv[0] === '17') {
    // Writing to a file
    fs.writeFileSync('./lf.txt', 'hello\n盘\n');
    fs.writeFileSync('./crlf.txt', 'hello\r\n盘\r\n');
  }
}

void main();

Then I've ran node ./test.js X > ./tmp/X.txt and vary the X from 0 to 16, then run node ./test.js 17.

Or use this script on powershell.

for (($i = 0); $i -lt 17; $i++) { node ./test.js $i > ./tmp/$i.txt }
node ./test.js 17

The results show this:

Powershell ALWAYS converts everything to UTF-16 LE with CRLF, no exceptions, the only case where this doesn't occur is with 17, where nodejs directly writes to files.

»» ~/Desktop
 ♖ file *.txt                                                                                                       (master) pts/2 15:52:01
0.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
10.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
11.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
12.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
13.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
14.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
15.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
16.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
1.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
2.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
3.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
5.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
6.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
7.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
8.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
9.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
10.txt:    Unicode text, UTF-16, little-endian text, with CRLF line terminators
crlf.txt: Unicode text, UTF-8 text, with CRLF line terminators
lf.txt:   Unicode text, UTF-8 text

CMD doesn't actually support unicode characters on render, and it can't even show them anyway. So the Chinese character becomes just a question mark box. But what about LF/CRLF and encoding?

Well it turns out, that CMD preserves unicode encoding during piping. It does not convert it to UTF-16 LE. It also preserves line endings, it does not auto convert LF to CRLF. Basically it does what Linux does, and it is far more obvious about the behaviour.

»» ~/Desktop/cmd
 ♖ file *.txt                                                                                                       (master) pts/2 15:55:53
0.txt:    Unicode text, UTF-8 text
10.txt:   Unicode text, UTF-8 text
11.txt:   Unicode text, UTF-16, little-endian text, with CRLF line terminators
12.txt:   Unicode text, UTF-8 text
13.txt:   Unicode text, UTF-8 text
14.txt:   Unicode text, UTF-8 text, with CRLF line terminators
15.txt:   Unicode text, UTF-8 text
16.txt:   Unicode text, UTF-8 text, with CRLF line terminators
1.txt:    Unicode text, UTF-8 text, with CRLF line terminators
2.txt:    Unicode text, UTF-8 text
3.txt:    Unicode text, UTF-8 text, with CRLF line terminators
4.txt:    Unicode text, UTF-8 text
5.txt:    Unicode text, UTF-8 text, with CRLF line terminators
6.txt:    Unicode text, UTF-8 text
7.txt:    Unicode text, UTF-8 text, with CRLF line terminators
8.txt:    Unicode text, UTF-8 text
9.txt:    Unicode text, UTF-8 text, with CRLF line terminators
crlf.txt: Unicode text, UTF-8 text, with CRLF line terminators
lf.txt:   Unicode text, UTF-8 text
CMCDragonkai commented 2 years ago

Let's figure out a principle of least surprise here.

At first, I was thinking that we should use os.EOL to ensure that when printing to a terminal, we are always outputting the OS-specific line endings.

However if someone is working across platforms, then it can be quite surprising if users expect that a text file that they output/pipe contains CRLF while normal file writes from PK creates LF line endings. This is because we do tell users to use > and direction operators to write to files with our CLI.

We would preserve whatever line endings the user uses when they write the file to PK.

At the same time, it appears windows CMD and powershell understand \n and will create newlines.

Therefore:

  1. We should continue to use \n and not os.EOL when outputting anything on the terminal. Both CMD and powershell will render it as a newline.
  2. We should preserve the input newlines whatever they are if they are CRLF or LF. This means when outputting or writing to a file, we will preserve whatever binary data that was inputted in the first place. This should have integration tests checking this. If users get confused, we would ask them to ensure that they are using LF line endings to maintain compatibility... and we can eventually add an option to auto-convert CRLF to LF or LF to CRLF...

However we still have a problem:

  1. Is there a way to get CMD to render characters?
  2. Is there a way to tell powershell NOT to auto convert to CRLF, and not to auto convert to UTF-16?

Now according to https://nodejs.org/api/fs.html#fswritefd-string-position-encoding-callback

On Windows, if the file descriptor is connected to the console (e.g. fd == 1 or stdout) a string containing non-ASCII characters will not be rendered properly by default, regardless of the encoding used. It is possible to configure the console to render UTF-8 properly by changing the active codepage with the chcp 65001 command. See the chcp docs for more details.

But of course this would be an instruction to end users to set this up. Let me check.

CMCDragonkai commented 2 years ago

Using chcp 65001 is actually not enough. One must also be using a font that supports unicode characters. I switched to lucida console from the default "Consolas", but again it is not capable of displaying Chinese characters.

These 2 Q&A show how complex it is for windows to setup to properly use UTF-8:

CMCDragonkai commented 2 years ago

Using intl.cpl with:

image

That's sufficient to change to chcp 65001 by default across the entire opreating system. This could be recommended to end users... but again could be quite problematic if the rest of the operating system isn't ready.

At the same time, one must choose a correct font. Luicda Console or the default Consolas is fine for displaying some unicode, but only SimSun-ExtB can do Chinese characters.

The font selection doesn't appear to affect SSHing into the powershell. That probably relies on whatever chcp it is. So at the end of the day, users may recommended to use chcp 65001 or stick it into their profile.

THIS however does not fix the problem of piping. Still piping into any file proceeds to convert LF to CRLF and converts to UTF 16 LE.

CMCDragonkai commented 2 years ago

Alternatively users can be recommended to use the new Windows terminal program or ConHost or ConEmu as alternatives.

It just seems at least on Windows, terminal programs are just not first class atm.

CMCDragonkai commented 2 years ago

This Q&A https://stackoverflow.com/questions/40098771/changing-powershells-default-output-encoding-to-utf-8 explains how to actually change things like > and >> to using utf-8. However on 5.1, this still creates UTF-8 BOM.

Powershell Core which is v6+ and this is not installed by default on Windows atm... does default to utf-8 BOMless. So we can also point users to downloading and installing powershell v6 (powershell core) instead of the original powershell.

I suspect this is a similar issue with LF to CRLF. We may just wait for more later version of powershell.

CMCDragonkai commented 2 years ago

The end result is that:

  1. Async vs Sync - no need to do anything for now
  2. Encoding: To be explicit we should add in utf-8 as the second parameter to any process.stdout.write(..., 'utf-8'), this just ensures that we are in fact writing with utf-8. Alternatively we can use process.stdout.setDefaultEncoding('utf-8') and process.stderr.setDefaultEncoding('utf-8'). At the beginning! If we are using stdin, we should also set process.stdin.setEncoding('utf-8');. This is for read stream. Powershell may be converting this to UTF-16LE, but we will point to fixes relevant to CMD and powershell for the end user to deal with. CMD does not auto convert, but people don't really use CMD anymore.
  3. LF vs CRLF - again because we are using \n and not CRLF, it is up to the end user to fix this appropriately. CMD and Powershell both render \n properly though. These 2 issues are relevant though:
CMCDragonkai commented 2 years ago

@tegefaulkes @emmacasolin I've left this change https://github.com/MatrixAI/js-polykey/issues/401#issuecomment-1186819981 on by default in matrix-win-1. Note that SSH does not get affected by font changes. It does get affected by the code page though. Gitlab SAAS should also make the relevant changes but I don't know if they did.

CMCDragonkai commented 2 years ago

Some additional resources here https://shapeshed.com/writing-cross-platform-node/. Ignore the $HOMEPATH it should be %USERPROFILE or $USERPROFILE is actually more correct.

CMCDragonkai commented 2 years ago

Regarding async vs sync. As I've been trying to debug some test issues, having async console outputs by default is making this difficult. So let's standardise for 1. as well.

// Default behaviour on Node.js:
// Files: synchronous on Windows and POSIX
// TTYs (Terminals): asynchronous on Windows, synchronous on POSIX
// Pipes (and sockets): synchronous on Windows, asynchronous on POSIX
// In order to align Windows with POSIX behaviour:
if (process.stdout.isTTY) {
  process.stdout._handle.setBlocking(true);
} else if (os.platform() === 'win32' && !process.stdout.isTTY) {
  process.stdout._handle.setBlocking(false);
}

The relevant issue is https://github.com/nodejs/node/issues/11568.

The above might need to be done in PK, as it is a "global" application setting, rather than in js-logger.

CMCDragonkai commented 2 years ago

However if we want this to be the case for tests, which just the js-logger directly, they will need to set those settings in the js-logger instead.

CMCDragonkai commented 2 years ago

I think needs a PR on the js-logger for the StreamHandler. It will be necessary to ensure consistent behaviour.


This has been created https://github.com/MatrixAI/js-logger/issues/22

CMCDragonkai commented 2 years ago

These build tests run on stage build, not stage integration. So these have to be done before any further integration testing on Windows or MacOS specified in MatrixAI/Polykey-CLI#10.

CMCDragonkai commented 2 years ago

This can only be done after MatrixAI/Polykey#419 is merged. In MatrixAI/Polykey#419, I want to integrate jest-extended and fast-check for more robust concurrency testing, obvious js-db 5.0.0 will also solve a bunch of concurrency problems too. Alot of testing non-determinism may happen due to concurrency scheduling, so more robust model based checks should be done when testing asynchronous effects.

tegefaulkes commented 5 months ago

I'm trying to gauge weather this is done. I've recently fixed the mac and windows build. There has been a bunch of manual testing on the mac build so we can safely say that is working. Windows should be working but it has its own idiosyncrasies so it needs some of it's own manual testing. But the point is these builds are working

There are CI integration jobs for all builds and they are running and passing but the windows and mac builds just run the help text. So minimally it's checking runtime loading of all dependencies. So the platform specific integration tests need to be expanded. but I'm not sure if that's a requirement of this issue.

CMCDragonkai commented 4 weeks ago

I think this issue would evolve into a high level issue targeting all sorts of Windows related compatibility. We haven't done alot of work in Windows yet as we have still problems to fix in general. So fixing up platform specific issues for Windows has to be put till later, and identifying those tests are relevant.

CMCDragonkai commented 4 weeks ago

The above comments are all relevant to various windows related issues. We can group these all together under a Windows compatibility project.