CLI Beta Launch

[CMCDragonkai]

Specification

This epic focuses on the CLI beta launch targeting November 10 2023.

This follows from the 6th testnet deployment https://github.com/MatrixAI/Polykey/issues/551 and focuses on any UX issues that should be resolved before we launch. Plus any documentation related things and metrics for tracking how the launch went, as well as content we need to write to prepare for it.

We should also get our demo video fully polished as well.

One of the things we need to do is:

1. Add a audit or logs domain to PK at first to track information about the network, we will need this as part of our metrics.
2. Create a simple dashboard on testnet.polykey.com and mainnet.polykey.com so we can see what's going on and show everybody how the network is building up.

We are currently working through this list of issues. https://github.com/MatrixAI/Polykey-CLI/issues/40#issuecomment-1853222990

Additional context

• https://github.com/MatrixAI/Polykey/issues/551

Tasks

1. Spec out the experience for the launch
2. Schedule content releases
3. Plan out all UX-related issues

[tegefaulkes]

Here's a basic script I used for the demonstration.

starting an agent

1. start an agent with ./polykey agent start. Enter the password if its the first time starting
2. we can verify its running with ./polykey agent status

creating a vault with secrets

1. create a vault with ./polykey vaults create testvault.
2. we can list the vaults with ./polykey vaults list.
3. create a secret we want to add to the vault with echo somesecret > secret
4. add it to a vault with ./polykey secrets create ./secret testvault:secret
5. we can see the secret with ./polykey secrets list testvault
6. we can see the secret with ./polykey secrets get testvault:secret
7. lets modify the secret with echo newsecret > secret
8. and see it with ./polykey secrets get testvault:secret.

viewing the network

1. Polykey connects to the testnet at startup. we can see the active connections with ./polykey nodes connections
2. We can list all seen nodes with ./polykey nodes getall
3. I can form connections to nodes with ./polykey nodes ping <NODEID> This will find the node within the wider network as well as looking for it in the local network if multicast is enabled.

trusting a peer

1. For trust we need a 2nd node to to trust. If running a 2nd node on the same machine we can start it with ./polykey agent start -np ./nodeb
2. We can see it's nodeId wth ./polykey agent status -np ./nodeb
3. get the first nodes nodeID with ./polykey agent status.
4. verify you can connect to this node from nodeA with ./polykey nodes ping <NODEID>
5. have the nodes discover each other with ./polykey identities discover <NodeIdB>
6. have the nodes discover each other with ./polykey identities discover -np ./nodeb <NodeIdA>
7. nodeB needs to trust nodeA to allow it to share vaults. ./polykey identities trust -np ./nodeb <NodeIdA>
8. and vice versa ./polykey idenitites trust <NodeIdB>
9. We can now share the nodeA's vault with nodeB with ./polykey vaults share testvault <NodeIdB>
10. nodeB can now clone the vault with ./polykey vaults clone -np ./nodeb testvault <NodeIdA>
11. We can now list the vault with ./polykey vaults list -np ./nodeb
12. and list the secrets ./polykey secrets list testvault -np ./nodeb
13. and get the secret ./polykey secrets get testvault:secret -np ./nodeb

[CMCDragonkai]

We cannot abide by the default https://nodejs.org/api/events.html#nodeeventtargetsetmaxlistenersn. We may have legitimate reasons for having lots of event listeners on our event targets.

However we also need to keep track of this count internally. This can be part of our diagnostics domain to avoid internal memory and resource leaks by maintaining counters of all the resources being created.

We may be able to make use of some async monitors for this. It needs to be fast and may be runtime dependent. One thing I'd like to start with is tracking floating promises.

[CMCDragonkai]

We cannot abide by the default https://nodejs.org/api/events.html#nodeeventtargetsetmaxlistenersn. We may have legitimate reasons for having lots of event listeners on our event targets.

However we also need to keep track of this count internally. This can be part of our diagnostics domain to avoid internal memory and resource leaks by maintaining counters of all the resources being created.

We may be able to make use of some async monitors for this. It needs to be fast and may be runtime dependent. One thing I'd like to start with is tracking floating promises.

A quick solution right now is to raise the number marginally more. What would it need to be to avoid the warnings as of the current state. Are there situations where the deepest callstack may provide us the largest possible number of event listeners?

Also I'm interpreting this as due to our js-contexts usage right? There's no other places where we are dynamically getting lots of event listeners? And this could be a problem if during a large recursion into the same contextified function?

[tegefaulkes]

We cannot abide by the default https://nodejs.org/api/events.html#nodeeventtargetsetmaxlistenersn. We may have legitimate reasons for having lots of event listeners on our event targets. However we also need to keep track of this count internally. This can be part of our diagnostics domain to avoid internal memory and resource leaks by maintaining counters of all the resources being created. We may be able to make use of some async monitors for this. It needs to be fast and may be runtime dependent. One thing I'd like to start with is tracking floating promises.

A quick solution right now is to raise the number marginally more. What would it need to be to avoid the warnings as of the current state. Are there situations where the deepest callstack may provide us the largest possible number of event listeners?

Also I'm interpreting this as due to our js-contexts usage right? There's no other places where we are dynamically getting lots of event listeners? And this could be a problem if during a large recursion into the same contextified function?

I have a new issue that ties into this problem at https://github.com/MatrixAI/js-rpc/issues.

[tegefaulkes]

Going through the open issues I've picked out the following that still seem relevant here.

1. Polykey-CLI
   1. MatrixAI/Polykey-CLI/issues/31
   2. MatrixAI/Polykey-CLI/issues/78
   3. MatrixAI/Polykey-CLI/issues/74
   4. MatrixAI/Polykey-CLI/issues/73
   5. MatrixAI/Polykey-CLI/issues/37
   6. General UX testing and polish
   7. MatrixAI/Polykey-CLI/issues/76
   8. MatrixAI/Polykey-CLI/issues/38
   9. MatrixAI/Polykey-CLI/issues/30
2. Polykey
   1. MatrixAI/Polykey/issues/642
   2. MatrixAI/Polykey/issues/653
   3. MatrixAI/Polykey/issues/482
   4. MatrixAI/Polykey-CLI#99
   5. MatrixAI/Polykey-CLI#80
   6. MatrixAI/Polykey/issues/242
   7. Do mdns queries happen passively as well? Can we make a synchronious query of already seen services for our target NodeID when doing findByMDNS? It would solve some timeout nuance. Something to discuss
   8. MatrixAI/Polykey/issues/652
   9. MatrixAI/Polykey/issues/651
   10. MatrixAI/MatrixAI-Graph#42
   11. MatrixAI/Polykey/issues/648
   12. MatrixAI/Polykey/issues/641
   13. MatrixAI/Polykey/issues/625
   14. MatrixAI/Polykey/issues/594
   15. MatrixAI/Polykey/issues/564
   16. MatrixAI/Polykey/issues/537
   17. MatrixAI/Polykey/issues/523
   18. MatrixAI/Polykey/issues/476
   19. MatrixAI/Polykey/issues/461
3. js-quic
   1. MatrixAI/js-quic/issues/78
   2. MatrixAI/js-quic/issues/77
   3. MatrixAI/js-quic/issues/83
   4. MatrixAI/js-quic/issues/64

[CMCDragonkai]

And this https://github.com/MatrixAI/Polykey-Docs/pull/33.

@amydevs to review this tomorrow.

[CMCDragonkai]

Current priorities:

• https://github.com/MatrixAI/Polykey/issues/599 - @amydevs
• https://github.com/MatrixAI/Polykey-CLI/issues/71 - @tegefaulkes

The main things to finish off what is necessary for an online launch before focusing on bugs and features in PK's core.

These 2 should be doable by Christmas.

[tegefaulkes]

Things are mostly working with the docker integration tests now except for 1 problem.

One of the tests is failing due to a CLI call being made to an agent failing. It's failing with...

{"level":"ERROR","keys":"polykey.PolykeyClient.WebSocketClient","msg":"ErrorWebSocketConnectionLocal: WebSocket Connection local error - WebSocket could not open due to internal error"}
{"level":"ERROR","keys":"polykey.PolykeyClient.WebSocketClient.WebSocketConnection 0","msg":"ErrorWebSocketConnectionLocal: WebSocket Connection local error - WebSocket could not open due to internal error"}
{"type":"ErrorWebSocketConnectionLocal","data":{"message":"WebSocket could not open due to internal error","timestamp":"2023-12-20T03:55:11.025Z","data":{"errorCode":1011,"reason":"WebSocket could not open due to internal error"},"cause":{"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":36569},"stack":"ErrorWebSocketConnectionLocal: WebSocket could not open due to internal error\n    at WebSocket.openErrorHandler (/builds/MatrixAI/open-source/Polykey-CLI/node_modules/@matrixai/ws/src/WebSocketConnection.ts:693:16)\n    at Object.onceWrapper (node:events:629:26)\n    at WebSocket.emit (node:events:514:28)\n    at emitErrorAndClose (/builds/MatrixAI/open-source/Polykey-CLI/node_modules/ws/lib/websocket.js:1016:13)\n    at ClientRequest.<anonymous> (/builds/MatrixAI/open-source/Polykey-CLI/node_modules/ws/lib/websocket.js:864:5)\n    at ClientRequest.emit (node:events:514:28)\n    at TLSSocket.socketErrorListener (node:_http_client:501:9)\n    at TLSSocket.emit (node:events:514:28)\n    at emitErrorNT (node:internal/streams/destroy:151:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:116:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)"}}

Specifically the ECONNREFUSED when making the connection. The agent is running fine here, it works locally so I can only assume it's some weirdness with the DIND networking setup in the CI job.

I'll do a little more digging but this isn't the priority right now. So I may have to shortcut the tests for now. I can see that the connections are being made and the agent isn't crashing so that'll have to be enough for now.

[tegefaulkes]

I'm adding https://github.com/MatrixAI/Polykey-CLI/issues/90 to this to track all the changes needed for the CI integration testing

[tegefaulkes]

In no particular order, just picking out things to do.

1. MatrixAI/Polykey/issues/642
2. MatrixAI/Polykey-CLI#99
3. explore vaults domain optimizations
4. MatrixAI/Polykey-CLI/issues/78
5. MatrixAI/Polykey-CLI/issues/76
6. MatrixAI/Polykey-CLI/issues/73
7. MatrixAI/Polykey-CLI/issues/37

[tegefaulkes]

I need to make an issue about tracking down the potential EventTarget/AbortSignal handler leak. Right now we get warnings about it but I don't think it's a major issue.

[tegefaulkes]

So I've been looking into why the vaults and adding secrets is slow. From what I can tell the problem stems from two parts.

1. Each time we write a file we do a commit. I don't know how involved read/write wise making a commit with isomorphic git is but it's likely doing a few operations. From the vaults perspective, most of the time is spent on committing.
2. Just writing and reading a file in EFS is about 30 ops/s That is already not that fast. Even just opening an file and getting a fd is about 100 ops/s

I think we need to add some more bench-marking in EFS to narrow down the performance bottlenecks and try to optimise it.

[tegefaulkes]

New issue for the EventTarget leak at https://github.com/MatrixAI/js-quic/issues/80.

I've found the problem, just needs some discussion.

[CMCDragonkai]

I noticed that the polykey ECR image is still just polykey and it's name hasn't changed to polykey-cli. Like in:

registry_image='015248367786.dkr.ecr.ap-southeast-2.amazonaws.com/polykey'

@amydevs it should be changed to polykey-cli for our ECR images since polykey is going to be reserved for the library stuff.

This would align with the release output names given that we produce polykey-cli in:

I've also removed the image publishing documentation from README.md because it's no longer relevant now, it's being moved to Polykey-Docs under the development guide.

@brynblack you might want to keep awareness about this.

[CMCDragonkai]

Also I've regenerated the docs, it hadn't been done since this repo was first created. We would probably want to generate the docs automatically in our CI/CD instead of leaving it here in our source to avoid manual creation. It also reveals useful information like broken links too.

[CMCDragonkai]

Newly deployed docs looks alot better:

@brynblack feel free to give feedback as to whether auto generated docs is useful at all. I don't really read it myself since I just go through the source code.

[CMCDragonkai]

@tegefaulkes can you add the relevant issues into this epic via zenhub? And then update whether they are in-progress or otherwise. Keep this epic organised.

[CMCDragonkai]

The conclusion of this issue should update the version of polykey-cli to 0.X. So right now we are on 0.1.Y. So it should be whatever the latest 0.1.Y we end up with.

I reckon the next version after that will be 1.0.0.

[tegefaulkes]

@tegefaulkes can you add the relevant issues into this epic via zenhub? And then update whether they are in-progress or otherwise. Keep this epic organised.

I've been doing this already. I've been adding the new ones I've created and updated the status for the issues I've been working on. I haven't touched the status for others however.

[CMCDragonkai]

I've noticed that the integration:prerelease doesn't run normally. And therefore when we make a release tag, an image is pushed to the GitLab container registry, but it doesn't update the testnet tag. Which is confusing because the mainnet tag ends up being "later" then the testnet tag.

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/container_registry

In my infrastructure planning, I'm making diagram about how our distribution infrastructure works.

[CMCDragonkai]

Some feedback on the error reporting concept: https://github.com/MatrixAI/js-rpc/issues/57#issuecomment-1913884490.

I think we should swap the cause chain.

[tegefaulkes]

Going over the videos and discord comments I've compiled a shortlist of things to address.

1. ~Running Polykey-CLI with just the help text has a bit of a delay. It's slow to start, but I don't think slow in general. I think that the PKG or commander Has some startup cost involved. Probably PKG since it needs to unpack some stuff to even run.~ https://github.com/MatrixAI/Polykey-CLI/issues/102
2. ~When Oauthed on github the app is refered to as Polykey Desktop. it should be either just Polykey or Polykey CLI for the CLI specifically.~ https://github.com/MatrixAI/Polykey-CLI/issues/103
3. ~Sync node graph failed. Are the seed nodes running? Check AWS, otherwise I need to test locally. If that works then I need more details.~ https://github.com/MatrixAI/js-quic/commits/staging/
4. ~Allow for untrusted notifications but as a separate list from the trusted ones.~ https://github.com/MatrixAI/Polykey-CLI/issues/105
5. ~Identities search has no separation between listed identities. The formatting needs to be fixed.~ https://github.com/MatrixAI/Polykey-CLI/issues/106
6. ~When getting RPC errors we're meant to unwrap the remoteRPC error and just show the cause. This links into the verbosity affecting the error message.~ https://github.com/MatrixAI/Polykey-CLI/issues/107
7. ~trusting identities is case sensitive, we may want to fuzzy search and or recommend a similar identity if the one provided is wrong somehow.~ https://github.com/MatrixAI/Polykey-CLI/issues/108
8. ~Should have a command to reset local state~ https://github.com/MatrixAI/Polykey-CLI/issues/109
9. ~Should have the ability to list out all identity providers available~ https://github.com/MatrixAI/Polykey-CLI/issues/110
10. ~The identity claim still times too quickly~ Reopened https://github.com/MatrixAI/Polykey/issues/588
11. ~update to node 20 and switch out vercel/pkg for https://github.com/yao-pkg/pkg~ https://github.com/MatrixAI/Polykey-CLI/issues/111

I'll be creating new issues or adding to existing issues to address these.

[tegefaulkes]

We get the following warning when running the CLI

(node:610526) [DEP0112] DeprecationWarning: Socket.prototype._handle is deprecated
(Use `node --trace-deprecation ...` to show where the warning was created)

@amydevs mentioned it's due to MDNS. I think we need to look into removing the warning. I don't see any issues about this however so maybe we should make a new one and look into it?

It seems that MDNS is getting the handle (fd?) of the socket to modify it. I can see us needing direct access to the socket in native code for js-quic as well for a performance upgrade. So solving this can be applied there as well.

[tegefaulkes]

even shorter list:

1. https://github.com/MatrixAI/Polykey/issues/588
2. https://github.com/MatrixAI/Polykey/issues/453
3. https://github.com/MatrixAI/Polykey-CLI/issues/76
4. https://github.com/MatrixAI/Polykey/issues/537
5. https://github.com/MatrixAI/Polykey-CLI/issues/111
6. https://github.com/MatrixAI/Polykey-CLI/issues/22
7. https://github.com/MatrixAI/Polykey-CLI/issues/6

low low priority

1. https://github.com/MatrixAI/Polykey/issues/648
2. https://github.com/MatrixAI/Polykey/issues/653
3. https://github.com/MatrixAI/Polykey/issues/461
4. https://github.com/MatrixAI/Polykey-CLI/issues/17

@amy pick from these and address them

[tegefaulkes]

Secrets env command is completed now. I'll be moving on to working on the short short list OR fixing and re-enabling windows and mac CI test jobs

[amydevs]

pr MatrixAI/Polykey-Docs#56 should be ready now

[tegefaulkes]

I've noticed that the test NodeManager › with peers in network › findNode by signalled connections › handles offline nodes fails randomly and more often than I'd like. We need to look into it.

[tegefaulkes]

We're getting close to completing the main stuff. @amydevs is looking into the issues with discovery we found while testing. I'm thinking I should look into making polykey more tolerant of network failures and changes. This can tie into https://github.com/MatrixAI/Polykey/issues/461 as well.

I've already made some fixes to js-quic to handle network problems but it's far from comprehensive. Polykey itself needs to handle the QUICServer going down for network reasons. It should even support the network being unavailable for extended periods.

This ties in nicely with https://github.com/MatrixAI/Polykey/issues/461 since we'd need to support not having networking running for extended periods of time, but also support dynamically starting or stopping the QUICServer whenever we want. Either in respose to the user requesting it or having to restart it when the network changes or comes back up. Also, being able to toggle the QUICServer at will can be seen as a kind of stealth mode.

I just did some testing with Polykey-CLI. It seems things are tolerant of this already so the changes I made before seem more than fine for now. I tested this by starting a node and just turning off wifi. I did the expected and timeout after the timeout time. Re-enabling the wifi and pinging a seed node caused it to re-connect to the Polykey network just fine as expect. The process did not crash at any point while doing this. Previous crashing due to network issues may have been fixed when I fixed a race condition relating to this just recently.

That said, there may be corner cases where the socket could just fail for one reason or another. The question is how tolerant do we want to be about this? How does this handle switching between wifi networks or interfaces? If we have two active interfaces and one starts to fail then do we seamlessly switch, or are we just tolerant of that? I suppose all theses questions will bear out with more rigorous user testing.

So based on that quick test I think as things are we're fine and tolerant of network failures in that the process will no crash in these cases. I don't think there's anything to address about that right now but we should keep it in the back of our mind with further testing.

[tegefaulkes]

Priorities going forward..

Discovery problems

Right now there are problems with discovery

1. Bug when processing duplicate and missing cryptolinks
2. No feedback when doing discovery, back-grounded or active discovery.
3. Currently periodic re-discovery happening. So discovery always needs to be manually triggered.
4. GG doesn't seem to be updated when processing new information during discovery.
5. We want to seamlessly trigger discovery on any friends/followers on Github
6. No removal of old or invalid nodes from the gestalt.

Reviewing the repos I found the following old issues that relate to these problems.

• https://github.com/MatrixAI/Polykey/issues/691 Newly created by Amy. Outlines periodic rediscovery of vertices and automatic discovery of peers. Points 5, 3.
• https://github.com/MatrixAI/Polykey/issues/462 Outlines a few features, fixes and optimisations that need to be addressed for discovery. Including periodic rediscovery. 3.
• https://github.com/MatrixAI/Polykey/issues/328 Outlines rediscovery mostly and better error handling for discovery. Points 1, 3.
• https://github.com/MatrixAI/Polykey-CLI/issues/163 Bug reporting points 6, 1, 2, 4.
• https://github.com/MatrixAI/Polykey-CLI/issues/162 Addresses point 2.

Looks like all points are addressed across this. Some double up. Should we create a single PR addressing all of these? Seems like a decent re-work/update of discovery. All of these are on Polykey except for point 2 which is both front-end and back-end.

Performance problems.

There are 2 performance issues.

1. Slow startup time of commands. This is likely to be two things. Either Commander doing a bunch of runtime processing for constructing commands. Or PKG doing a bunch of unpacking to run. https://github.com/MatrixAI/Polykey-CLI/issues/102
2. The EFS runs pretty slow which results in very slow vaults performance. https://github.com/MatrixAI/js-encryptedfs/issues/76

Lingering issues

We still need to finish up the following active issues

• https://github.com/MatrixAI/Polykey-CLI/issues/22
• Small problem with recent change in MDNS causing test failures in Polykey.

Moving forward

Moving forward we want to focus on any UI/UX issues and bugs we encounter during user testing.

[tegefaulkes]

I'm going to make a new epic for tracking discovery problems.

[CMCDragonkai]

@pablo.padillo this is the main important engineering issue atm.

Status update:

1. Remaining subissues here are not part of CLI beta 30, they can be moved out to be worked on in the backlog.
2. The main bug discovered during your testing with externals and internals in relation to having double claims should be fixed. Pending that this is all part of a released PK CLI executable - if it is not, it requires a 0.3.1 release.
3. If so, then you must re-attempt another demo run - but this time without bothering with removing your old claims. It's fine to have multiple claims to the same identity from multiple nodes.

  I1
 /  \
N1   N2

Even if N1 is destroyed, from the perspective of I1, and N2, the entire gestalt is all 3 vertexes.

Therefore if N3 and I2 exists.

I2
|
N3

And I1...I2 were connected in some way.

Then N3 should be able to discover I2 and I1, and also discover N1 and N2.

Thus N3 should be able to connect to N2 - either via manual discovery or automatic discovery.

The work that @brian.botha and @amydevs is on right now, are all UX oriented features that help automate or reduce the amount of work for the entire demo process.

This includes:

1. Being able for N3 to share with ANY vertex of I1, so it can share to I1 or N2 or N1, and it achieves the same thing. So that N1 and N2 can both pull/clone the vault. This reduces the demo process steps. - ENG-132
2. Background discovery should be automatic between I2 gestalt and I1 gestaslt. So that it should not be needed for N3 to manual-discover of N2/I1. Background discovery should be interpretable and predictable to the first-order - the immediate neighbourhood. - ISSUE?
3. Being able to have notifications be completely delay tolerant. - ENG-37 https://github.com/MatrixAI/Polykey/pull/695

These 3 things may come after version 0.3.1.

[CMCDragonkai]

So upon release of 0.3.1, remove those other remaining subissues, as I consider the beta release to be done. @pablo.padillo should be redoing the demo with the understanding the more than 1 claim is fine to work with. Subsequent release of PK should be addressing the 3 items posted above.

[tegefaulkes]

@pablo.padillo this is the main important engineering issue atm.

Status update:

1. Remaining subissues here are not part of CLI beta 30, they can be moved out to be worked on in the backlog.
2. The main bug discovered during your testing with externals and internals in relation to having double claims should be fixed. Pending that this is all part of a released PK CLI executable - if it is not, it requires a 0.3.1 release.
3. If so, then you must re-attempt another demo run - but this time without bothering with removing your old claims. It's fine to have multiple claims to the same identity from multiple nodes.

  I1
 /  \
N1   N2

Even if N1 is destroyed, from the perspective of I1, and N2, the entire gestalt is all 3 vertexes.

Therefore if N3 and I2 exists.

I2
|
N3

And I1...I2 were connected in some way.

Then N3 should be able to discover I2 and I1, and also discover N1 and N2.

Thus N3 should be able to connect to N2 - either via manual discovery or automatic discovery.

The work that @brian.botha and @amydevs is on right now, are all UX oriented features that help automate or reduce the amount of work for the entire demo process.

This includes:

1. Being able for N3 to share with ANY vertex of I1, so it can share to I1 or N2 or N1, and it achieves the same thing. So that N1 and N2 can both pull/clone the vault. This reduces the demo process steps. - ENG-132
2. Background discovery should be automatic between I2 gestalt and I1 gestaslt. So that it should not be needed for N3 to manual-discover of N2/I1. Background discovery should be interpretable and predictable to the first-order - the immediate neighbourhood. - ISSUE?
3. Being able to have notifications be completely delay tolerant. - ENG-37 Backgrounding of Notifications Domain Polykey#695

These 3 things may come after version 0.3.1.

https://linear.app/matrix-ai/issue/ENG-298/follow-permission-and-social-links-during-discovery has been created to track point 2. above.

[CMCDragonkai]

@CryptoTotalWar I think upon closing this issue, we should be planning for a hard launch. You should create a new issue for that and associate subissues for a hard launch. I'm not sure if it should go into Polykey - but we could associate issues to the MatrixAI-Graph for any general work.

[CryptoTotalWar]

NOTES Rough Draft WIP. Will clean up.

Critical Issues

1. https://github.com/MatrixAI/Polykey/issues/592 Pretty major problem, results in nodes randomly crashing. "General Fixes for connection stability" - important from a network stability standpoint. We don't want the program or network to fail on you.

2. https://github.com/MatrixAI/Polykey/issues/597 - An oddity and doesn't really break things, but should be looked into. "was producing multiple node certificates when it should have only been producing one" - Minotr bug dont include

3. https://github.com/MatrixAI/js-rpc/pull/47 https://github.com/MatrixAI/Polykey/issues/588 https://github.com/MatrixAI/Polykey/pull/589 - Critical from a usability standpoint. Otherwise not strictly broken. "increasing the Authentication Timeout window for PK Identities Authenticate - important from a usability standpoint

4. https://github.com/MatrixAI/Polykey-CLI/issues/44 - Critical from a usability stand point. "General PK CLI polish and fixes" - pretty major (I will say this is a feature enhancement from a usability standpoint).

5. https://github.com/MatrixAI/js-timer/issues/15 - macro task resource leak. "just a fix where the timers are not being cleaned up properly... a resource leak where they weree lingering in memory when they should have been done cleaned up" - major - resource leaks always a problem and the memory would pile up and crash the node so an improtant fix.

Needed but functional without

• major fix where we fixed NAT hole punching in multiple directions whereas in certain network situations there could be single direction hole punching whereas it should have been symmetric in certain situation. - Important

https://github.com/MatrixAI/Polykey/issues/605 - Not super big issue, Right now it means more stricter nats can't be punched to. I have an idea for a fix. Not strictly a major error, just something we did not want to be broken. Important for reliability of usage in different network situations.

• There was a fix where we weren't taking into account an OLD node ID when authenticating a connection. So when a node renewed its ID (renewed its key) you should still be able to authenticate a connection with an old node but we werent handling that properly. https://github.com/MatrixAI/Polykey/issues/593 --- VERY MINOR

• "small fixes to the timeout middleware" https://github.com/MatrixAI/js-rpc/issues/19 https://github.com/MatrixAI/js-rpc/pull/42 - NOt important

A lot of this was just fixing up bugs.

"Major Feature enhancement

• we overhauled how we discovered nodes in the network and made connection between each other with nat hole punching in a more decentralized way whereas before it was more centralized where all hole punching went through seed nodes but in this case any node could be used to whole punch and discover other nodes https://github.com/MatrixAI/Polykey/pull/618

• we did add the Polykey ENV command which allows injecting secrets into the environment when running a command," which simulate env files in a more secure encrypted manner. https://github.com/MatrixAI/Polykey-CLI/issues/31

• Amy did a lot of work related to standardizing the output formats for the CLI. https://github.com/MatrixAI/Polykey-CLI/issues/22

  Features WIP

• Audit domain - https://github.com/MatrixAI/Polykey-CLI/issues/177

• Feedback to discovery to inspect the discovery q but also discovery events as they happen when triggering discount https://github.com/MatrixAI/Polykey-CLI/issues/162 - discovery feedback

[tegefaulkes]

0.3.1 has been release a few days ago. I'm going to resolve this issue now.

[CryptoTotalWar]

Engineering Updates Summary for Blog Post

Critical Fixes

• Connection Stability: Implemented general fixes to enhance network stability, addressing critical issues where nodes could crash unexpectedly. This fix is crucial for ensuring reliability. Issue #592 - Important

• Authentication Enhancements: Increased the Authentication Timeout window for PK Identities Authenticate to improve user experience during the login process. PR MatrixAI/Polykey-Docs#47, Issue #588, PR #589 - Important

• CLI Usability: Enhancements and polish applied to the Polykey CLI to improve usability. Issue MatrixAI/Polykey-Docs#44 - Important

• Resource Leak Fix: Addressed a significant issue where timers were not being properly cleaned, leading to potential memory buildup and crashes. Issue MatrixAI/Polykey-Docs#15 - Important

Feature Enhancements

• NAT Hole Punching: Refined our NAT hole punching mechanisms to support more symmetric network configurations, crucial for the reliability of node communications. Issue #605 - Important

• Node Discovery Overhaul: Decentralized the node discovery process to enhance network connectivity and reduce reliance on seed nodes. PR #618 - Major Feature Enhancement

• Environment Variable Command: Added the Polykey ENV command for secure injection of secrets into the environment, simulating a more secure method of handling environment variables. Issue MatrixAI/Polykey-Docs#31 - Major Feature Enhancement

• CLI Output Standardization: Standardized the output formats of the CLI, improving interface consistency across commands. Issue MatrixAI/Polykey-Docs#22 - Important

Features in Progress (WIP)

• Audit Domain: Developing an audit domain for better monitoring and tracking of network activities. Issue #177 - WIP

• Discovery Feedback: Enhancing feedback mechanisms for node discovery to allow real-time insights into the discovery process. Issue #162 - WIP

Not Included

• Minor bugs and enhancements not critical to the user experience were deemed not important for the blog post, ensuring focus on impactful updates.

This summary will guide the blog post update regarding the CLI enhancements and fixes since the beta release. Please review and confirm the inclusion of these points. - @tegefaulkes

[CMCDragonkai]

I think that should go to your blog issue? This issue is closed. @CryptoTotalWar