Automatic Syncing of Standups in `samepage` graphs

[dvargas92495]

• we run a Smartblock each day on the DNP of the samepage graph that outputs each contributor's section
• Each contributor's section is EITHER a
  • "cross-app embed" into their respective tool
  • "cross-app reference" into their respective tool

The goal is to allow contributors to work in their preferred tool, but also have a central place to view what others on the team are working on today.

[mdroidian]

Trying to test a cross graph query. Getting error:

instrument.ts:132 POST request to http://localhost:3003/extensions/roam/query failed: Only URLs with a scheme in: file, data are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'c:'

Looks like it could be a windows path.resolve() issue (source) in package/scripts/internal/appPath.ts

Will look into this tomorrow.

[mdroidian]

I don't believe it's path.resolve() anymore. But I have no idea what it is. My URL looks correct. I'm not sure why Received protocol 'c:' comes up, I don't see it while walking through the breakpoints.

And the backend samepage give me

api Received Request post /extensions/roam/query +23s
api Executed post /extensions/roam/query in 8ms +13ms

😵

[dvargas92495]

Take a look at https://github.com/samepage-network/samepage.network/blob/main/api/extensions/$id/$file/post.ts#L3-L14

This is the api handler that gets called locally that then routes the request to the appropriate [app]-samepage repo. I don't see a path.resolve there, so there might be something going wrong here

[mdroidian]

normalizing the URL and adding the prefix file:/// seemed to fix this issue for me. I got

message-notebook Found connection 18...1dc for 4b083...d3bf3 +14s
message-notebook 182...3c6f1ed2e messaged 4b08...dd3bf3 as online true with REQUEST +4ms

But pulling the most recent changes gives me a new error:

POST request to http://localhost:3003/page failed: Error: No targets specified

[dvargas92495]

That's a user error we throw when the user calls for a notebook request with no target defined - this is going to require some debugging to investigate why not

[mdroidian]

https://github.com/samepage-network/samepage.network/commit/2e237ace990c66c36b8ce66a606d5f8b51a71414 It looks like these changes stopped target from being passed to the args I haven't dug too deep, but I'm wondering if it is the workspace: notebooks.label, change as the label column in my notebooks table is blank. I'll test this next session.

[dvargas92495]

Ohhh - yea run npx ts-node scripts/apply.ts --sql. This will run a migration to copy the workspace values to the label values.

Some backstory: most cloud apps have a workspace id that is globally unique and a workspace name that isn't necessarily. we want to start tracking both, and soon make the app/workspace combo globally unique in our notebooks table, while then using label as a display name for users.

[mdroidian]

Hmm. I don't see a scripts/apply.ts, but I do see a scripts/commands/apply.ts so I tried npx ts-node scripts/apply.ts --sql but my labels are still blank. 🤔

Also, when I it seems that I am getting through (changing notebooks.label to notebooks.workspace for example), all my queries are returning No Results. I tested with live samepage as well. I'm wondering what piece I might be missing here.

[dvargas92495]

sorry typo - npx ts-node scripts/cli.ts apply --sql

All of our commands are run by npx ts-node scripts/cli.ts [command] [args]

[dvargas92495]

Live SamePage is not expected to have cross requests to be working yet as we are still trying to nail the API locally.

Figuring out why the queries are returning No Results (after workspace/label is sorted) is considered in scope for this task. One thing that would be super helpful is voicing things that are making it specifically hard to debug, and what feature ideas could alleviate those pains

[mdroidian]

Just an update of where I'm at. Looks like No Results is from path being set to public/data/requests/ and /requests doesn't exist on my system.

https://github.com/samepage-network/samepage.network/blob/6082e1a9389ac49932d0f1fd4768f7a5db1a526c/app/data/downloadFile.server.ts#L31-L32

Tomorrow I'll take a look at where that is getting set and why it doesn't seem to exist.

[dvargas92495]

we use the file system (and in production, AWS S3) to cache the answer to requests, in case the other target notebook is offline.

so the question to investigate here is: why is the target notebook not sending the response down and saving it?

[mdroidian]

Well, it works now. 😀 I'm not 100% sure but I might have been getting No Results because I updated samepage-network without realizing it had /package updates and I didn't updated roam-samepage 😐 Fortunately, where I spent a lot of my time will help with understanding these next tasks.

A few interesting things now: 1) first query returns No Results but subsequent queries return results 2) Smartblock <%QUERYBUILDER%> returns blank blocks 3) I went to add a test query to the samepage graph (which I have in the app not browser), added roam-samepage, then got a post/disconnect loop

I'll start with 1) and look into where/when method gets set

[dvargas92495]

Yup - 1 is a known challenge to solve.

Right now, the way the request protocol works is that app A sends the request, and only returns a response from cache. It doesn't ever request from the target app directly - instead it sends them the message basically saying "hey, I was looking for this data, send it back to me whenever you can".

We'll want to have a case for handling when "whenever you can" is immediate, which is actually the 95+% use case with most of these cloud-based applications.

Even for non-cloud based ones (LogSeq, Obsidian), we may end up supporting a cloud based entry point for them down the line

[mdroidian]

I went to test the new round of changes (Start to stabilize cross app querying (#17) · samepage-network/roam-samepage@cea14c0) and I am getting some errors and no results are returned. samepage.network, roam-samepage, and roamjs-query-builder are all up to date.

The target notebook is getting the proper query and returning results in handleRequestOperation but the target notebook throws two errors:

POST http://localhost:3003/page 400 (Bad Request)

t   @   instrument.ts:163
handleFetch @   apiClient.ts:35
(anonymous) @   apiClient.ts:101
apiClient3  @   apiClient.ts:131
handleRequestOperation  @   handleRequestOperation.ts:24
await in handleRequestOperation (async)     
handler @   crossNotebookRequests.ts:103
(anonymous) @   setupMessageHandlers.ts:66
handleMessage   @   setupMessageHandlers.ts:64
receiveChunkedMessage   @   setupMessageHandlers.ts:99
registry_1.samePageBackend.channel.onmessage    @   setupWsFeatures.ts:160

Uncaught (in promise) Error: POST request to http://localhost:3003/page failed (400): Failed to parse request. Errors:

Path `` had the following union errors:
  - Invalid discriminator value. Expected 'create-notebook' | 'add-notebook' | 'list-user-notebooks' | 'authenticate-user' | 'ping' (invalid_union_discriminator)
  - Expected `request` to be of type `object` but received type `undefined`
    at apiClient.ts:41:13
    at async handleRequestOperation (handleRequestOperation.ts:24:5)
(anonymous) @ apiClient.ts:41

Any ideas?

[dvargas92495]

what's the shape of the request that the target notebook sends?

[mdroidian]

This?

{
    "conditions": [
        {
            "uid": "-Ae1l2WRL",
            "relation": "with text",
            "source": "node",
            "target": "DemoCrossQueryData",
            "not": false,
            "type": "clause"
        }
    ],
    "returnNode": "node",
    "selections": [],
    "schema": "node-query"
}

[dvargas92495]

I meant the full request sent to api.samepage.network - should be something like {method: "notebook-response", notebookUuid: "...", ...}

[mdroidian]

{
  "notebookUuid": "4b083c01-a75c-48aa-a516-d7c580dd3bf3",
  "token": "{redacted}",
  "method": "notebook-response",
  "response": {
    "results": [
      { "text": "DemoCrossQueryData", "uid": "SY6NmX2iv" },
      { "text": "DemoCrossQueryData1", "uid": "sDX6jkAEL" },
      { "text": "DemoCrossQueryData", "uid": "0sSqBw25w" }
    ]
  },
  "target": "18222c4a-b00d-47f3-866f-5713c6f1ed2e",
  "requestUuid": "eee98bea-ecc0-46f1-9df0-a85ba190548a",
  "messageUuid": "37b2a8b4-2cfe-4ba5-a51e-e46763d292c1"
}

[dvargas92495]

hm and you've pulled latest samepage? it's failing a parse error. This schema used to require a request parameter but no longer does

[mdroidian]

My apologies, I didn't realize I was still on a forked version of this repo. I made changes to my workflow that should stop this error from happening again.