Adding HTTP server to desktop version

[msageryd]

I'm struggling to get the callbacks to work.

1. In order to make a request to a non http endpoint I'm using the open npm package
2. I'm using express.js to setup handlers for x-success and x-error callbacks

The data is received correclty, but the open library uses the default browser, so Chrome is opened as soon as I send a status 200 back after getting the callback.

I'm probably doing something stupid or overcomplicating stuff.

Also, I cannot make the request at all without providing the callback urls. Is there no way to just get the data in the reponse body of the initial request?

[czottmann]

Nice to see a fellow dev looking at ActionsURI! 🤙🏼 I'm not exactly sure what you're trying to do there. The receiver is Obsidian, which is hardcoded (because Obsidian registers the obsidian:// protocol), and ActionsURI is getting its incoming data from Obsidian.

I cannot make the request at all without providing the callback urls. Is there no way to just get the data in the reponse body of the initial request?

There is no way, no. These aren't HTTP request, this is XCallbackURL all the way down. Request+response are async by default. The receiver is a black hole, basically, there's no concept of "response body". ¯\_(ツ)_/¯

[msageryd]

Aha, I see. I don't think my "open" call is the culprit when it comes to the browser opening up. Actions-uri calls my callback uris with window.open(), which I think opens the default browser. I don't know how to mitigate this.

[msageryd]

I have tried the following, and nothing works:

1. using file:// as uri scheme for the callbacks and trying to catch the response in a file Didn't work. I don't think Obsidian has permission to write to files.

2. registering a custom uri-scheme, obsidian-callback:// Same problem as no 1

3. Listening for the callback on tcp (instead of http). Doesn't work

How are you able to listen to these callbacks in MacOS actions? Do you have any example code?

To clarify: I can get hold of your response via the callback, but the default browser is also opened, which is a no-go for me.

[msageryd]

Update.. The XCallback concept seems to be quite hard to work with.

I found a library for making it easier on my side. The library works perfect when I communicate with Bear, but I cannot get any callbacks from actions-uri. This makes me wonder if window.open() really is the correct way to initiate the callbacks.

The only way I can get information back is to use a regular http uri for x-success. But As I stated earlier, this concept will always open the default browser as well. I haven't fond any way to mitigate this.

The functions you expose in the actions-uri is perfect for what I need to build, but unfortunately I cannot use them =(

The xcall library: https://www.npmjs.com/package/xcall

[msageryd]

I think the easiest way would be to have a regular http server like Omnisearch. If I find time I might try to implement this in actions-uri. Unfortunately, I don't have time right now.

https://github.com/scambier/obsidian-omnisearch/blob/master/src/tools/api-server.ts

Edit: I just noticed that Omnisearch is GPL3, which does not play well with your MIT license. The linked code cannot be used as it is, but it can serve as inspiration.

[czottmann]

The XCallback concept seems to be quite hard to work with.

It's what's mandated by Obsidian. And since (to me) Actions URI is basically "just" the backend for Actions For Obsidian, and working well, I'm fine with that. ;)

This makes me wonder if window.open() really is the correct way to initiate the callbacks.

It is. There's no other way to return x-callback-url (XCU) responses from Obsidian. The Obsidian client environment is not a node env – it may be on desktop but it isn't on mobile.

How are you able to listen to these callbacks in MacOS actions? Do you have any example code?

I use an old app, Lincastor v2.4 (note: not Lincastor Browser). It allows you to register custom protocols with macOS, receives calls to those URLs, and hands them to your scripts. For example, I have x-callback-test:// set up on my dev machine which just logs all incoming calls to a file:

Lincastor's original website is offline by now but still available using the Web Archive: LinCastor App. The download on that page still works, even though it might take a while. I've been using it for years, it's good, it runs on Sonoma w/o issues.

There's also Lord-Kamina/SwiftDefaultApps: Replacement for RCDefaultApps, written in Swift. which lets you see which apps have registered what URL schemes on your local machine. There might be a text editor or other tool that could be used as an XCU receiver.

[msageryd]

Thanks a bunch! I tried to register a custom scheme, but I didn't "hear" anything getting called back. I falsley assumed that Obsidian (Electron) was not permitted to use custom schemes, so I gave up on that. It seems likely that I made something wrong, I will revisit this concept again.

[czottmann]

IIRC, the first time Obsidian tries to send a request to another app, you'll be asked for permission. That dialog will appear once only.

And Obsidian is not deciding what to do or where to send the outgoing URL, it merely hands it over to macOS/iOS which then passes it on. Obsidian itself has no concept of "another app" here.

[msageryd]

LinCastor worked great, thank you. I have also realised that registering a custom uri scheme is not straight forward, at least not on Mac. It kind of works, but includes some manual handling (adding to plist), etc. This would not let me create a simple reusable Alfred workflow.

Instead I went another route with seems to work very well. I now have a working plugin which "wraps" actions-uri in an http server. This lets me send all actions-uri requests to my plugin via http on a specified port. My plugin awaits the x-callbacks internally and forwards to data to the original request in the response body. The key to this solution is that it's much simpler to register the uri scheme from within Electron, which already have functions for this. I think they will also work on Windows, but I haven't tried yet.

I don't have time to finish this now, but my POC works great.

So, if I would publish this as an Obsidian plugin it needs a name. Would you rather it being a completely separated plugin, or would it be ok to name the plugin "obsidian-actions-uri-server"?

[czottmann]

So, if I would publish this as an Obsidian plugin it needs a name. Would you rather it being a completely separated plugin, or would it be ok to name the plugin "obsidian-actions-uri-server"?

Would your plugin require the user to install Actions URI separately, or what do you have in mind? Because bundling Actions URI as part of your plugin would cause major routing issues in Obsidian.

[msageryd]

Yes, I think it's cleaner to keep them completely separate, in the same way you don't bundle Omnisearch. My plugin alone will not interfere with anything.

• The endpoint for reaching my plygin will be http://localhost:customPort
• Everything added to this uri will be forwarded to obsidian://actions-uri
• I will await callback on a registered Obsidian protocol handler, currently named actions-uri-server
• Everything I get back in the callback, including any errorCode and errorMessage is sent back in the reponse body to the original request.

Example:

http://localhost:3333/tags/list?vault=my-vault

[msageryd]

Another option would be to make my plugin completely agnostic so it can convert any x-callback based plugin into an http endpoint. Either via a plugin setting (protocol = "actions-uri") or maybe by letting the user add the protocol in the original call, http://localhost:3333/actions-uri/tags/list?vault=my-vault.

In this case I should probably name the plugin something more generic.

[czottmann]

Another option would be to make my plugin completely agnostic so it can convert any x-callback based plugin into an http endpoint.

I like that idea a lot, actually.

maybe by letting the user add the protocol in the original call

The protocol here is obsidian:, the XCU host is actions-uri. But if I read you right, you want to take the original XCU host, path and parameters, and pass the whole set to your server http://localhost:3333/, correct? If so, cool idea.

[msageryd]

Settled then. I'll make a completely separate plugin. I suppose there are some more plugins utilising the x-callback concept since this seems to be standard in Obsidian.

I'll try to sort out my use of terminology. I knew that obsidian: is the protocol. Got a bit carried away with the terminology since I used registerObsidianProtocolHandler to register actions-uri-server.

I'll finish the plugin as soon as I get the time and ping back to you. Will give it another name.

And to clarify, here is an example:

Client makes this request: http://localhost:3333/actions-uri/tags/list?vault=my-vault

My plugin forwards to: obsidian://actions-uri/tags/list?vault=my-vault

My plugin awaits callback to a registered Obsidian protocol handler. All data from this callback will be sent back to the original request in the responde body.

[msageryd]

Almost there, but I have a stubborn problem. How ever I try, I cannot make calls to obsidian:// without the Obsidian app is put to front. Do you know of any settings that could mitigate this?

I'm not sure if this is even solvable. When you call window.open(), the opened app is supposed to be brought to front for security reasons, if I'm corectly informed. By registering a protocol handler in Obsidian I mitigated the problem I had in Node, where the default browser opened. At least I have control over the process now and can forward the data in a response body, bit it seems impossible to not put Obsidian to the front.

[czottmann]

Almost there, but I have a stubborn problem. How ever I try, I cannot make calls to obsidian:// without the Obsidian app is put to front. Do you know of any settings that could mitigate this?

There aren't any, I'm afraid. The OS decides that the handler app needs to be in front. This is the reason why my app, Actions For Obsidian, is bringing Obsidian to the front every time it sends out a request, and why Obsidian forces AFO to the front on the response. It's a security feature in Apple's operating systems, I don't know how other OS'es handle it.

AFAIK there's only one way to have two apps communicating via XCU in the background on iOS/macOS: both apps must explicitly permit incoming background calls by the other app. And it has to be mutual. I asked Obsidian (the company) whether they would be open to do that for/with AFO, but they weren't interested.

If you want to keep bouncing ideas, let me know the GH repo of your plugin, and we'll keep discussing it there!

[msageryd]

@czottmann The enforced "open-bahaviour" is a show-stopper for me. The goal for me is to use Alfred with Obsidian. The Alfred command dialogue is automatically closed as soon as it looses focus, i.e. when Obsidian gets focused I loose my Alfred dialogue.

I have now ventured into another path. I simply added an http server directly in your plugin. As of now it is a PoC, i.e. a hack, but it works great.

I now have two options:

1. clean up the code and make some design decisions with you to enable http in your original plugin
2. maintain an http-only fork of actions-uri

• You said earlier that an http server was not in your roadmap. Is this true even if you'd get the http server in a PR? My http addon reuses all of your route structure and route handlers, only some minor changes are needed.

My current solution and clarification about the "design decisions":

• An http server is spun up in onload()
• this.registerRoutes builds an object, this.registeredRoutes, keyed with fullpath. A handler is added to each entry
• this.handleRequest parses the query params and calls the handler for the route, the handler awaits the result from handleIncomingCall which is returned to handleRequest and then returned as a response body to the original request.

Todo:

• x-error and x-success should not be mandatory in Zod (maybe based on plugin setting)
• http server should be optional based on plugin setting
• if focusOrOpenFile gets triggered we actually want to focus Obsidian

The code is here: (N.B, it's a hack as of now =)
https://github.com/msageryd/obsidian-actions-uri

Here is how an example call looks in Postman:

[czottmann]

Nice work!

The enforced "open-bahaviour" is a show-stopper for me. The goal for me is to use Alfred with Obsidian. The Alfred command dialogue is automatically closed as soon as it looses focus, i.e. when Obsidian gets focused I loose my Alfred dialogue.

Understandable!

You said earlier that an http server was not in your roadmap. Is this true even if you'd get the http server in a PR? My http addon reuses all of your route structure and route handlers, only some minor changes are needed.

Hmm. On the surface, that HTTP server adds only minor complexity, true, but as they say, the "devil's in the detail", right? And I got to be honest, at the moment I have enough on my plate as it is.

But please understand that I'm not against your proposal. I just don't want to deal with it right now. So here's my counter offer ;) – use your fork for a month or so, jot down and iron out the weird edge cases and minor issues that will surely arise. We'll talk again in September, and with everything you've learned by then, we should be able to come up with a solid implementation that can be mainlined.

What do you think?

[msageryd]

That's a great plan. I'll try to build the http support as separated as possible. This will make my fork easy to upgrade with your future upgrades.

[czottmann]

Ace! 🤜🏼🤛🏻

If possible, please replicate the way Omnisearch conditionally imports the server-related code, as I don't want to maintain two versions, one for mobile and one for desktop. 😅

Ping me if you need something or when you want to bounce ideas etc.!

[msageryd]

I have some thoughts.

My needs:

1. ability to make a request without x-success or x-error params and no attempts to make callback
2. files should open if needed

I seems like you had some intention to solve (1) with the silent param, but I can't see that this check is implemented. https://github.com/czottmann/obsidian-actions-uri/blob/75dd46a91d99d53dedc49cd6bd0b6a1b13fbc65f/src/main.ts#L216

Even if it was implemented it wouldn't quite suit my needs. I need two separate parameters:

• silent
• nocallback

The silent param is fine as it is (since it is not implemented for the callback handling) The nocallback param would prohibit the callbacks and also accept params without x-success and x-success.

Easiest fix for (2) might be to simply omit the callback if callback functions are missing in params. What do you think?

[msageryd]

I just noticed that callback is only performed if callback functions are provided. My current fix to (2) is to simply remove x-success and x-error from all route schemas, i.e. they are now optional as per the default schema.

[czottmann]

I would be remiss not to mention it before we progress any further: have you seen https://github.com/coddingtonbear/obsidian-local-rest-api yet? It might suit your needs…

silent is used all over the place, it's for suppressing the opening of the requested note. The XCU call itself will open Obsidian, but with silent the note won't be focussed.

I just noticed that callback is only performed if callback functions are provided. My current fix to (2) is to simply remove x-success and x-error from all route schemas, i.e. they are now optional as per the default schema.

Yes. When registering the routes for the server, I'd strip out the x-* parameters of the incoming call. I'm not currently looking at code, so that's just my first impulse here. 😅

Additionally: The vault parameter isn't strictly needed, either. If the server is answering, a vault is already open. Whether it's the correct one, well. The XCU call parameter vault forces Obsidian to open the right vault before handing the call over to the plugin; with an HTTP call, we don't have that luxury. We could check whether the vault is the right one, and return an error if it isn't, but we can't switch vaults on demand.

[msageryd]

Thanks for the heads-up on the rest api. I read through all of the Obsidian plugins, I think actions-uri is, by far, the best API. I might want to change or add some routes for moving forward, but it almost perfectly suits my needs.

As for the callback params. I'm trying to use as much of your infrastructure as possible without interfering. This includes your schema checking. For some routes you are extending the schema to make the callback params mandatory, which prohibits me from making requests to those routes without callback functions.

In the default schema, callback functions are optional. By removing the callback functions from all schema extensions they are now optional everywhere (in my fork).

Example, old tags.ts:

const listParams = incomingBaseParams.extend({
  "x-error": z.string().url(),
  "x-success": z.string().url(),
});

My version

const listParams = incomingBaseParams.extend({});

[msageryd]

I have now implemented a less intrusive http server, but the diff against your repo is quite big. A lot of diffs are due to differences in our prettier settings. If you could add your prettier settings to the project I would get rid of a bunch of diffs.

This is new in my current version:

• removed all x-success and x-error from the local schemata for each route.
• created separate HttpServer class which is only imported on desktop (should also check settings when implemented)
• created separate registerRoutes() for http routes. This is embedded in the HttpServer class

Todo:

• Create settings for enableHttpServer and httpPort
• Ensure that the original error handling works without x-error, i.e. test with bad requests

[czottmann]

A lot of diffs are due to differences in our prettier settings. If you could add your prettier settings to the project I would get rid of a bunch of diffs.

Added to main branch. (Didn't use a config file for years, prettier was configured in VS Code.) I'm an idiot! Since I write a lot of Deno code these days, I'm also deno fmt to format all my TS code these days. I use the very sensible defaults.

As for the rest: Sounds good, keep going, but as I've said, I don't have the capacity to look at the diffs at the moment 😅

[msageryd]

I installed Deno and the deno extension in vscode. After configuring the workspace to use the deno formatter, almost all formatting diffs are gone. But there are some left. I wonder if you have some custom settings for your Deno formatter?

Here is an example diff (green is formatted with my formatter):

Here is another diff:

[czottmann]

Wild. No other config, no. Running deno fmt main.ts (Deno 1.44 as well as 1.45.5) from anywhere yields this formatting for me. Same results with explicit --no-config flag set.