Package SilverBullet as a native mobile (and desktop) application

[pm64]

I know this is on the radar, so let's make it official! I'm not sure if any physical groundwork has been laid, though I believe @zefhemel made a reference to React Native in one of the videos. Also, in terms of supported platforms, is it better to track this effort as a single issue, or should we create separate issues for Android and iOS?

[zefhemel]

Thanks for raising this. This is another good topic to discuss strategy on. Here are my current thoughts, I’m open to your perspectives of course.

First, building a mobile app (any mobile) app is effort. It takes effort that would be taken from other efforts (opportunity cost). This is important, because for me at least this is a side hobby project and I have limited time, likely this will be the case for others as well.

That said, what’s the ideal case in my mind? The perfect solution would be 100% native apps for each mobile system, specifically one for iOS and one for Android. Everything would be implemented custom, still hosting plug code in a JavaScript sandbox so that those all work. The mobile app would have offline sync capability, somehow pulling your entire space into the app, sync it with some central place when online and somehow also allowing real-time collaboration.

There’s a few aspects of this ideal scenario that are challenging:

1. Two native apps for two operating systems to maintain (iOS and Android)
2. Practical question: I think technically Apple doesn’t allow serious extension of native apps via Javascript, although we may slide under the radar, it’s risky
3. We need to build a sync engine to sync files between your mobile device and some centralized location
4. And the big one: the code editor we use on the web (CodeMirror) is a HUGE, likely multi-year project to replicate in native code (let alone doing this twice). We could get away with something much simpler, but the experience for sure would be worse.

So this doesn’t seem very realistic.

One step down would be addressing item 4: the code editor, by making that part of the UI (which is almost all of it, to be honest) just be CodeMirror in a web view. This will work (it does right now when you open SB in a mobile web browser), but it defeats a little bit the value of being a “truly” native app when 90% of the UI is now a web view. There are a few glitches on iOS I’ve noticed, and they’d be hard to address properly. Problems 1-3 still hold.

Next step “down” would be to use React Native, this largely solves item 1: maintaining two separate native apps. This still leaves item 2-3.

The question at this stage is: do we really benefit from anything react native offers, or could we use something like Ionic framework? Either way, this still leaves 2-3.

What we could do to solve the sync engine issue is simply have the mobile app talk directly to a Silver Bullet backend (like the browser frontend does right now). This would work great, but would break if you don’t have access to the Internet. So the question is: how key is being offline capable?

If being offline capable is the “killer feature” of a mobile app there’s still two routes:

1. Go with something like RN, Ionic and build a simple sync engine
2. Stick with the current PWA route and build a sync engine that syncs with e.g. IndexedDB. This is something I prototyped at some point but didn’t pursue because of opportunity cost (adds complexity and I had plenty of other things to focus on). It is possible to build a 100% offline capable web app this way.

So this is where I stand for now:

For now, Silver Bullet works on mobile. I use it every day. I have a SB VM running in my local network, I connect to it remotely via Tailscale and as long as I have working Internet (90% of the time) the experience is pretty good. All functionality works.

We can build simple “native” wrappers around this experience, to have a proper App store presence: we can build an Electron app for desktop and an Ionic (or similar) wrapper for iOS and Android. This wouldn’t give functionality benefits, just some convenience over the current PWA model.

If offline capability turns out to be important for people, I’d aim to build a sync engine at the “web” level: that is, allow syncing of a space locally to the client (browser) via IndexedDB. This would benefit both desktop and mobile at the same time.

Thoughts?

[pm64]

While I understand the appeal of a 100% native implementation, I believe this case to be an exception, since the editing experience should be as identical as possible across platforms, and the most practical way of ensuring that is to use CodeMirror everywhere. For many users, SB's state-of-the-art editing experience will be its most important feature, so it should be presented consistently.

I'm intrigued by the idea of a sync engine that syncs with IndexedDB. A solid implementation of this, combined with a willingness to use CodeMirror on all platforms, could reduce the need for native mobile apps, at least with respect to offline use and syncing. But PWAs have other limitations that will annoy us eventually, so I think the approach will have to be native apps (using a web view -- not "100% native"), storing documents in the normal manner on the device's file system.

With respect to Apple - we should get clarification on the restrictions, but to my knowledge Obsidian's Javascript-based plug-in ecosystem is thriving on that platform. Still, we need absolute certainty on this.

On the sync engine - I think this is a case where Obsidian's approach is excellent, so we may be wise to draw inspiration from their model. Users should be able to set up their own sync process through cloud storage providers, or host their own sync server (as you suggest), facilitating e2ee syncing across devices. And perhaps an official hosted offering could be made available to users that don't want to host their own sync server, as a revenue stream for this project (similar to Obsidian's "Sync" service).

With regard to the framework, others might need to chime in. My personal, biased preference would be Electron on Linux and MAUI on everything else (Android/iOS/Windows/Mac). But that would mainly serve to minimize my own learning curve, I'm not sure what's most appropriate for the project.

With regard to being "offline capable" -- I can confidently say yes, this is huge. Not just a killer feature, it's mandatory. So in my opinion, route no. 1 ("go with something like RN, Ionic and build a simple sync engine") seems to be our destiny.

[yorrd]

Okay, I think I can help here. Addressing a couple of points from above.

Framework

We use Capacitor / Electron for our apps in the agency. I've never had a customer where that would've been an issue. With a little love, you'll get good enough performance. From biased experience, many users prefer having the same look as opposed to the respective "native" look and feel of the respective operating system. Bundling with capacitor is quite easy, we have set up hundreds of apps this way over the past years.

Apple

Can confirm definitely, we have several apps currently in the app store with Apple that use basically only a wrapped PWA. They do have in debth usability tests which will need to be passed, but that's manageable.

Deployment

There is no inherent issue here, but it takes time to get all the store account information and marketing material together. Also, there will be yearly cost for the accounts.

Sync Engine & Offline

I disagree with @pm64 on Obsidian's approach. I think it's good for power users who can establish their own sync but that will always prevent the regular day-to-day user from actually really using it in my experience with our team of 15.

Offline is not a problem as stated in your comments. We could even just let the whole application work as it does on desktop, spawning a local webserver that serves from file system (cloud sync tbd). Meteor is built this way as well, which we used to use until a couple of years ago. From my perspective, the question should be, do we want the mobile app to be a "client" or a fully functional SB instance with its own permanent persistence. IndexDB is not a good idea to permanently store important data in the mobile app ecosystems since it's treated as cache data afaik, so that would be the client approach which relies on an external sync service. If the project goes down that route, a built-in sync service is essential iyam.

---

Sorry, I'm on the run, no time to review in depth atm. Again, I think it's essential to decide first on whether the mobile app should be another client to an external server or a fully fledged SB instance on which I don't have an opinion on yet.

[zefhemel]

From a choice of technology perspective a few preferences: overall I like to believe "boring is good." That's why I picked TypeScript (has been around for some time), React (although technically now Preact), and initially node.js and npm (although recently to the only a few-years old Deno, because the node.js setup just became too much of a mess). For this reason I'd also prefer proven solutions for a desktop and mobile app.

For desktop this would logically mean Electron. My only concern with this is that we'd likely be distributing and running 2 full JS runtimes with this desktop app. As @yorrd suggested, I'd indeed imagine at least an initial implementation (also to keep things simple) would simply be the server running locally "inside the app" (so we need to ship Deno) and Electron being an extremely light browser view (so are shipping a full Chrome instance) on top. This will consume a fair amount of resources. Computers are big and fast these days, but... it's a bit wasteful. Something like Tauri which utilizes the OS's native browser renderer would be a bit lighter weight, but is also much more experimental.

For the mobile side, I have not kept up what's happening on the "web app wrapper" side, but I've seen the Ionic name around for quite a few years, and I think Capacitor comes from that world, right? I'd also lean to a light weight mobile wrapper to start. In general any amount of client specific (be it PWA, desktop, mobile) code I'd like to keep to a minimum to keep the code base small and testing surface as small as possible.

Indeed the question is, especially for the mobile client: would this just be a client or a full blown environment (also providing the backend). I think here we can be pragmatic and start with purely a client and then work on integrating a backend as well. Here we will likely have to build something custom, since I don't see Deno running in a mobile app anytime soon. However, building something custom may be feasible. I see Capacity has plugins for e.g. native file system access. Some things will simply not work on mobile, like running of shell commands (which plugs can do right now).

The sync topic is another big one. I'm following the Obsidian community from a distance, and I see people complaining about using their devices native sync capability. This may be possible to overcome to build a simple sync engine of our own, or allow to use the native iCloud drive stuff. Again, I have a sync engine parked somewhere and can it bring it back. Sync is a whole can of worms, but perhaps somebody is interested building or integrating a very solid system :)

[pm64]

A few points:

• I don't see tremendous benefit in starting with client-only, then working toward a "backend", since paths to functional backends are fairly short.
• The beauty of Obsidian's approach is the simplicity. Each platform is treated the same, with files stored on the device's file system. This gets you two important things for free:
  • Offline support
  • Decoupling of the storage and sync concerns
• @yorrd, can you be more specific about which part of the above will "always prevent the regular day-to-day user from actually really using it"? Maybe you were referring to something else?
• @yorrd, the concern about Apple isn't related to PWAs, it has to do with their "no app store within an app store" policy with respect to SB's plugs ecosystem. This will require an analysis of Apple's policy, hearsay isn't good enough.
• "Boring is good" +1

One initial observation is that the "Web" side of SB is doing virtually all of the heavy lifting, and the "server" side is easily ported. So it's totally feasible to actually build some prototypes to evaluate. (I nearly have a working MAUI implementation, just need to resolve some apparent service worker limitations in WebView2.)

Given this, one approach would be to identify our requirements and priorities, pump out some prototypes, then measure how well each framework stacks up.

If you were to ask me about requirements/priorities, I'd say the following areas would provide the most value for the most users:

• True native experience (outside of the Web view)
• Performance
• Offline operation
• Accessibility
• Access to device hardware, push notifications, etc.

So our goal would be to deliver in these areas while maximizing code reuse across platforms.

Some quick points on sync:

• @zefhemel yes, Obsidian users regularly identify issues with their own sync solutions (i.e., simply using their own cloud storage provider). I believe the current state of affairs is that there are providers that work flawlessly on specific platforms (for example, iCloud on iOS), but no single provider that works flawlessly across all platforms. But I see this as a momentary annoyance related to current cloud storage products and features, not an inherent flaw or weakness in Obsidian's approach, and it will go away over time. Still, Obsidian provides its official Sync service for users that want flawless operation across platforms, which we should do as well. The difference is that our users will have the option to self-host that service.
• If we go with locally-stored, synchronized copies of spaces on each device, there's just one additional piece we'll need: a selective sync feature, for users with massive spaces and limited storage on one or more devices. I've seen several implementations of this feature in different domains, and believe this is an opportunity to make one that's actually good :)

And some miscellaneous notes:

• With regard to Deno on mobile: this is obviously a no-go on iOS, but I did notice that at some point, a Deno package was added to the Termux ecosystem (Termux is an open-source Android terminal and package manager). However, when I try installing that package today, I get a "package not found" error. I asked the Deno community about this and was met with blank stares :) Regardless, I think we agree this isn't a viable path.
• About shell commands: maybe not in iOS, but plugs can do this all day in Android :) But yes, eventually it will probably be necessary for some plugs to do platform-specific things, so we need the plugs to carry some kind of metadata indicating platform and/or device support.

UPDATE: as I get deeper into the code, I'm beginning to appreciate how interdependent the server and PlugOS are. My MAUI experiment is most certainly a no-go, and I have retracted some of my above analysis.

[yorrd]

@pm64 As on why I don't believe Obsidian's sync is viable for a regular user: I believe having to set up your own sync is a major blocker to adoption. This is not based on hard fact numbers but from my 2 year experience in the Obsidian ecosystem I have followed many threads where people have just left the conversation or directly stated that this made them switch. Therefore I believe the issue to be a big pain point for users that are not familiar with setting up file system sync solutions themselves. This being said, I understand what you mean by it being an technically elegant solution.

I agree mostly with your priorization, but I'd put offline operation higher up the list. Not being able to edit notes on the subway is a big factor for me personally. Just my subjective opinion though.

---

As on whether plugs are viable on the app store: I asked our legal to have a quick informal look (no guarantees) and they reported on which sections they deemed relevant, judging from the guidelines which they had already read through for an earlier customer project:

Listed under acceptable business models (https://developer.apple.com/app-store/review/guidelines/#other-business-model-issues)

• (i) Displaying your own apps for purchase or promotion within your app, provided the app is not merely a catalog of your apps.

Listed under unacceptable:

• (i) Creating an interface for displaying third-party apps, extensions, or plug-ins similar to the App Store or as a general-interest collection.

... therefore not banning behavior such as Obsidian plugins or SB plugs, as far as I can tell without wanting to give legal advice here.

---

@zefhemel based on our discussion so far and as you have stated above, I believe a client-version would be easiest to implement first, treating local file storage as cache only. This introduces the question of diffing different versions which have been edited offline. I have started implementing an Obsidian sync plugin a while back based on y.js (and therefore the CRDT paradigm) and have started thinking about implementing sync using CRDT only, while simultaneously opening the door to shared editing. Haven't got a finished prototype yet though. Do you both think that's a possible way forward? Many in the industry seem to be going down the CRDT route for offline first applications.

[pm64]

Want to keep the momentum going here, even though we're all probably getting pulled in different directions for the holidays.

@yorrd, thanks for your research. It sounds like the reason plugin ecosystems like Obsidian's and SB's are in compliance is that the available extensions are not "similar to the App Store" and do not represent a "general-interest collection". This makes sense.

With respect to the architecture.. as I noted in an update to my comment last week, when I opened this issue I didn't yet have a full appreciation of how intertwined PlugOS and the "server" piece are. When I reached the server code in my last review, I naively expected to find some basic file system I/O and SQLite calls. Of course, it was anything but that. But I can see why this approach was necessary.

Assuming the mobile and desktop versions of SB will use the same strategy for storage and sync, in one sense this is good news for me, because it means framework or "wrapper" for the mobile app is necessarily limited to those that are Javascript-based, which should accelerate our path to an actual plan.

One last note for now. Obsidian has a major flaw: the startup time is unacceptably long, particularly on mobile. Based on the conversations I've seen around this, it seems this can't be fixed. In fact, at least one 3rd-party product has been released that is marketed, at least in part, as a solution to this: a browser extension that bypasses Obsidian to add content directly to the user's vault (see Fleeting Notes).

The part I'm not sure about is whether Obsidian painted itself into this corner by using Electron? This seems to be the perception among some users, but I'm not sure if it's true. If it is, we should learn from this and avoid Electron in favor of a more performant framework.

[yorrd]

One last note for now. Obsidian has a major flaw: the startup time is unacceptably long, particularly on mobile. Based on the conversations I've seen around this, it seems this can't be fixed.

Yeah that's definitely a problem. I'm not sure about Electron being the source. I use Mailspring as my daily driver which is built on Electron. Startup times are good (<1s) with 10+ mail accounts and tens of thousands of locally stored mails incl attachments. My guess is that Obsidian is building an in memory cache on startup and isn't very efficient about doing so, just a guess though. It says "loading vault" or something along these lines, and when I start from CIFS / NFS, it's way slower, so I'm guessing it's limited by the file system.

Juding from the issue mentioned above, @zefhemel seems to be going after a sync solution between mobile and desktop. So maybe a good way forward would be to leverage that and do offline storage as a "hook" in the sync system? Just generally speaking here. If we do that, contents in the app will end up being ephemeral anyways, so we might as well put them in cache storage (indexDB/whatever). I like that thought because bundling for the stores is going to be a lot easier. In fact that's something I could just quickly build a bundle for an test it out.

@zefhemel just wondering, have you registered the store accounts for silver bullet? Are there name collisions you're aware of? Happy to assist and chip in on the cost to make this happen. A well synced mobile app will be a huge benefit to the ecosystem.

[zefhemel]

Right, so here's the state of my progress and thinking now.

Phase 1: Desktop

As you may have noticed, I merged a simple Electron based desktop app. It lives here: https://github.com/silverbulletmd/silverbullet/tree/main/desktop I also integrated it into the build pipeline, so that every time I tag it builds a new version with the latest SB version (and since auto updating is integrated, it will upgrade too — at least on Mac and Linux and I suppose it should on Windows once I properly sign it).

Builds are here, and note advertised anywhere yet: https://github.com/silverbulletmd/silverbullet/releases

I went the road of least resistance, which means. The app bundles:

• A copy of Deno
• The silverbullet.js single JS bundle (the same that you get from https://get.silverbullet.md)
• A very thin Electron wrapper that spawns Deno on a random port and opens a BrowserView pointing to it for each folder
• Some niceties like the ability to open multiple spaces, remember open windows and window placements, ability to open developer tooling etc.

In effect, this means you'll be running 3 copies of the v8 JavaScript VM (one for Deno, one for Electron's NodeJS, and one for the Chrome BrowserView) which is a bit crazy. However, I did manage to implement this in 2 days. So there's that.

There may be a way out of the 3 JS VMs approach, which could emerge of part 2 of the plan which is...

Phase 2: Mobile

So here I'm working on a branch using https://capacitorjs.com. Which is the thin wrap-a-browser-view-in-a-mobile-app part of the Ionic framework. For visibility I've opened a PR: https://github.com/silverbulletmd/silverbullet/pull/281

Since I'll not be able to run Deno on mobile, I'll have to take a more integrated approach. My idea is to still build the whole mobile web app with ESBuild using Deno as a build system as right now, but replace the "HttpSpace" implementation (to access files, run some syscalls remotely) with one that is talking to the local file system, and reimplementing all syscalls using CapacitorJS plugins (there are APIs to get access to the FS and I found a SQLite library as well). If I can get this to work, it may be possible to port this approach to the desktop app later too.

Once I get this to work (which may take some more time). We move on to...

Phase 3: Sync

So with a desktop app and mobile, how do we share stuff? For this, either we leverage something like iCloud, or build a simple sync engine: #261

Phase 4: Optimize

E.g. by applying the same approach to desktop as mobile, thereby eliminating the need for a Deno runtime on both.

At this stage the question wil be: will people want to use this purely as desktop and mobile apps> And if so... do we still need the "web app" version at all?

We'll see once (and if) we get there.

[zefhemel]

Addressing some things @yorrd mentioned: I have not registered any app names anywhere, but I don’t expect this to be a problem? I can always call it “Silver Bullet mobile” if somehow this is taken, not sure how globally unique these names need to be.

And regarding offline/online mode: yeah, my thinking is indeed: let’s go the sync route. Keep a copy locally on device (desktop and mobile) and sync it either through OS mechanisms (like iCloud, Dropbox) outside of SB’s control, or through a sync engine mentioned in #261 in which case you’d have to deploy some server.