obsidian 1.4.5 supported? - yes

wangweitung commented 1 year ago

obsidian 1.4.5 supported？

acheong08 commented 1 year ago

Downloading and testing in a moment

wangweitung commented 1 year ago

Downloading and testing in a moment

my obsidian updated to 1.4.5 automatically

it seems losing the connection with the remote vault

and can not see any error log ...

acheong08 commented 1 year ago

Looks like they added mitigations to prevent the requests from being intercepted. It's also not working for me

acheong08 commented 1 year ago

var fn = "https://" + [String.fromCharCode(97, 112, 105), "obsidian", "md"].join(".");

They obfuscated the URL a bit...

wangweitung commented 1 year ago

var fn = "https://" + [String.fromCharCode(97, 112, 105), "obsidian", "md"].join(".");

They obfuscated the URL a bit...

is it easy to fix？

acheong08 commented 1 year ago

is it easy to fix？

The URL obfuscation is just funny. I'm still trying things out.

acheong08 commented 1 year ago

Looks like the main problem is that plugins are now loaded after this line of code runs:

var mn = "obsidian-account"
          , gn = window.fetch;

which ensures that even if I override window.fetch, they maintain a copy of it

It's somewhat obvious they are intentionally trying to stop this from working (I guess it is a security issue if plugins can intercept requests)

truongmanhsang commented 1 year ago

Things are complicated now, maybe we just stay at version 1.3.7. 😅

wangweitung commented 1 year ago

Looks like the main problem is that plugins are now loaded after this line of code runs:
var mn = "obsidian-account"
          , gn = window.fetch;
which ensures that even if I override window.fetch, they maintain a copy of it

It's somewhat obvious they are intentionally trying to stop this from working (I guess it is a security issue if plugins can intercept requests)

seems llike the iOS and the jailbreakers.

so the current solution is to stay at 1.3.7 if I want to use custom sync.

maybe this should be added to the readme.

acheong08 commented 1 year ago

CC @CzBix. They're much more experienced in JavaScript. Maybe there is a way. Until then, you'll need to stay at 1.3.7

xqdoo00o commented 1 year ago

Is that possible to register or change service worker js? If so, you can intercept fetch from service worker

acheong08 commented 1 year ago

Is that possible to register or change service worker js? If so, you can intercept fetch from service worker

I'm not sure. When trying to register a service worker:

Service Worker registration failed: TypeError: Failed to register a ServiceWorker: The URL protocol of the current origin ('app://obsidian.md') is not supported.

Trying window.addEventListener("fetch",... didn't do anything.

CzBiX commented 1 year ago

I haven't found a reliable solution yet. but this recent update shows the official stance on these things. Cat and mouse games aren't fun, and most modifications can be easily blocked by officials.

acheong08 commented 1 year ago

For flatpak to downgrade:

sudo flatpak update -commit=df4ad574288b02356d5318f6d5c1edd1d7d88320a5c08d96472d695cd509bff4 md.obsidian.Obsidian

acheong08 commented 1 year ago

Note: If you had already signed in & connected to a vault, it stays connected despite upgrading to 1.4.5. It seems the mitigations are only at the login section.

f0ff886f commented 1 year ago

I lost sync on all my devices that updated to 1.4.5. iOS and MacOS both are unable to connect to the vault that was previously connected. The only thing that changed was Obsidian version, so it seems at least I lost access after upgrade without logging in from scratch.

Funny thing for them to do, considering the CEO was so nice and received so much praise. It's not that they did this just as security either, the string obfuscation was clearly aimed at these plugins. Business moves: say one thing, do the other.

acheong08 commented 1 year ago

I lost sync on all my devices that updated to 1.4.5. iOS and MacOS both are unable to connect to the vault that was previously connected. The only thing that changed was Obsidian version, so it seems at least I lost access after upgrade without logging in from scratch.

For MacOS, it is still possible to manually patch the obsidian.asar file within Obsidian.app. You can also download the previous DMG from https://github.com/obsidianmd/obsidian-releases/releases/tag/v1.3.7

IOS, I have no idea.

wangweitung commented 1 year ago

I lost sync on all my devices that updated to 1.4.5. iOS and MacOS both are unable to connect to the vault that was previously connected. The only thing that changed was Obsidian version, so it seems at least I lost access after upgrade without logging in from scratch.

For MacOS, it is still possible to manually patch the obsidian.asar file within Obsidian.app. You can also download the previous DMG from https://github.com/obsidianmd/obsidian-releases/releases/tag/v1.3.7

IOS, I have no idea.

how about Android and win？

it this possible to patch apk and exe installer？（seems we are going too far away ...）

acheong08 commented 1 year ago

how about Android and win？

You can also patch those since you can sideload rather than being forced through an app store. You can either replace the URL directly (so no need for plugin) or replace their copy of window.fetch with just window.fetch

Or just downgrade the app (get apk/exe from obsidian-releases)

CzBiX commented 1 year ago

Found multiple solutions that don't require modifying the official app, but I won't share them here. since there's no point in updating the plugin further, unless the official attitude changed.

wangweitung commented 1 year ago

Found multiple solutions that don't require modifying the official app, but I won't share them here. since there's no point in updating the plugin further, unless the official attitude changed.

Perhaps most of us will have to buy the official sync service which will make the official side much more confident they make the right choice.

scr4tchy commented 1 year ago

Obsidian's CEO @kepano stated, days ago:

Impressive! It's fun to see the diversity of ways people sync/backup their Obsidian files. The nice thing about storing all your notes on your device is that it makes it possible to move and edit your Markdown files in many different ways. That diversity of solutions is what makes the ecosystem of Markdown tools resilient over the long term.

There are already a handful of tools that allow you to sync your notes for free, including Git, Syncthing, and some other options more specialized for Obsidian (see community plugins).

[...]

Reverse engineering things is a fun technical challenge, and also helps us find potential holes in our system. The main problems I see with your solution: 1. it could easily break in a future update to the app, 2. "Obsidian Sync" is a trademark, so you should consider renaming the repo otherwise it can be confused for an official tool — that would be my only request

Empashis on **that would be my only request**. So I would be you, I'd send him a message, and challenge his own words. It may be that the promise can be honored, and that the community can retain it if you rename the repository. It's not like this is going to change dogshit for their revenue (but the corpo drones's understanding and tunnel vision is probably too poor to see this) - as retail will just buy their subscription anyways; and the advanced users will just use Logseq and every other available free tools out there.

acheong08 commented 1 year ago

it could easily break in a future update to the app,

I think that might have a been a hint lol

acheong08 commented 1 year ago

and the advanced users will just use Logseq and every other available free tools out there.

I'm also planning on moving over to Logseq. I thought I would give Obsidian a try despite it being closed source due to its reputation as community driven & friendly. Since I've only installed Obsidian for a week or so, I'm not too heavily invested yet.

acheong08 commented 1 year ago

From Logseq:

During the beta test, we're giving all of our active Open Collective contributors access to Logseq Sync. That means you need to have monthly recurring donation of either $5 per month or $15 per month. We'll implement a normal subscription system once we launch Sync to the general public.

Looks like Logseq is also lacking a self-hostable sync option. Well, there's my next project I guess. Since the client is open source, it should be much much easier.

Edit: Looks like it's actually possible to reuse this server and just write a compatible plugin

Aetherinox commented 1 year ago

Very sad to hear. I've been a Obsidian Sync customer and I'm debating on cancelling my subscription now and just using Syncthing. If they're going to pull stuff like this, then I'm going to throw them in the category just as the other ass companies.

I wanted this as a secondary solution to back up the vault to a local database and make it easier but also keep Obsidian Sync on another device. Hopefully it gets fixed, for now, I'm not upgrading Obsidian.

kepano commented 1 year ago

We were able to patch some vulnerabilities with 1.4.5 — but that doesn't mean you can't keep trying!

What we're okay with:

Reverse engineering for fun, as a technical challenge, or to help build a plugin
Making plugins/services that let users sync/publish for free (there are many of them already)
Submitting those plugins to the official directory, as long as they meet our developer policies

What we're not okay with:

Using Obsidian trademarks and code in a way that could confuse users into thinking it's an official solution
Leaving security vulnerabilities open
Reverse engineering for the purpose of copying our code to make commercial solutions, see the Restrictions section of our terms

For reference here's what I sent to @acheong08 via email on Aug 29

I noticed this bit in your FAQ which is not quite accurate:

Why is this not part of the community plugins? A: This cuts into their profits since they charge $10/month for sync

If this were true then we would not allow any sync/publish plugins at all. There are many third party sync/publish plugins in the official community directory e.g. Obsidian Git, Self-hosted LiveSync, Yet Another Obsidian Synchronizer, etc. You are welcome to create your own syncing plugin that has its own client/server code and submit it — as long as it adheres to the Developer Policies.

It's important to delineate between lowercase "syncing" and "Obsidian Sync". Anyone can create syncing tools, whereas "Obsidian Sync" refers specifically to the official first-party service offered by Obsidian.

We plan to keep Obsidian Sync first-party for a simple reason: users expect Obsidian to take their data very seriously. When users choose Obsidian Sync, we want to make sure that they do not experience data loss or privacy issues that might stem from third-party code. Of course it helps us cover our development and maintenance costs as well, but we think this is important for the long term health of Obsidian since we are 100% user-supported, not funded by VCs/investors.

acheong08 commented 1 year ago

Using Obsidian trademarks and code in a way that could confuse users into thinking it's an official solution

I think rev-obsidian-sync or obi-sync are somewhat reasonable. I also make it quite clear that this is unofficial.

Leaving security vulnerabilities open

I suppose this is valid. Plugins intercepting and modifying requests could be a nightmare if not fully reviewed (causing issues with official features etc)

Reverse engineering for the purpose of copying our code to make commercial solutions

This is by no means commercial nor have I copied your code.

I don't have access to your server code in the first place.
I have not copied any client code since this is a server...
According to https://www.supremecourt.gov/docket/docketfiles/html/public/18-956.html, APIs can't be coprighted.

we want to make sure that they do not experience data loss or privacy issues that might stem from third-party code

My initial reason for wanting to self host was for privacy & data integrity reasons. It gives more control over backup & data is only sent to a place I control.

kepano commented 1 year ago

I think rev-obsidian-sync or obi-sync are somewhat reasonable. I also make it quite clear that this is unofficial

Thanks for changing it. Previously it was less clear since the repo was called "Obsidian Sync".

I suppose this is valid. Plugins intercepting and modifying requests could be a nightmare if not fully reviewed (causing issues with official features etc)

I want to note that your solution isn't exactly an "open-source Obsidian Sync alternative". It is a Sync server emulator with a plugin that tricks the Obsidian Sync client to connect to the emulated server, making full use of the Obsidian Sync client in a way that is unintended by us.

It takes over the account system in a way that presents the user the identical interface for the official account system. It then also uses the official Sync plugin, including the syncing code and the interfaces to setup and manage the synchronization.

This is dangerous because a non-technical user (someone who doesn't understand that this system works by taking over the real account system) using this system could run into data loss due to bugs in the emulated server, and be led to think that Obsidian Sync is buggy.

What we did in the update is to prevent plugins from taking over the account system and pretending to be the official Obsidian account, and official Obsidian Sync client. It is indeed a security vulnerability that we felt necessary to address.

This is by no means commercial nor have I copied your code.

Apologies, I didn't mean to imply that your tool has commercial intent — I was trying to be exhaustive about the cases where reverse engineering can be a problem.

acheong08 commented 1 year ago

This is dangerous because a non-technical user (someone who doesn't understand that this system works by taking over the real account system) using this system could run into data loss due to bugs in the emulated server, and be led to think that Obsidian Sync is buggy.

Although I doubt a non-technical user would be able to set up the sync server & manage plugins not in the official list etc, I somewhat recognize the concerns.

I won't be working on the plugin anymore & will be moving on to trying to bring compatibility with this API to Logseq so that this project isn't a complete waste.

^{(still feel like users should at least have the choice to do whatever dangerous thing they want, perhaps an option to allow override with big red warnings - not a recommendation, just a dream)}

wangweitung commented 1 year ago

there is no need to discuss any more,issue closed.

t3chguy commented 1 year ago

@acheong08 the work you have done is awesome. As a prominent member of the open source community, I have to say the behind the curtains way that Obsidian.md has dealt with this is crap. Security is one thing. Transparency is a far more important one.

https://github.com/t3chguy/rev-obsidian-sync-plugin seems to almost work on 1.4.5. It lets me auth & connect to a vault but the websocket connection fails. Will give it another go sometime soon.

t3chguy commented 1 year ago

Switched /ws for /ws.obsidian.md to bypass this check

Aaaaand it works.

https://github.com/acheong08/obi-sync/commit/63bd1c4ed6097a44541ca40bf6ebdc6548957d4a

https://github.com/acheong08/rev-obsidian-sync-plugin/compare/master...t3chguy:rev-obsidian-sync-plugin:master

t3chguy commented 1 year ago

Huh I wonder if there's an even easier way to do this - by injecting a custom vault into the IDB. (truncated as a lazy redact even though there's no secrets in this test vault)