Support local-only mode and network configuration

[FredericJacobs]

Have you read a contributing guide?

• [X] I have read CONTRIBUTING.md
• [X] I have searched the existing requests and didn't find any that were similar
• [X] I have considered creating a pull request instead and want to proceed

Clear and concise description of the problem

I would like to be able to use AnyType without the backup sync. I have not found a way to do that.

Suggested solution

Include a setting to allow disabling the use of the backup node.

I don't mind paying to not use any ressources! :)

Alternative

No response

Additional context

No response

[fuksman]

Hi @FredericJacobs, I think we can address this with the self-hosting configuration release. Thanks for the suggestion! Also, you can block outgoing external connections from Anytype using a firewall on your machine. Technically this will allow you to use Anytype without the backup sync.

[fenilgmehta]

Both solutions (i.e. self-hosting configuration & firewall) are quite complex. Can we please have a simple checkbox/switch in settings to control this.

[fuksman]

@fenilgmehta yes, we will deliver it in the upcoming releases, along with simplification of self-hosting.

[fenilgmehta]

@fuksman thanks a lot 🙏 Since this issue is closed, is there any other place (e.g. GitHub issue or some link) which we can use to check the status? Is it planned for this year?

[fuksman]

We're also working on mirroring the tasks from our internal tracker to the GitHub project. Until we implement this, I'm opening the issue to make it possible to check the status.

And yes, these features are planned for this year. Stay tuned 🙌

[tempuserr]

Will disabling backup sync also prevent Anytype clients on other devices from/to syncing content on the local network?

Because it would be a good option to disable WAN/Internet backup sync while still leaving sync of clients on local network possible.

[fuksman]

Hello @tempuserr, yes, it'll be possible to turn on "local-only mode": when data is synchronized between your own local devices but isn't backed up to external nodes.

[TheFuzzStone]

To make self-hosting/syncing more convenient, it would be very cool to have an option to tell the Anytype application where it should store the database/files, i.e. choose a custom location.

[hrqmonteiro]

We are already on 2023Q3. When will this be a thing?

[fuksman]

Hello @hrqmonteiro, this task was moved to 23Q4 on October 2. Unfortunately, we had to postpone this feature as the team is focused on delivering multiple spaces.

[ilugakov]

Hello @tempuserr, yes, it'll be possible to turn on "local-only mode": when data is synchronized between your own local devices but isn't backed up to external nodes.

Will storage limits be considered in this mode?

[fuksman]

@ilugakov there are no file storage limits on your device, they work only on the network level.

[okyq]

@fuksman Hello😊, I have two questions. 1. If local mode is used in the future, can two devices synchronize when they are on the same local network? 2. In the current mode, if my file exceeds 1 GB, does it mean that only the file beyond 1 GB cannot be synchronized to the backup node, but the two devices can still synchronize on the same local network?

[fuksman]

Hey @okyq,

1. Yes, devices in the same network can sync without access to external infrastructure.
2. Yes, 1GB is a restriction on the external infrastructure side. It doesn't restrict data on devices and their synchronization.

[tempuserr]

ad. 1. from my research (about 3 months back) it is not possible to sync locally in Anytype before some kind of "initialize" database on backup node. It means that you can only do local synchronize after "registering" device in cloud. So it effectively means that you can't synchronize local devices in 100% isolated local network - devices needs to connect to backup node in cloud firstly, so if someone needs 100% privacy it is not possible with current (3 months ago) versions of Anytype. There is thread in this repo about this issue.

[fuksman]

@tempuserr, that's what local-only mode is about: it will allow you to create/recover your spaces without a connection to the external infrastructure.

[tempuserr]

But all claims from anytype.io website like "local first", "it can work without a connection to the internet", "users' data stays with them even when they are not connected", "peer-to-peer", "allow users to share files, data or resources directly with each other without the need for a central server or authority" and the overall emphasis on privacy and security in the descriptions comes with an unclear and somewhat obscured 'fine print' that device registration in the Anytype cloud is required to use the local mode at all.

Of course I appreciate all effort for creating this app! But the image created on the website that Anytype is an application guaranteeing privacy is simply false because it is unable to operate in an isolated network without internet access at all. The only solution to run Anytype in private isolated environment - as I know after days of reading forums - is to self host own backup node (which is too complicated for an casual user). And users should be fully aware of this, not to be fooled by half-truth claims. My two cents.

[okyq]

@tempuserr Hello, you can conduct a test, which has already been successfully completed on my devices. You need to prepare three devices: A, B, C. Install Anytype on devices A and B. Then, use device C to enable a hotspot. Connect devices A and B to the hotspot of device C. Remember: device C only needs to enable the hotspot, without connecting to other Wi-Fi or using mobile data. This way, device C can only provide a Wi-Fi without access to the external network. Your devices A, B, and C are in the same local network, and they cannot access the external network at all, operating entirely locally. Now you can start the test: modify or add notes on device A, and you will find that it synchronizes to device B at an extremely fast speed. I believe this aligns with the original design intent of Anytype and corresponds to their advocated "local-first" approach. Because you have complete control over the data. The reason I abandoned Obsidian is that Obsidian cannot quickly sync between my multiple devices, while Anytype achieves this. It ensures data security and enables data synchronization across multiple devices. Thanks again to the Anytype team.

[okyq]

@tempuserr In other words, connecting to the network is only necessary for synchronization or backup purposes:

1. If you want to use Anytype's backup node, you need to connect to the external network.

2. If you don't need backup and only require synchronization, it's sufficient to keep the devices on the same local network; there's no need to connect to the external network (in this case, your data is already 100% secure).

3. If you neither need backup nor synchronization, you can disable Anytype's network permissions.

[tempuserr]

I made few tests like this (with Android-2-Android, Win-2-Win and Android-2-Win clients; with hotspots and router LAN) - it can work as you mention, BUT only after client is "registered" in backup node in Anytype's cloud. As can be seen here you can't add "clean and fresh" Anytype client (e.g. using Recovery Phrase) to local network - it needs firstly to connect to cloud to be registered. After this, sync like you described will work in isolated network. But from privacy point of view this device is compromised after connecting and registering in cloud.

Just my two cents if someone stumbled upon this thread enticed like me by the claims of "local first" / "without the need for a central server or authority".

edit: ad. 2. I have no success with add/initialize new clients/devices (especially with Recovery Phrase) to local sync without registering in cloud. But maybe there are some changes in this manner in versions since September '23. If I find some free time I will re-check this.

[TheFuzzStone]

I believe this aligns with the original design intent of Anytype and corresponds to their advocated "local-first" approach.

I don't think so. Simply because when you create a new account and create your first note, the app talks to the nodes first, and saves my note there. Also, the app doesn't give the user a choice or ask them if they want to send their data to external nodes? Moreover, there is currently no option to disable sending data to external nodes.

I don't see the "local-first" principle in this approach.

I'm very grateful that we have such a great application, but I don't want to store my data anywhere but on devices under my control. Therefore, I am very much looking forward to the "Offline mode" button + to be able to specify a custom location where to store the database. And then I will organize my own synchronization with what I need using Syncthing.

If you neither need backup nor synchronization, you can disable Anytype's network permissions.

What if you need synchronization in the local network (simply because there is no way to specify a custom location and use Syncthing), but don't need backup to external nodes? That's right, there are not many options so far:

1. special rules for firewall;
2. "crutches ":

fallocate -l 1.5G anytype_offline - get a 1.5 GB file and attach it to some note.

[okyq]

Yes, I also look forward to Anytype's pure local mode. Therefore, this issue remains open, and my response simply reflects my own perspective . In fact, after creating an account and before starting to record notes, you can disable Anytype's network permissions (Anytype doesn't require any personal information for creating an account, which is better than Notion in this regard). This means that none of your subsequent notes can be synchronized to Anytype's nodes. My approach is to add specific firewall rules to ensure that devices can synchronize within the local network.

[okyq]

Indeed, Anytype operates without requiring any network permissions; all data is stored locally with default backups on their nodes (though it's possible to use special methods to opt out of backups). As an open-source and free software, they have done a commendable job, albeit without providing users the power of choice. I believe they will continue to improve and offer users more flexibility as they evolve.

[fuksman]

@tempuserr, in September, you could have encountered a bug. This bug caused account recovery (which involves retrieving your spaces and objects using a recovery phrase) to always attempt to connect to the external network. As a result, if there was no connection, the recovery process would fail, instead of checking devices on the local network. This issue has been fixed, and all tests recommended by @okyq should work.

@TheFuzzStone, I believe there are differing opinions on the definition of "local first." That's why we make a distinction between "local first" and "local only." Anytype apps create, edit, and encrypt your objects on your device before synchronizing the changes with the external infrastructure. For example, if you create a space and its first object while on a flight without an internet connection, everything will function as expected, except for the backup. In our perspective, this is genuinely "local first." However, it is not strictly "local only."

Currently, if you want to avoid storing encrypted data in Anytype Network, there are three ways to achieve this:

1. Firewall rules.
2. Running self-hosted nodes in your local network.
3. Passing empty network configuration when building apps from source.

Creating a large file will halt the synchronization of files, but not objects and changes made to them.

Indeed, when we deliver local-only mode as an app setting, a much easier and preferable option will be available. Stay tuned; this is planned for the coming release in December.

[ffxxe]

@fuksman, I tested @tempuserr scenario on my end.

My setup: 1) Windows, AnyType 0.36.0. 2) Phone as a hotspot (with disabled cell and wifi networks). 3) MacOS, AnyType 0.36.0.

The first try

On the MacOS: "Create New Account" -> It showed me a recovery phrase -> I saved it to the txt file -> "Go to the app".

Then I transferred the recovery phrase txt from macOS to Windows through a USB drive.

On the Windows: "Log in" -> I pasted the recovery phrase from the txt -> "Log in" -> The app showed me an import backup option.

On the macOS: Settings -> Export:

• Markdown
• Any-Block JSON
• Any-Block Protobuf All with enabled: "Zip archive", "Include files", and "Include archived objects".

Transferred them from macOS to Windows through a USB drive again.

Import results:

• Markdown: "open ***: file does not exist"
• Any-Block JSON: "failed to extract config: config.json not found in archive"
• Any-Block Protobuf: "failed to extract config: config.json not found in archive"

I couldn't sync them.

The second try

I created on the macOS a new account. Logged in on the Windows and the app just started working. The status "Backup node" is a yellow inscription "Syncing...". The app didn’t create anything in the "./anytype/data//nodeconf/" folder.

After connecting to the internet in the folder appeared file and the status "Backup node" became green inscription "Synced".

To sum up looks like local sync without initialization works, but not really well.

[derNicklas]

?

[fuksman]

@derNicklas done, available in pre-release version, will be fully released tomorrow.