search

filecoin-station / core

Station is a visionary network that connects your computer's idle resources to the Filecoin network and rewards you with FIL
https://filstation.app
Other
27 stars 8 forks source link

Bug: module caches are synced to iCloud #75

Closed bajtos closed 1 year ago

bajtos commented 1 year ago

Station Core tells modules to store state files in ~/Library/Application Support:

https://github.com/filecoin-station/core/blob/ac03923e1dc0494fa82e7a50760f16dd430fbc2b/lib/paths.js#L31-L37

According to Apple File System Conventions, these files are backed up by iTunes and iCloud.

Use this directory to store all app data files except those associated with the user’s documents. For example, you might use this directory to store app-created data files, configuration files, templates, or other fixed or modifiable resources that are managed by the app. An app might use this directory to store a modifiable copy of resources contained initially in the app’s bundle. A game might use this directory to store new levels purchased by the user and downloaded from a server. All content in this directory should be placed in a custom subdirectory whose name is that of your app’s bundle identifier or your company. In iOS, the contents of this directory are backed up by iTunes and iCloud.

I am proposing to change Station Core to use the cache path instead, the same way as Station Desktop does.

https://github.com/filecoin-station/desktop/blob/7ab56a4479ec1fe96bceed1c3c787cc365d9ca6f/main/consts.js#L30-L31

    case 'darwin': // macOS
      return path.join(app.getPath('home'), 'Library', 'Caches', app.name)

Quoting from File System Conventions:

Use this directory to write any app-specific support files that your app can re-create easily. Your app is generally responsible for managing the contents of this directory and for adding and deleting files as needed. In iOS 2.2 and later, the contents of this directory are not backed up by iTunes or iCloud. In addition, the system removes files in this directory during a full restoration of the device. In iOS 5.0 and later, the system may delete the Caches directory on rare occasions when the system is very low on disk space. This will never occur while an app is running. However, be aware that restoring from backup is not necessarily the only condition under which the Caches directory can be erased.

See also macOS Library Directory Details:

Application Support - Contains all app-specific data and support files. These are the files that your app creates and manages on behalf of the user and can include files that contain user data. Caches - Contains cached data that can be regenerated as needed. Apps should never rely on the existence of cache files. Cache files should be placed in a directory whose name matches the bundle identifier of the app.

bajtos commented 1 year ago

Why is this a problem: we don't want to back up the content cached by the Saturn L2 Node to the user's iCloud account, do we?

juliangruber commented 1 year ago

Hmm, I don't know. We're promising that the folder passed to modules will be persisted, as the module is also expected to store the metrics count in there. A cache folder makes sense for the Saturn L2 module's file storage, but not for the metrics file, and not for arbitrary files that any module can store.

Use this directory to write any app-specific support files that your app can re-create easily.

This is true for the L2 cache, but not the metrics and also not other files necessarily.

What do you think?

juliangruber commented 1 year ago

Neither is optimal, agreed, but I think the more persisted a location is, the more correct it is.

You quoted above:

In iOS, the contents of this directory are backed up by iTunes and iCloud.

This isn't running in iOS, only macOS.

bajtos commented 1 year ago

We're promising that the folder passed to modules will be persisted, as the module is also expected to store the metrics count in there. A cache folder makes sense for the Saturn L2 module's file storage, but not for the metrics file, and not for arbitrary files that any module can store.

It makes me wonder when are the files in the cache folder deleted. I remember that Windows offer an action to get more free space by removing temporary files. If that deletes app caches but leaves other app data in place, then we definitely want to store our counters in the app data, not the app cache folder.

IMO, this tells us that we need more than one directory.

In iOS, the contents of this directory are backed up by iTunes and iCloud.

This isn't running in iOS, only macOS.

Yeah, the docs are not the best.

In my macOS Time Machine settings, I see that ~/Library/Caches is excluded from backups, but ~/Library/Application Support is not.

At the same time, many directories inside Application Support are excluded from backups via filesystem metadata (see this AskDifferent response for more details: https://apple.stackexchange.com/a/25833/51077).

It could be a viable option to exclude cache files from backups using filesystem metadata. I don't think that's something for modules to implement though, IMO it should be handled by Station and/or Zinnia.

juliangruber commented 1 year ago

Sgtm, exposing multiple directories is reasonable for module builders and maps to concepts existing in all major operating systems 👍

bajtos commented 1 year ago

Sgtm, exposing multiple directories is reasonable for module builders and maps to concepts existing in all major operating systems 👍

In that case, I propose to rework ROOT_DIR to STATE_DIR and CACHE_DIR - both in Desktop Core and Zinniad. WDYT?

juliangruber commented 1 year ago

Sounds good to me!

bajtos commented 1 year ago

In that case, I propose to rework ROOT_DIR to STATE_DIR and CACHE_DIR - both in Desktop Core and Zinniad. WDYT?

Opened an ADR here: https://github.com/filecoin-station/zinnia/pull/152

bajtos commented 1 year ago

Closing in favour of follow-up issues: