[Bug]: Text app saving/versioning is not consistent

dee-kryvenko commented 9 months ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

I am observing data in my S3 bucket (AWS) inconsistent with content on my Nextcloud server. See repro steps for more details, I did came across this issue https://github.com/nextcloud/server/issues/41917 which may or may not be related, my symptoms are slightly different and this is not a new issue - I observed it for two months, I was waiting for https://github.com/nextcloud/fulltextsearch_elasticsearch/issues/326 to be fixed before reporting this on the off chance it was related.

Steps to reproduce

Mount S3 external folder.
Create a new blank file test.md via NextCloud UI, click save, exit the editor.
Look into Details -> Versions, there will be Initial version and Current version, both 0 B.
Look on S3 - you will also see two distinct versions, the first will always be Standard storage class and second (current) will be Intelligent-Tiering - not sure if this is AWS or NextCloud doing, I would expect them both be Intelligent-Tiering since my bucket is configured for it. This is a first sign of trouble, that there are two versions from get-go and that one version is a wrong storage class.
Open test.md in UI, add first line content, save the file, exit the editor.
Observe in Details -> Version that there is now a third version Current version correctly opening. The first two empty versions do not open, they are presenting a loading spinner forever and nothing pops up in the logs.
Observe on the S3 bucket that you have a new third (current) version with a correct content of first line.
Open test.md in UI, add second line content, save the file, exit the editor.
Observe in Details -> Version that there is now a new version, with correct new content.
Observe on the S3 bucket that you have a new (current) version with a correct content.
Open test.md in UI, add third line content, save the file, exit the editor.
Observe in Details -> Version that there is now a new version, with correct new content.
Observe on the S3 bucket that you still have old version as current with only two lines, and third line nowhere to be seen. I left it like that overnight thinking it may eventually sync, but it didn't. No errors in logs.
Adding more changes will trigger sync again. It is some kind of a race condition, sometimes I get inconsistent data on odd lines, sometimes - on even, sometimes it just works. Sometimes it is missing to create local versions too.

Additional details

I did not notice causality between how fast I am making changes and inconsistencies, it may occur on singular isolated edits. Although I did notice correlation, it is occurring more often on fast edits, which is a typical use case for a Markdown KB (add a thought, save, edit, save etc - and you never know if your latest edit actually made it to S3).

You may notice under disabled apps I have files_versions_s3 - I tried to install it to see if it makes any difference, it didn't. It was having this behavior before I ever installed it.

Expected behavior

Expected to see clear linear version history both in NextCloud and S3, and most importantly - expecting to always see latest and consistent content on S3. In my home use scenario, S3 should be the most reliable storage for most important documents and KB, because my home lab hardware may always be stolen or destroyed, and traditional scheduled backups are never latest. This bug is preventing me from fully trusting NextCloud over Google Drive. I may never be sure that my off-site data is consistent.

Installation method

Official All-in-One appliance

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Other

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

# docker exec --user www-data nextcloud-aio-nextcloud php occ config:list system
{
    "system": {
        "one-click-instance": true,
        "one-click-instance.user-limit": 100,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "check_data_directory_permissions": false,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "filelocking.enabled": true,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "***REMOVED SENSITIVE VALUE***",
        "overwriteprotocol": "https",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "27.1.4.1",
        "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "loglevel": "2",
        "log_type": "file",
        "logfile": "\/var\/www\/html\/data\/nextcloud.log",
        "log_rotate_size": "10485760",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "enabledPreviewProviders": {
            "1": "OC\\Preview\\Image",
            "2": "OC\\Preview\\MarkDown",
            "3": "OC\\Preview\\MP3",
            "4": "OC\\Preview\\TXT",
            "5": "OC\\Preview\\OpenDocument",
            "6": "OC\\Preview\\Movie",
            "7": "OC\\Preview\\Krita",
            "0": "OC\\Preview\\Imaginary"
        },
        "enable_previews": true,
        "upgrade.disable-web": true,
        "mail_smtpmode": "smtp",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "activity_expire_days": "30",
        "simpleSignUpLink.shown": false,
        "share_folder": "\/Shared",
        "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
        "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
        "updatedirectory": "\/nc-updater",
        "davstorage.request_timeout": 3600,
        "htaccess.RewriteBase": "\/",
        "dbpersistent": false,
        "files_external_allow_create_new_local": false,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "allow_local_remote_servers": true,
        "preview_imaginary_url": "http:\/\/nextcloud-aio-imaginary:9000",
        "default_phone_region": "US",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587 ",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "data-fingerprint": "d810320451ca5c99313f7c2bbc7e472f",
        "appsallowlist": []
    }
}

List of activated Apps

# docker exec --user www-data nextcloud-aio-nextcloud php occ app:list
Enabled:
  - activity: 2.19.0
  - admin_audit: 1.17.0
  - announcementcenter: 6.7.0
  - app_api: 1.3.0
  - calendar: 4.5.3
  - cfg_share_links: 4.2.0
  - checksum: 1.2.2
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contacts: 5.4.2
  - dashboard: 7.7.0
  - dav: 1.27.0
  - deck: 1.11.2
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_antivirus: 5.4.0
  - files_archive: 1.1.3
  - files_external: 1.19.0
  - files_fulltextsearch: 27.0.1
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - fulltextsearch: 27.0.2
  - fulltextsearch_elasticsearch: 27.0.5
  - groupfolders: 15.3.1
  - imageconverter: 1.3.5
  - integration_giphy: 1.0.8
  - integration_google: 2.1.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - metadata: 0.19.0
  - nextcloud-aio: 0.4.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.8.1
  - notifications: 2.15.0
  - notify_push: 0.6.5
  - oauth2: 1.15.1
  - oidc: 0.7.1
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - richdocuments: 8.2.3
  - secrets: 1.5.2
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - side_menu: 3.11.2
  - socialsharing_email: 3.0.0
  - spreed: 17.1.3
  - support: 1.10.0
  - survey_client: 1.15.0
  - systemtags: 1.17.0
  - tasks: 0.15.0
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_nextcloud_notification: 3.8.0
  - twofactor_totp: 9.0.0
  - twofactor_webauthn: 1.3.2
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - bruteforcesettings: 2.7.0
  - contactsinteraction: 1.8.0 (installed 1.8.0)
  - encryption: 2.15.0
  - files_versions_s3: 0.2.1 (installed 0.2.1)
  - suspicious_login: 5.0.0
  - user_ldap: 1.17.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

joshtrichards commented 9 months ago

Looks like maybe nextcloud/text#4823 (fixed by nextcloud/text#4824) and/or nextcloud/text#3404 (fixed by nextcloud/text#4286). The former should have been released with NC V27.1.4 (it was backported in nextcloud/text#4968), but I see no version bump in the shipped Text app (from v3.8.0) in a stock NC v27.1.4 installation so maybe something got messed up there. I don't think nextcloud/text#4286 was backported, but it should be IMO.

So, in any case, this may be a Text app only issue, but until confirmed I'll leave here in this repo (plus it's a shipped app so we definitely want it to work well!)

Cc: @max-nextcloud @juliushaertl

dee-kryvenko commented 9 months ago

Hmmm it didn't occur to me that the issue may be with UI not with S3 storage. Indeed both local and S3 versions are getting messed up in the end. Let me test this with a) md files on local disk and b) files other than md files.

dee-kryvenko commented 9 months ago

Ok yes, I can definitely reproduce the same with SMB external storage as well. https://github.com/nextcloud/text/issues/3404 gave me the key - it is increasingly likely to reproduce if I keep making changes and saves without exiting text editor.

Doesn't look like I can reproduce it on S3 external storage by uploading new versions of files via UI, or editing it on my machine mounted as webdav. Good news I don't have to worry about integrity of my non-text files. Bad news, my KB might be corrupted as much as in my own head lol

nextcloud / text