[Bug]: Random crash on Win11

pcuci commented 1 week ago

Verified issue does not already exist?

[X] I have searched and found no existing issue
[X] I will be providing steps how to reproduce the bug (in most cases this will also mean uploading a demo budget file)

What happened?

App crashes randomly, console logs show a file read error:

VM89:2 Server Log: [Exception] [Error: EBUSY: resource busy or locked, open 'C:\Users\pcuciureanu\Documents\Actual\My-Finances-15bd76d\metadata.json'] {
  errno: -4082,
  code: 'EBUSY',
  syscall: 'open',
  path: 'C:\\Users\\pcuciureanu\\Documents\\Actual\\My-Finances-15bd76d\\metadata.json'
}

I've detailed the symptoms in Discord: https://discordapp.com/channels/937901803608096828/1171096337111064626/1282059693145198772

Cannot reproduce the issue reliably, the crash/hang happens mostly when on the Budget page or the Reports page (but this could only be a coincidence)

Where are you hosting Actual?

Docker

What browsers are you seeing the problem on?

Desktop App (Electron)

Operating System

Windows 11

pcuci commented 1 week ago

I should mention that my data file name is Finances and not My-Finances-15bd76d - so it's not clear what is trying to read that file and crash the app

Shouldn't there also be a Finances file as well? (I'm sync'ed against local docker compose'd actual-server)

When I go through the motion to reconnect to the server, I see only my desired Finances data file:

pcuci commented 1 week ago

Ok, no. (I deleted the local file)

When I download from the server, the downloaded version is named My-Finances-15bd76d even though in-app name is Finances

I guess this further confuses the bug? :-)

MikesGlitch commented 1 week ago

This is a weird one. I wonder if anyone else is seeing this (please chime in if you are).

The folder name is the budget id - that's why it has the extra characters, they enforce uniqueness. "Finances" is the budget's display name so that's all as expected.

I've been running the windows app for months and have never seen this. The only thing I can think of is maybe some other program is looking at that metadata.json file - OneDrive maybe?

App crashes randomly, console logs show a file read error:

Does the app close or does it become unresponsive/unable to navigate?

Are you able to replicate this often? If so, it would be good if you could try out the Windows release here: https://github.com/actualbudget/actual/actions/runs/10758945924/artifacts/1905808658

I made a small tweak to something that was saving to that file quite a lot. Maybe it will fix it.

pcuci commented 1 week ago

Does the app close or does it become unresponsive/unable to navigate?

It hangs and most pages go blank, then I have to restart the desktop app.

Installed the new version, will monitor for errors.

OneDrive maybe

I've also moved the data file outside the purview of OneDrive, and it's already behaving better. I.e.: restarting Actual asked me to pick from the remote sync'd list of data files.

Oops. Spoke too soon. On 2nd restart:

VM15:2 Server Log: ab [Error]: PostError: file-not-found
    at uA (C:\Program Files\WindowsApps\actualbudget.org.ActualBudget_24.9.0.0_x64__6q3amkrs0bv0p\app\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:115:5390)
    at uD (C:\Program Files\WindowsApps\actualbudget.org.ActualBudget_24.9.0.0_x64__6q3amkrs0bv0p\app\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:115:6263)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async lJ (C:\Program Files\WindowsApps\actualbudget.org.ActualBudget_24.9.0.0_x64__6q3amkrs0bv0p\app\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:135:2852)
    at async C:\Program Files\WindowsApps\actualbudget.org.ActualBudget_24.9.0.0_x64__6q3amkrs0bv0p\app\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:135:1592 {
  meta: undefined,
  reason: 'file-not-found',
  type: 'PostError'
}

After a 3rd restart, same exact behaviour as before, new local Unregistered file created.

Strangely, however, when I reconnect to actual-server I do only see my Finances data file(?)

Thanks!

pcuci commented 1 week ago

Actually, no. The data file location change did not persist:

Also my yyyy-mm-dd setting reverts to the default mm/dd/yyyy

And SimpleFIN got unchecked, even though I see the green dots next to each connected account.

pcuci commented 1 week ago

Ok, the behaviour is erratic, after yet another restart of the app, I now see the new location...

however, the date format reverted again:

Ok, restarted one more time, seems to have persisted(?):

MikesGlitch commented 1 week ago

You're doing a lot of things at once and I'm having trouble following 😆

If you change your file directory through the settings page we don't automatically move your budget file or settings. We should improve this functionality as a part of #3371.

That means when you re-downloaded from the server, you did so without your settings, so the last two comments are expected.

If you want to change directory, first change the directory on the Settings page, then manually move your current budget folder to the new directory that you've specified as per the instructions (that nobody reads 🤣).

pcuci commented 1 week ago

That means when you re-downloaded from the server, you did so without your settings, so the last two comments are expected.

Ok.

I've shut down the docker compose'd server to attempt to reproduce this.

and the settings seem to have persisted on the client app.

Then, restarting the server.

Though the UI says I can sign out from an offline server(?)

Reconnecting

Then restarted the desktop app once more, and settings keep persisting as they should.

I'll monitor for stability, but so far so good.

Thanks for your help, @MikesGlitch!

pcuci commented 1 week ago

Ok, it's back—this invalidates the OneDrive hypothesis :-(

 Server Log: [Exception] [Error: EBUSY: resource busy or locked, open 'C:\Users\pcuciureanu\apps\Actual\My-Finances-15bd76d\metadata.json'] {
  errno: -4082,
  code: 'EBUSY',
  syscall: 'open',
  path: 'C:\\Users\\pcuciureanu\\apps\\Actual\\My-Finances-15bd76d\\metadata.json'
}

(anonymous) @ unknown
 Server Log: node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: EBUSY: resource busy or locked, open 'C:\Users\pcuciureanu\apps\Actual\My-Finances-15bd76d\metadata.json'] {
  errno: -4082,
  code: 'EBUSY',
  syscall: 'open',
  path: 'C:\\Users\\pcuciureanu\\apps\\Actual\\My-Finances-15bd76d\\metadata.json'
}

Node.js v20.11.1

Nothing I do can bring the app back, only a restart works.

And on reboot, same old story:

MikesGlitch commented 1 week ago

Cool, thanks for reporting. I'll see if I can replicate this and get back to you.

MikesGlitch commented 1 week ago

I have a theory. Are you installing via the .exe installer on the release page or are you using the Windows Store app?

If it's the .exe installer maybe your antivirus is mistreating it because it doesn't have a certificate. It could be scanning the files when they change and locking them while doing so. The Windows Store app has a certificate, so if you use that then it must be something else.

Another thing that might help, can you share your metadata.json file?

pcuci commented 1 week ago

Initially, I installed via the windows store, then via the link you shared.

Hm, which one is which?

What to investigate re: antivirus?

pcuci commented 1 week ago

Uninstalled both, just to make sure nothing else is running, then installed only the latest fix you shared.

Same problems.

Perhaps for you to reproduce, I'm noticing that if I switch between months quickly (I have 3 or 4 at a time in the Budget view), it crashes quite reliably.

But you gotta be fast, clicking wildly :-)

Server Log: Loaded spreadsheet from cache (29658 items)

2 Server Log: ab [Error]: PostError: file-not-found
    at uA (C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:115:5390)
    at uD (C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:115:6263)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async lJ (C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:135:2852)
    at async C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:135:1592 {
  meta: undefined,
  reason: 'file-not-found',
  type: 'PostError'
}
ab [Error]: PostError: file-not-found
    at uA (C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:115:5390)
    at uD (C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:115:6263)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async lJ (C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:135:2852)
    at async C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:135:1592 {
  meta: undefined,
  reason: 'file-not-found',
  type: 'PostError'
}

2 Server Log: ab [Error]: PostError: file-not-found
    at uA (C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:115:5390)
    at uD (C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:115:6263)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async lJ (C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:135:2852)
    at async C:\Users\pcuciureanu\AppData\Local\Programs\desktop-electron\resources\app.asar\node_modules\loot-core\lib-dist\bundle.desktop.js:135:1592 {
  meta: undefined,
  reason: 'file-not-found',
  type: 'PostError'
}

 Server Log: Loading fresh spreadsheet

 Server Log: Loaded spreadsheet from cache (29598 items)

VM21:2 Server Log: [Exception] [Error: EBUSY: resource busy or locked, open 'C:\Users\pcuciureanu\apps\Actual\My-Finances-15bd76d\metadata.json'] {
  errno: -4082,
  code: 'EBUSY',
  syscall: 'open',
  path: 'C:\\Users\\pcuciureanu\\apps\\Actual\\My-Finances-15bd76d\\metadata.json'
}

(anonymous) @ VM21:2

MikesGlitch commented 1 week ago

"Actual Budget" is the store app, "Actual 24.9.0" is the exe instlled app. You can uninstall the "Actual 24.9.0" now that we know that hasn't fixed it.

I've tried going mad with clicks and I can't break it 🙃

cant break it

The next time it locks, can you try the steps here - it might help us figure out if it's another program causing it: https://www.thewindowsclub.com/how-to-tell-which-process-is-locking-a-file

If you have Microsoft Powertools installed there's a utility called "File Locksmith" that's a bit easier to use and could also identify the lock.

To make sure it's not antivirus you could try turning it completely off, including "realtime protection" and then see if you can replicate. We're fishing at this point - I'm running out of ideas.

Oh, could you also try running as "Administrator" - if another app is locking the file running as admin might give Actual precedence.

pcuci commented 1 week ago

This confirms Actual is hijacking itself :-)

Following your lead, and a few pages of gpt-4o later, I used Process Monitor to observe the catch-22 race condition—most other tools I tried only search for active locks.

I believe a simple retry on a file system error trying to recreate this file might solve the issue?

pcuci commented 1 week ago

Maybe here? Sorry, I'm new to Electron, don't know exactly how I could debug this locally.

https://github.com/actualbudget/actual/blob/95ed7aaf2711fe62bbe3a6b60a1e157bf98da00e/packages/loot-core/src/server/cloud-storage.ts#L225

MikesGlitch commented 1 week ago

Maybe here? Sorry, I'm new to Electron, don't know exactly how I could debug this locally.

https://github.com/actualbudget/actual/blob/95ed7aaf2711fe62bbe3a6b60a1e157bf98da00e/packages/loot-core/src/server/cloud-storage.ts#L225

That code only runs when you import a budget for this first time, and the file should never be deleted/modified by anything outside of Actual.

That screenshot you posted shows MsSense.exe using the file - you can see where it creates the OPLOCK and then releases it after Actual trips up. MsSense is a part of Defender for Endpoint (enterprise MS Defender).

don't know exactly how I could debug this locally

If you're up for debugging you just need to clone the code, install deps and run yarn start:desktop. That should run the desktop app.

I'm tempted to do nothing here, we're about to move the metadata.json file into the DB so the problem will disappear on its own. I'm guessing this only affects computers with that kind of security software installed.

Did you try running as admin? That might help you work around the issue until metadata.json is moved to the DB.

pcuci commented 1 week ago

Did you try running as admin?

Crash persists

This might be related? https://github.com/actualbudget/actual/pull/3398

until metadata.json is moved to the database

Ah, that'll probably do it :-), happy to wait for this refactor. Can't find an associated issue, do you have a link I could monitor?

Cheers!

MikesGlitch commented 1 week ago

Can't find an associated issue, do you have a link I could monitor?

I don't think there is one but there's been work happening towards that end within the last few days - the PR's named "(synced-prefs)". I'll update this issue when It's done - hopefully by next release.

For further reading - it looks like we're not the only ones affected by this issue - there's some other suggestions too: https://techcommunity.microsoft.com/t5/microsoft-defender-for-endpoint/is-there-any-way-avoiding-files-locking-by-defender-for-endpoint/m-p/3937819

MikesGlitch commented 1 week ago

I found out that we're not going to remove the metadata.json file entirely, so I've mitigated the issue with retries.

If you want to test it, you can get the download here: https://github.com/actualbudget/actual/actions/runs/10780875572/artifacts/1911159741

Otherwise it should be available in the next release. I've tested it myself by locking the file manually - should work a treat.

In future we'll need to think about debouncing the calls to the file system, but for now this should fix it.

actualbudget / actual