syncthing crashes since v1.26.0.0

Sylonin commented 10 months ago

Description of the issue

since the latest version syncthing crashes immediately after the app starts it

Reproduction Steps

sometimes the app works fine but otherwise syncthing just starts crashing. sorry I can't provide more details :(

if I use the button to force the app to not run syncthing and then force start it it'll launch for ~2 second (status icon goes green, appnseems to connect to the syncthing spi) but then syncthing crashes after that 1 second :(

Version Information

App Version: 1.26.0.2
Syncthing Version: Settings -> About -> syncthing version is blank
Android Version: LineageOS 19.1
Device manufacturer: OnePlus
Device model: 9 Pro

Device platform info

[ro.product.ab_ota_partitions]: [abl,aop,bluetooth,boot,cpucp,devcfg,dsp,dtbo,engineering_cdt,featenabler,hyp,imagefv,keymaster,modem,multiimgoem,odm,oplus_sec,oplusstanvbk,product,qupfw,qweslicstore,shrm,splash,system,system_ext,tz,uefisecapp,vbmeta,vbmeta_system,vbmeta_vendor,vendor,vendor_boot,vendor_dlkm,xbl,xbl_config]
[ro.product.board]: [lahaina]
[ro.product.brand]: [OnePlus]
[ro.product.build.date]: [Mon Nov  6 04:02:33 UTC 2023]
[ro.product.build.date.utc]: [1699243353]
[ro.product.build.fingerprint]: [OnePlus/OnePlus9Pro_EEA/OnePlus9Pro:11/RKQ1.201105.002/2109102008:user/release-keys]
[ro.product.build.id]: [TQ3A.230901.001]
[ro.product.build.tags]: [release-keys]
[ro.product.build.type]: [userdebug]
[ro.product.build.version.incremental]: [cc8b4d8489]
[ro.product.build.version.release]: [13]
[ro.product.build.version.release_or_codename]: [13]
[ro.product.build.version.sdk]: [33]
[ro.product.cpu.abi]: [arm64-v8a]
[ro.product.cpu.abilist]: [arm64-v8a,armeabi-v7a,armeabi]
[ro.product.cpu.abilist32]: [armeabi-v7a,armeabi]
[ro.product.cpu.abilist64]: [arm64-v8a]
[ro.product.device]: [OnePlus9Pro]
[ro.product.first_api_level]: [30]
[ro.product.locale]: [en-US]
[ro.product.manufacturer]: [OnePlus]
[ro.product.model]: [A0001]
[ro.product.name]: [OnePlus9Pro]
[ro.product.odm.brand]: [OnePlus]
[ro.product.odm.device]: [OnePlus9Pro]
[ro.product.odm.manufacturer]: [OnePlus]
[ro.product.odm.model]: [A0001]
[ro.product.odm.name]: [OnePlus9Pro]
[ro.product.product.brand]: [OnePlus]
[ro.product.product.device]: [OnePlus9Pro]
[ro.product.product.manufacturer]: [OnePlus]
[ro.product.product.model]: [A0001]
[ro.product.product.name]: [OnePlus9Pro]
[ro.product.system.brand]: [OnePlus]
[ro.product.system.device]: [OnePlus9Pro]
[ro.product.system.manufacturer]: [OnePlus]
[ro.product.system.model]: [A0001]
[ro.product.system.name]: [OnePlus9Pro]
[ro.product.system_ext.brand]: [OnePlus]
[ro.product.system_ext.device]: [OnePlus9Pro]
[ro.product.system_ext.manufacturer]: [OnePlus]
[ro.product.system_ext.model]: [A0001]
[ro.product.system_ext.name]: [OnePlus9Pro]
[ro.product.vendor.brand]: [OnePlus]
[ro.product.vendor.device]: [OnePlus9Pro]
[ro.product.vendor.manufacturer]: [OnePlus]
[ro.product.vendor.model]: [A0001]
[ro.product.vendor.name]: [OnePlus9Pro]
[ro.product.vendor_dlkm.brand]: [OnePlus]
[ro.product.vendor_dlkm.device]: [OnePlus9Pro]
[ro.product.vendor_dlkm.manufacturer]: [OnePlus]
[ro.product.vendor_dlkm.model]: [LE2125]
[ro.product.vendor_dlkm.name]: [OnePlus9Pro]
[ro.product.vndk.version]: [33]

Android Log

https://src.dyne.link/-/snippets/3

Catfriend1 commented 10 months ago

Uh oh, this is not the wrapper but the SyncthingNative crashing. Please try to capture all line of SyncthingNative involved and contact upstream at forum.syncthing.net . Hopefully someone has an idea why the go code crashes. Does the app work fine after starting afresh (backup config first!)?

Mine is still working fine. 🤷‍♂️

Sylonin commented 10 months ago

Ah ok, sorry, my bad

I'll contact upstream if the issue happens again - I cleared the app data and imported an export I made prior to clearing data and it seems stable now. :)

phillipberndt commented 10 months ago

Had the same issue, and the same workaround seems to have helped for me. Thanks for sharing it!

For the record:

Syncthing native excited with code 2, no errors or panics in the log. I expect that it did panic and print the panic output to a different file descriptor than where other logs land!? Tried the log to file option but didn't manage to then find the logfile.

blu3id commented 10 months ago

Also having the same issue. The workaround didn't work. I also can't get more useful logging beyond the native exit of code 2.

However this is likely related to networking somewhere as I can reliably cause Syncthing to crash when started if connected to a particular network with mobile data on. It seems to start fine on other networks and if either mobile or wifi is disabled. It remains stable if started and then networks change. I suspect that upstream is unlikely to see this bug because Synching remains running constantly rather than being started on multiple different networks.

I will try to get useful logging and post bug report upstream.

phillipberndt commented 10 months ago

The problem returned for me a day later, seems I just got lucky when testing before. Tried upstream Syncthing v1.26.0, and indeed it doesn't have the issue (so far).

Catfriend1 commented 10 months ago

The problem returned for me a day later, seems I just got lucky when testing before. Tried upstream Syncthing v1.26.0, and indeed it doesn't have the issue (so far).

Do you mean the official Syncthing Android app ?

phillipberndt commented 10 months ago

Right. With settings imported from an export from the fork.

Catfriend1 commented 10 months ago

I'll compare the build recipe for the native next dev session. Maybe go or ndk version differs?

blu3id commented 10 months ago

As another data point: I have done what @phillipberndt did and imported export from fork into the official app and started it while connected to the network that will reliable cause the fork to crash and it didn't. So while I still suspect this is networking related upstream/official build doesn't seem to be impacted so may well be build options related.

blu3id commented 9 months ago

Managed to capture some logs but at first glance there doesn't seem to be much more detail than in @Sylonin original report. Sharing here in case it is useful: https://gist.github.com/blu3id/7a4dda385fa1b8fb37beb483ccbfd87b

makew0rld commented 9 months ago

I'm also seeing pretty constant crashes, even after clearing app data and cache and re-importing.

JanickGers85 commented 9 months ago

Same here: the app crashes, the native one no, with the very same settings backup restored.

Edit: tried with v1.26.0.0 and latest v1.26.1.0, both version.

Here app logs https://pastebin.com/TmWSacRe

Catfriend1 commented 9 months ago

To sum up:

it does not seem to be caused by the NDK or Go version in use (they were bumped and problem still exists)
it does not affect all devices - mine work fine without a crash observed
if some of you had a crash-free experience exporting the config to the official app, it is not a config issue triggering the crashes.

What about backing up your data, let it sync up to 100% in-sync state (if this is still possible?!) and then go to settings > ... > reset database? Does a full rescan and db rebuild help here to avoid the crash? At least worth a try, because if you "export and import to switch to official" your database is not imported (the official app doesn't have this feature in the code).

Catfriend1 commented 9 months ago

@JanickGers85

Same here: the app crashes, the native one no, with the very same settings backup restored.

Edit: tried with v1.26.0.0 and latest v1.26.1.0, both version.

Here app logs https://pastebin.com/TmWSacRe

What is before those lines? I've heard from the upstream maintainers they are the most important ones to see where the crash came from. You can capture them by "adb logcat v" (Just google Android Debugging via USB).


01:22:44W/SyncthingNativeCode   net/fd_unix.go:117 +0x2cc
01:22:44W/SyncthingNativeCode 
01:22:44W/SyncthingNativeCode goroutine 172 [select]:
01:22:44W/SyncthingNativeCode github.com/syncthing/syncthing/lib/dialer.dialTwicePreferFirst.func2()
01:22:44W/SyncthingNativeCode   github.com/syncthing/syncthing/lib/dialer/public.go:162 +0xec
01:22:44W/SyncthingNativeCode created by github.com/syncthing/syncthing/lib/dialer.dialTwicePreferFirst in goroutine 194
01:22:44W/SyncthingNativeCode   github.com/syncthing/syncthing/lib/dialer/public.go:161 +0x3c4
01:22:44W/SyncthingNativeCode 
01:22:44W/SyncthingNativeCode goroutine 171 [IO wait]:```

Soundtoxin commented 9 months ago

Just hit this as well. Is there a way to downgrade to the previous version on F-Droid? I'm hesitant to uninstall and then install an old version since it may discard settings. I see the previous versions listed but don't see a way to install. I exported my config just in case, but only from the latest broken version, so who knows if it's even good.

GBT7 commented 9 months ago

@Soundtoxin Others in this thread sound like they've had success in exporting to the official/non-forked app.

And for whatever it's worth I am experiencing this same bug.

Exporting, downgrading, then importing again worked for me.

Catfriend1 commented 9 months ago

I still don't know where the crash exactly happens in the native code to forward the ticket to the go code devs of SyncthingNative. What a tricky case, it doesn't crash on my phone nor emulator.

There was a different problem those days when Android auto-revoked the notification permission on the app. This has been fixed in the latest 1.26.1.0 release where the app politely asks again for the permission.

Did anyone of you try different release channels? https://github.com/Catfriend1/syncthing-android/releases/tag/v1.26.0.2

We have 3 of them. Do all three end in the same crash?

libreinator commented 9 months ago

I have the same issue (similar logs) and I found it does not crash when the other devices are offline at the time syncthing-fork on android starts.

This is what I have tried so far:

start syncthing on desktop, then start syncthing-fork on android: syncthing-fork crashes (I believe the crash occurs as soon as both instances connect with each other)
stop syncthing on desktop, start syncthing-fork on android, then start syncthing on desktop: syncthing-fork does not crash and the files sync without any problem

I'm on Syncthing-Fork v1.26.0.2 with syncthing v1.26.0 on android (installed from f-droid) and syncthing v1.26.1 on arch linux.

Catfriend1 commented 9 months ago

This is an interesting observation. My setup connects Android with an always on server. Should it then crash when turning on the app? From reading your post yes, but indeed, it does not on my phone.

libreinator commented 9 months ago

This is an interesting observation. My setup connects Android with an always on server. Should it then crash when turning on the app? From reading your post yes, but indeed, it does not on my phone.

I just found out syncthing-fork does not crash when the desktop and phone are connected to different networks and they communicate using relays. Maybe your server is not on the local network and you're connected using relays or some other way?

Catfriend1 commented 9 months ago

@libreinator I have turned off all discovery methods on my phone (4 disabled check boxes). Then I configured my servers IP in the device entry (Android). The server sits on the same network, connected via Lan cable, my phone uses Wifi.

Update: I have now tried enabling the 4 discovery + connection methods, disabled static IP and it did not crash.

Catfriend1 commented 9 months ago

What about doing adb logcat v > logfile.log and grabbing the whole crash report from the beginning?

codemole commented 9 months ago

FWIW, I can confirm @libreinator's observation about importance of instance start order. Some additional notes:

the connection type for me is TCP WAN, so I'm guessing no relay is involved and thus it's not a requirement.
syncthing-fork connects to two other instances for me. All have distinct syncthing versions. I only need to have the v1.24.0 (FreeBSD) instance shut down for the ST-Fork to start correctly. The other instance is v1.25.0 (Linux) and it's running state doesn't affect the Android app (v1.26.0.2)

Edit: in case it's relevant, both of my instances share different folders with ST-Fork, they're not syncing a common resource.

Catfriend1 commented 9 months ago

@codemole could you please try to upgrade the other nodes to v1.26.1 ? Since 1.25+ upstream has added a multiple connections handling to Syncthing. Maybe there' s some version incompatibility bug triggering the crash?

My nodes all have 1.26+.

Ref: https://docs.syncthing.net/v1.26.0/advanced/device-numconnections.html

codemole commented 9 months ago

@Catfriend1, I can update the Linux client, it didn't seem to change anything. FreeBSD is more complicated, since stable only has 1.24 in their repos. For the latest version I think I'd need to build syncthing myself. Not a FreeBSD pro though, might be some other way.

Another observation, if I put the shared folder on Pause, the v1.24 instance can remain running and ST-Fork will start just fine. I can even resume it afterwards and the sync will continue without issue.

Catfriend1 commented 9 months ago

Just as a sidenote: I think we could benefit from taking this issue to the Syncthing forum. My feeling says this is Syncthing(native) related and has nothing to do with the wrapper I'm maintaining here. Plus, the crash logs above are also native parts of Syncthing spitting them out.

codemole commented 9 months ago

@Catfriend1, fair enough. What would be the best place to report this to syncthing guys? Just straight into the forum or github is better?

Catfriend1 commented 9 months ago

Please capture the full native crash log and show it on the forum together with the pointer to the different versions.

codemole commented 9 months ago

As a final note, I've tried syncing the v1.24 instance to v1.26.1 (Linux) and v1.26.1 (Windows) and neither to crashes upon start. The Android version seems to be the only one affected.

Catfriend1 commented 9 months ago

Very strange, but not my code :(

blu3id commented 9 months ago

Forum post: https://forum.syncthing.net/t/syncthing-1-26-crashing-on-android-in-syncthing-fork/21115

Also for reference I have now managed to capture the Logcat output of the crash that has slightly more detail: https://gist.github.com/blu3id/cddab7352ce6586e3ba999e346795206

Catfriend1 commented 9 months ago

@blu3id The log is perfect ! That's what we need so upstream devs can tell us where it crashes and hopefully give advice what do to next. It seems Syncthing's indexHandler has a Problem.

0xCA commented 9 months ago

Great to see that there is an issue and some progress on this problem. I updated today and have the same issue with exit code 2 notification and no logs.
Can't really say much, unfortunately:

It happens (mostly?) with a scheduled mode. ~~When I force start it, it runs ok.~~ It also crashes shortly after the manual start.
I'm still not sure if it happens every hour, or some launches are ok.
The phone mainly syncs with two always-on servers, both are in the same local network as the phone itself.
It also happened on mobile network just now. There is a failed transfer of a newly taken photo.
It starts ok when the other peer is offline at the moment.

Catfriend1 commented 9 months ago

Yeah, upstream did a fix on that. Will need to wait for 1.27 to be released including the fix.

Catfriend1 commented 9 months ago

Please report back if https://github.com/Catfriend1/syncthing-android/releases/download/v1.27.0.0/com.github.catfriend1.syncthingandroid_github_v1.27.0.0_7302b491.apk works without crashing. It contains the latest upstream commits from https://github.com/syncthing/syncthing/pull/9235 .

0xCA commented 9 months ago

Please report back if https://github.com/Catfriend1/syncthing-android/releases/download/v1.27.0.0/com.github.catfriend1.syncthingandroid_github_v1.27.0.0_7302b491.apk works without crashing. It contains the latest upstream commits from syncthing/syncthing#9235 .

Sorry for the late reply. It worked exactly once, I was finally able to sync everything that was not synced since the code 2 appeared. Then it had exactly the same issue.

GBT7 commented 8 months ago

1.27.0.1 has been working for me, albeit inconsistently, for the past while. Every once in a while it would crash with error code 2, and I just hit exit and reopened it and it was fine.

Now, even though seemingly nothing has changed, it has stopped working for me altogether, and just crashes with error 2 again, like it did before on 1.26

Soundtoxin commented 8 months ago

I still get crashes / syncthing going to sleep pretty often on 1.27.0.1 but occasionally it must manage to sync because I do get pictures I took appearing on my PC most of the time.

Catfriend1 commented 7 months ago

Try the latest 1.27.2.1 version, please.

GBT7 commented 7 months ago

127.2.1 is working so far but I can't guarantee its stability, as while 127.0.1 worked to start with, it also crashed unpredictably. For now I'll say it's probably hopefully fine, and then I'll leave another comment if it starts crashing again.

Catfriend1 / syncthing-android