Open smart7324 opened 1 year ago
I also had this issue today and I could only fix it with a database maintenance run (command below). my environment infos:
root@Nextcloud:# apache2 -v
Server version: Apache/2.4.41 (Ubuntu)
Server built: 2023-03-08T17:32:54
root@Nextcloud:# php --version
PHP 8.1.17 (cli) (built: Mar 16 2023 14:38:17) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.17, Copyright (c) Zend Technologies
with Zend OPcache v8.1.17, Copyright (c), by Zend Technologies
root@Nextcloud:# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal
root@Nextcloud:# cat /var/www/nextcloud/version.php
$OC_Version = array(26,0,0,11);
$OC_VersionString = '26.0.0';
$OC_Edition = '';
$OC_Channel = 'stable';
$OC_VersionCanBeUpgradedFrom = array (
'nextcloud' =>
array (
'25.0' => true,
'26.0' => true,
),
'owncloud' =>
array (
'10.11' => true,
),
);
$OC_Build = '2023-03-21T09:23:03+00:00 62cfd3b4c9ff4d8cdbbe6dcc8b63a1085bb94e3d';
$vendor = 'nextcloud';
(PostgreSQL) 12.14 (Ubuntu 12.14-0ubuntu0.20.04.1)
How I fix the loop: alias FIX_LOOP='cd /var/www/nextcloud && sudo -u www-data php ./occ maintenance:repair' and then wait 30 minutes for the rate limiting to cool down.
iOS is the latest 16.04 (20E247)
here is an excerpt from my logs when I tried to log in with my admin account:
Please answer to this if I should provide more info
I gave it a try, but this didn't work for me. Same issue. It also happened to me on a clean new install. So we definitely need help here. At this time NC is completely unusable on Safari no matter what apple device...
(moved from #33919)
This problem does not seem to have been solved in v26.0.0.11 - even though https://github.com/nextcloud/server/pull/35419 was merged - seeing how as I'm currently unable to login using Firefox/Android on a device which had a single tab open yesterday. Deleting site data does not change this, nor does running occ maintenance:repair.
I can login using a different browser but not with Firefox, all I get is an empty page showing the site logo and the footer - there is no error message but no login/password request either.
This does not work:
occ maintenance:repair
This does work:
The error message in the log is the one which has been shown countless times already: Tried to log in "username" but could not verify token
:
{"reqId":"aupvuif3Msicz86FxhbY","level":1,"time":"April 06, 2023 06:06:04","remoteAddr":"192.168.9.2","user":"--","app":"core","method":"GET","url":"/login","message":"Tried to log in frank but could not verify token","userAgent":"Mozilla/5.0 (Android 9; Mobile; rv:109.0) Gecko/112.0 Firefox/112.0","version":"26.0.0.11","data":{"app":"core"}}
{"reqId":"q8JEudtB0oT3gfNqLYye","level":1,"time":"April 06, 2023 06:06:04","remoteAddr":"192.168.9.2","user":"--","app":"core","method":"GET","url":"/apps/theming/image/background?v=27","message":"Tried to log in frank but could not verify token","userAgent":"Mozilla/5.0 (Android 9; Mobile; rv:109.0) Gecko/112.0 Firefox/112.0","version":"26.0.0.11","data":{"app":"core"}}
The really annoying thing is that I do not get a chance to login at all since the login/password request does not show up - only the site logo and the footer on an otherwise empty page.
Another thing which does work:
This way I do get a login/password request. It seems that Firefox' Clear cookies and site data is not enough to actually clear everything related to the page.
Update: This now happens multiple times per day, which is a lot worse than it was before updating to NC 26
This is really a serious issue. Right now, I can't use NC with Safari... I am getting logged out every page refresh, so it's completely unusable. Are there any updates? :)
Some of the new issues could be related to a safari bug: https://bugs.webkit.org/show_bug.cgi?id=255524
Maybe iOS 17 brings a change or the root cause is found somewhere else, either way I hope this will soon be solved because sometimes I can’t log into my NC for days
Seems to be fixed for me with iOS 16.5 and macOS 13.4.
Updated a few days ago and for me it seems to be just as bad as before. Haven’t replied earlier because I wanted to gather some data.
I'm no longer experiencing any issues, also on NC 27.0.0. We can close here.
I updated ~12h ago and just had this issue reappear. Setup is NC in a Ubuntu 20.04 LXC run on Proxmox 7.4-3.
Kernel: 5.15.107-2-pve Ubuntu: Ubuntu 20.04.6 LTS PHP: PHP 8.1.17 (cli) (built: Mar 16 2023 14:38:17) (NTS) Copyright (c) The PHP Group Zend Engine v4.1.17, Copyright (c) Zend Technologies with Zend OPcache v8.1.17, Copyright (c), by Zend Technologies Apache Server version: Apache/2.4.41 (Ubuntu) NC version: 27.0.0.8
Reverse Proxy: Nginx-Proxy-Manager RP version: 2.10.3
Client: iOS 16.5 - Safari
Did you do anything else than simply updating NC to fix this? It is getting more and more frustrating to use NC since I can't access it ~50% of the time I need to
Okay hm, I also didn’t experience the bug on NC 26 since iOS 16.5… I did not change anything, but it’s just working.
So I reopen this issue for you.
Thanks a lot
that’s weird… Do you think my or any reverse proxy could be an issue since my TLS connection is terminated there? I can’t really think of anything else that could cause this in my setup
Honestly I don't think so, as I also had this issue and don't have a reverse proxy. I also did some debugging, but I haven't found anything... Is it working with other browsers for you?
I rarely use other devices to access my NC, but I had a few situations where this error occurred with my employer provided laptop. On my Debian laptop with Firefox, I had a kinda similar error where i was kinda logged in, but was repeatedly kicked out of NC with the error message in the browser „you are not logged in“. Even when I logged out and back in, this error would persist. I blamed a weird cookie issue and just let it be.
Another possibility that just came to mind: I’m basically always connected to my VPN server at home, which gives my phone, my Mac and NC the same public IP address. Could this be an issue?
(Just for clarification: the issue for me is almost exclusively in iOS, macOS only caused this error once since NC 24 plus the rare occurrences on windows or Linux with Firefox)
Hm very interesting… Sorry, but I don’t know if your ip can be a source of the issue. Maybe someone else can help?
Yeah me neither, I’m just throwing guesses at the wall here to see what sticks haha
To anyone reading this: all suggestions are welcome
Btw, I played around on my work phone (also iPhone and safari) and was able to provoke the error relatively quickly with two open tabs and some reloads/NC-App switching The error occurred but I was not logged out however, that also happens a lot
Hi, We have also been struggling with this problem for about two months. Even an update to version 27 has not brought any improvement. On the contrary, we have the feeling that the bug has increased significantly in recent weeks. In the meantime, our power users can no longer use Nextcloud on certain days.
Even deleting the cookies only helps to a limited extent. After deleting them, they are simply set again and the problem is back.
Our Nextcloud is connected to a very large LDAP directory of our institution. We have about 70 active users (once a week) and about 20 power users (every day, several hours). We are thinking that a connection to the LDAP could be increasing the problem, but probably the trigger is somewhere else.
Access is via a reverse proxy (nginx). There, too, we have already changed some settings for header modification, but without any noticeable effect. In addition, the token errors are occurring more and more frequently with reports of a brute force attack. For this reason, we have to deactivate the brute force detection in the meantime in order not to be locked out all the time. Apparently, Nextcloud counts every expired cookie as a failed login.
It is frustrating. The error pattern is so varied that it is difficult for us to identify the origin of the error.
Yes, that’s also my experience And that’s on a very small instance with only me as a user. what client devices do your users use? Maybe we have an overlap and can help narrow down the scope for the devs
Yes, great idea. We have tested our way through various browsers: Chrome, Edge, Firefox and Opera. The problem is the same everywhere. Most users use Windows machines. However, the problem also occurs with our iOS, iPadOS and Android users. Also with Safari, Brave, Opera, Chrome... We haven't had a chance to test it on MacOS yet.
Sometimes our users are even logged out of the Nextcloud apps (iOS+Android). Talk in particular (which we use a lot).
We initially thought there was a connection with the use of Nextcloud calendars via CardDAV or in connection with app passwords, which a handful of our users are using. However, we could not find any further evidence for this.
yeah, that's pretty much my device variety, just at a way smaller scale. I have an iPhone with iOS 16.5.1(c), an iMac with macOS 13.2.1 (both most recent Safari), Laptop with Debian 11 + Firefox and a Windows laptop from my employer with Windows 10, now windows 11. All devices had the issue with always the same symptoms that have been described here multiple times.
Just now I updated my NC instance to 27.0.1 and I could not immediately provoke the error. I'll report back in a few days if the issue surfaces again.
Edit: just finished reading through the changelog and at the very bottom, one point stood out "Send CSRF token in rawStat": https://github.com/nextcloud/viewer/pull/1798 @MrRies MAYBE, just maybe, our issue could be solved with this. If possible in your large setup, try updating to 27.0.1 and see if it behaves differently.
update after a few days:
the message "tried to log in $USER but could not verify token" still appears when tabs get reactivated (browser opened after some Time on iOS e.g.) or occasionally when you have multiple tabs open, but no issues as in kicked out, rate limited and unable to log in.
So from my user+admin perspective, the symptoms are mended but the cause still persists in some form. Since it's on multiple OSs and different browsers on these and NC changed behaviour after an update, I assume the issue is still somewhere in NC.
I'll post another update in like 2 weeks or when I am facing the initial issues again.
anything new in your setup? @MrRies
Update from my setup: It's happening rarely, but still relatively predictable. For example, a tab has been open for a few days but not active for most of this time, then another gets opened. This, in my case, triggers the spew of "tried to log in $USER but could not verify token" messages (see screenshot).
Even after the brute force cooldown and a successful login, I get these log messages but without a kick.
Would be great if anyone from the team took a look in here to tell us what other infos to provide. My setup did not change from this message.
I finally lost my patience and tried to assign NC a "unique" domain. I have quite a few subdomains and cnames under the domain that NC runs on, which lead me to the hail mary to move it to its own domain.
I'll report back with my findings in about a week or earlier if it's the same as before.
Never mind, my loose theory was proven wrong unsurprisingly. As soon as I have one „older“ tab and open another, I get kicked out immediately.
It seems NC is messing up the cookies with different tabs, like it tries to verify tab1 with cookie 2 and thus throws errors.
I just don’t know how to troubleshoot this or what information to provide from where
Sorry for the long absence. We also updated to 27.0.1, but this did not lead to any change.
Our access is via a subdomain using a reverse proxy (nginx). Out of sheer desperation, we tried deactivating the settings "Block Common Exploits" and "Cache assets" in the nginx proxy manager for this subdomain. This resulted in the error message appearing just as often, but it no longer has any consequences for the users. We can currently work in several tabs again without any problems. However, the problem is so random that we are not sure whether this was really the cause of the problem or pure coincidence.
Sorry for the long absence. We also updated to 27.0.1, but this did not lead to any change.
Our access is via a subdomain using a reverse proxy (nginx). Out of sheer desperation, we tried deactivating the settings "Block Common Exploits" and "Cache assets" in the nginx proxy manager for this subdomain. This resulted in the error message appearing just as often, but it no longer has any consequences for the users. We can currently work in several tabs again without any problems. However, the problem is so random that we are not sure whether this was really the cause of the problem or pure coincidence.
that's a valuable tip, I'll try to deactivate that as well
We are seeing the same problem with NC 27.0.1 on PHP 8.1
Log gets flooded every some seconds with "Tried to log in $USER but could not verify token". Only very few users are affected by it.
I applied the "trick" @MrRies suggested (deactivate caching, web sockets and 'block common exploits' in Nginx proxy manager) and it seems to remedy the symptoms i.e. being kicked out. However I still have the messages in the log, but for some reason NC doesn't lock me out anymore.
So, it could be a (hopefully) temporary fix until the actual root issue is fixed.
Discovered https://github.com/spring-projects/spring-security/issues/11921 and it could be a similar issue. Nextcloud uses a token-based remember me mechanism. Two or more requests may race for the same database token.
Attempt on a simpler, hopefully more robust session handling: https://github.com/nextcloud/server/pull/40543
If someone is okay with experimenting on a production system see https://github.com/nextcloud/server/pull/40628.
Nextcloud 27.x patch can be fetched from https://github.com/nextcloud/server/commit/37eefca9f11eb4fe0ba6a662ad642565e6843bac.patch.
This will harden the token refresh logic and log more specific errors when refreshing tokens doesn't work.
This should give us valuable insights into the failing remember-me login.
I‘d be up to test this but I’m not sure how to apply this patch… Is there a guide in the documentation somewhere on how to do this?
And how do I get the "patch" command? Sorry, I've never seen that command before and looking for "ubuntu install patch GitHub" doesn't exactly yield the results I'm looking for haha
No worries. Let's wait for a tech-savvy person to test the patch and report findings based on logfile analysis.
ah nvm, I thought patch was a php script like occ. I applied the patch and I'm changing the log level to debug
will report back when I find anything
The only thing I could see immediately; I think the screenshot with the two tokens was a remnant from before the patch apply because I logged in from my work phone.
About the "No public access[...]" messages: do you have an idea what could cause this? DAV sounds like calendar or contacts.
exception trace:
{file=/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php, line=89, function=beforeMethod, class=Sabre\DAV\Auth\Plugin, type=->}, {file=/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php, line=456, function=emit, class=Sabre\DAV\Server, type=->}, {file=/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php, line=253, function=invokeMethod, class=Sabre\DAV\Server, type=->}, {file=/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php, line=321, function=start, class=Sabre\DAV\Server, type=->}, {file=/var/www/nextcloud/apps/dav/lib/Server.php, line=365, function=exec, class=Sabre\DAV\Server, type=->}, {file=/var/www/nextcloud/apps/dav/appinfo/v2/remote.php, line=35, function=exec, class=OCA\DAV\Server, type=->}, {file=/var/www/nextcloud/remote.php, line=172, args=[/var/www/nextcloud/apps/dav/appinfo/v2/remote.php], function=require_once}
Activated an already open tab after some time, this is what happens:
Interesting is that the log says I am not logged in, but there's nothing in the webUI that suggests that. I am not even kicked out and everything else loads normally
I think there are some requests that fail, other succeed. It's a bit random. Sometimes that ends your session, other times it can survive.
Thanks for the insights.
Sure, any time just ping me when you have anything new, I'm happy to test stuff
Addendum: I just opened a new NC tab on my phone (closed the ones from earlier after testing the patch) and had to log in again
it’s not new behavior, but I had to enter my username and PW twice and then the 2FA code before it let me in
idk when exactly I was kicked out but when I find the section in the logs I’ll post a screenshot
Double login is a known issue and related to lost sessions
Could you please apply https://github.com/nextcloud/server/commit/02591953bc488aa424f058035ad39fa7b3beb723.patch as well? It's an amendment to https://github.com/nextcloud/server/pull/40628 so that it logs the request that wins the race for the token.
Will do when I’m home
All-Messages-search-result-part-1.csv All-Messages-search-result-part-2.csv
I applied the patch and played around a bit (I redacted any tokens, IPs and domains). Is this format helpful for you? Should I test specific scenarios or filter for certain keywords? Because right now this is basically the full log after the patch apply and reboot on debug level.
The format is fine. Thanks!
First:
2023-09-27T22:25:56.086+02:00;Nextcloud;Remember-me token TOKEN1/some_extras?/ for root replaced by TOKEN2
Later:
2023-09-27T22:28:02.133+02:00;Nextcloud;Tried to log in root but could not find token TOKEN3 in database
2023-09-27T22:28:02.133+02:00;Nextcloud;Tried to log in root but could not find token TOKEN3 in database
2023-09-27T22:28:02.133+02:00;Nextcloud;Tried to log in root but could not find token TOKEN3 in database
2023-09-27T22:28:02.133+02:00;Nextcloud;Tried to log in root but could not find token TOKEN3 in database
But then also
2023-09-27T22:28:02.133+02:00;Nextcloud;Remember-me token TOKEN3 for root replaced by TOKEN4
so token2 is never used. token3 appears out of nowhere.
Did you have more than one browser or devices connected? e.g. desktop+phone.
Ah yes, my bad
I was just focused on provoking the error and didn't think about multiple devices. I just recreated the situation with just one device but the overall situation seems to be the same:
What I did: closed the NC app on my Mac and closed tabs on my work phone. Then after some minutes I opened Safari on my private phone, closed an inactive tab and opened a new one. Only when I opened the new tab, logs started appearing. Once the new tab loaded, I logged in (twice, as noted yesterday) and you see the logs above.
Edit: Due to the time stamps all reading the same, I just want to point out that the newest entry is on top and the oldest at the bottom
I have a new idea. What if the remember-me logic does its job but the concurrent requests cause the web session to be deleted from the database? that would also end a session. I'll prepare some more logging patches :weary:
⚠️ This issue respects the following points: ⚠️
Bug description
As soon as I open Nextcloud in a new tab, I get redirected to login page and have to login again. Then always the first login fails/nothing happens, so I have to login twice. I am seeing lots of "Tried to log in "user" but could not verify token" errors in log.
It is only happening on Safari (macOS, iOS, iPadOS), tried several versions, also did a clean install of Nextcloud 26 and still the same. Also tried with another user account on a different Mac.
At first I thought it could be related to #33919, but it doesn't seem to be the case. I really spent many hours in trying to get this fixed, but I have no clue, why it is not working.
Steps to reproduce
Expected behavior
The user should still be logged in and not be redirected to login page.
Installation method
Community Manual installation with Archive
Nextcloud Server version
26
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.1
Web server
Apache (supported)
Database engine version
MySQL
Is this bug present after an update or on a fresh install?
Fresh Nextcloud Server install
Are you using the Nextcloud Server Encryption module?
None
What user-backends are you using?
Configuration report
List of activated Apps
Nextcloud Signing status
No response
Nextcloud Logs
Additional info
No response