search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
27.32k stars 4.06k forks source link

[Bug]: Very slow access to apps.nextcloud.com/api/v1/apps.json for checking app updates #45162

Open hammeractual opened 6 months ago

hammeractual commented 6 months ago

⚠️ This issue respects the following points: ⚠️

Bug description

Often to always failed when checking for Apps Update caused by very slow access to https://apps.nextcloud.com/api/v1/apps.json Connection speed on the Client and on the Server seems normal, but when accessing https://apps.nextcloud.com/api/v1/apps.json either from the server or from the client is pretty slow (mostly failed to load) for json file to be loaded completely.

Steps to reproduce

  1. Login
  2. Go to Admin Page
  3. Overview Menu
  4. Never finish "Checking apps for compatible versions"

Expected behavior

Checking update for apps completed successfully.

Installation method

Community Manual installation with Archive

Nextcloud Server version

28

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated from a MINOR version (ex. 22.1 to 22.2)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

Configuration report

No response

List of activated Apps

No response

Nextcloud Signing status

No response

Nextcloud Logs

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 903254 out of 6408674 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json
    GET /ocs/v2.php/apps/updatenotification/api/v1/applist/28.0.5.1
    from x.x.x.x by e2cd37b6-6ebf-103d-84c2-addc165358a8 at 3 Mei 2024, 09.09.50

Additional info

No response

bigcat88 commented 6 months ago

This issue is related more to the appstore and related to infrastructure, imho.

Sometimes getting of json from "api/v1/apps.json" can fail(when appstore is under high load), but in 98-99% cases it should be fine.

hammeractual commented 6 months ago

This issue is related more to the appstore and related to infrastructure, imho.

Sometimes getting of json from "api/v1/apps.json" can fail(when appstore is under high load), but in 98-99% cases it should be fine.

Could you please tell the infrastructure division/department to check/troubleshoot the issue?

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

sdasdasda

bigcat88 commented 6 months ago

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

It works for me now and has worked since the very beginning of today. there was no peak load in the appstore today, there shouldn't have been any slowdowns

hammeractual commented 6 months ago

Since it can't load the apps.json file, I can't update the apps and also can't accessing the Appstore too to install apps. :(

It works for me now and has worked since the very beginning of today. there was no peak load in the appstore today, there shouldn't have been any slowdowns

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

joshtrichards commented 6 months ago

@hammeractual What does curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test from your Nextcloud Server's command line look like? Here's how it looks for me from a test instance:

www-data@ef86c134f021:/tmp$ curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.9M  100 13.9M    0     0  6545k      0  0:00:02  0:00:02 --:--:-- 6548k
www-data@ef86c134f021:/tmp$

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 903254 out of 6408674 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

40082 has a manual approach for doing so. Also it's tracking the feature request.

hammeractual commented 6 months ago

@hammeractual What does curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test from your Nextcloud Server's command line look like? Here's how it looks for me from a test instance:

www-data@ef86c134f021:/tmp$ curl https://apps.nextcloud.com/api/v1/apps.json -o apps.json-test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.9M  100 13.9M    0     0  6545k      0  0:00:02  0:00:02 --:--:-- 6548k
www-data@ef86c134f021:/tmp$

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 903254 out of 6408674 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json

How to increase the curl timeout in the nextcloud to fetch apps.json file completely? Do you have any clue?

40082 has a manual approach for doing so. Also it's tracking the feature request.

Thanks for the info

image Here is mine.

Meanwhile, the ping result is pretty good to that address. image

bigcat88 commented 6 months ago

Here is mine.

I was wrong. It seems that this is not a problem with the appstore code itself, since during high loads the appstore simply does not serve the file at all, but if it has already started serving, it does it quickly, because the file itself is given by NGINX(reverse proxy).

joshtrichards commented 6 months ago

Meanwhile, the ping result is pretty good to that address.

Something does not add up. Your throughput is incredibly slow relative to your latency. You're getting like 6558 B/s (!).

Can you sanity check that curl performance by downloading something from elsewhere:

Maybe curl https://github.com/nextcloud/server/archive/refs/tags/v28.0.5.zip -o /dev/null

P.S. Is your Server @ Hetzner?

hammeractual commented 6 months ago

Meanwhile, the ping result is pretty good to that address.

Something does not add up. Your throughput is incredibly slow relative to your latency. You're getting like 6558 B/s (!).

Can you sanity check that curl performance by downloading something from elsewhere:

Maybe curl https://github.com/nextcloud/server/archive/refs/tags/v28.0.5.zip -o /dev/null

P.S. Is your Server @ Hetzner?

image Here is mine

Quite fast while curl to another server/address.

hammeractual commented 5 months ago

Any updates guys? :'(

I've tried to increase timeout value in Fetcher.php to 1800 seconds, but no success.

[appstoreFetcher] Warning: Could not connect to appstore: cURL error 28: Operation timed out after 1800000 milliseconds with 3661824 out of 6421307 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json GET /settings/apps from x.x.x.x by e2cd37b6-6ebf-103d-84c2-addc165358a8 at 7 Mei 2024, 17.14.29

joshtrichards commented 5 months ago

Any updates guys? :'(

I think the challenge is we're having a hard time reproducing this and there doesn't seem be an explanation for why your environment is experiencing this.

It's tough to fix without a clear (or even suspected possible) cause. :-)

joshtrichards commented 5 months ago

P.S. Is your Server @ Hetzner?

You may have missed this question. I asked because your ping times are ~1-2 ms.

At the time I was taking a shot in the dark that maybe there is something unusual about Hetzner's internal client-to-client connectivity. Today I ran across a note in the Portainer Knowledge Base while trying to help out a user on the forums that suggests Hetzner's private networking uses an MTU of 1450. Got me thinking about this again.

the-djmaze commented 2 months ago

Not only that. Try to load https://apps.nextcloud.com/apps/mail It takes 1 minute and 13 seconds for TTFB!

Ok, it is hosted at the terrible Hetzner, but this is not only Hetzner at fault.