search

scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
57 stars 95 forks source link

curl fails on Connection reset by peer when downloading manager repo list #9314

Closed soyacz closed 1 day ago

soyacz commented 1 day ago

Packages

Scylla version: 6.3.0~dev-20241115.5bc03da0c4c1 with build-id 616f46f4e7794bef8367877dfce801c0ddae5371

Kernel Version: 6.8.0-1018-aws

Issue description

Although we have --retry set for curl command it still may fail quickly and not retry at all.

Command: 'sudo curl -o /etc/apt/sources.list.d/scylla-manager.list -L https://downloads.scylladb.com/deb/debian/scylladb-manager-3.4.list --retry 5 --retry-max-time 300'
Exit code: 35
Stdout:
Stderr:
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (35) Send failure: Connection reset by peer

Possibly it's an issue with curl: https://github.com/curl/curl/issues/11124

Impact

Failed test, need to rerun

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Cluster size: 3 nodes (i3en.2xlarge)

Scylla Nodes used in this run:

OS / Image: ami-0f5e612ec05c7b727 (aws: undefined_region)

Test: scylla-master-perf-regression-latency-650gb-with-nemesis Test id: c0781a3c-b29f-4328-92c0-ae265063affb Test name: scylla-master/perf-regression/scylla-master-perf-regression-latency-650gb-with-nemesis Test method: performance_regression_test.PerformanceRegressionTest.test_latency_write_with_nemesis Test config file(s):

Logs and commands - Restore Monitor Stack command: `$ hydra investigate show-monitor c0781a3c-b29f-4328-92c0-ae265063affb` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=c0781a3c-b29f-4328-92c0-ae265063affb) - Show all stored logs command: `$ hydra investigate show-logs c0781a3c-b29f-4328-92c0-ae265063affb` ## Logs: - **db-cluster-c0781a3c.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/c0781a3c-b29f-4328-92c0-ae265063affb/20241120_061924/db-cluster-c0781a3c.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/c0781a3c-b29f-4328-92c0-ae265063affb/20241120_061924/db-cluster-c0781a3c.tar.gz) - **sct-runner-events-c0781a3c.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/c0781a3c-b29f-4328-92c0-ae265063affb/20241120_061924/sct-runner-events-c0781a3c.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/c0781a3c-b29f-4328-92c0-ae265063affb/20241120_061924/sct-runner-events-c0781a3c.tar.gz) - **sct-c0781a3c.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/c0781a3c-b29f-4328-92c0-ae265063affb/20241120_061924/sct-c0781a3c.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/c0781a3c-b29f-4328-92c0-ae265063affb/20241120_061924/sct-c0781a3c.log.tar.gz) - **loader-set-c0781a3c.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/c0781a3c-b29f-4328-92c0-ae265063affb/20241120_061924/loader-set-c0781a3c.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/c0781a3c-b29f-4328-92c0-ae265063affb/20241120_061924/loader-set-c0781a3c.tar.gz) - **monitor-set-c0781a3c.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/c0781a3c-b29f-4328-92c0-ae265063affb/20241120_061924/monitor-set-c0781a3c.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/c0781a3c-b29f-4328-92c0-ae265063affb/20241120_061924/monitor-set-c0781a3c.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/scylla-master/job/perf-regression/job/scylla-master-perf-regression-latency-650gb-with-nemesis/29/) [Argus](https://argus.scylladb.com/test/4f088858-cbf2-4984-9d44-d319c207f521/runs?additionalRuns[]=c0781a3c-b29f-4328-92c0-ae265063affb)