search

scylladb / scylla-ccm

Cassandra Cluster Manager, modified for Scylla
Apache License 2.0
20 stars 64 forks source link

pin down urllib3<2 since botocore doesn't support it #453

Closed fruch closed 1 year ago

fruch commented 1 year ago

Look like new version of requests is pulling newer versions of urllib3 which are not comptible with botocore.

Ref: https://github.com/boto/botocore/issues/2926 Ref: https://github.com/psf/requests/issues/6443 Ref: https://github.com/urllib3/urllib3/issues/2168

tchaikov commented 1 year ago

created #454 so we can undo this once the related issues on upstream (botocore and requests) are addressed.

tchaikov commented 1 year ago

the test failed:

2023-05-09 14:35:02 INFO     ['./ccm', 'create', 'CCMCluster-cassandra', '-v', '3.11.4', '-n', '1']
2023-05-09 14:35:02 DEBUG    [stdout] Current cluster is now: CCMCluster-cassandra
2023-05-09 14:35:02 DEBUG    [stderr] 
2023-05-09 14:35:02 INFO     ['./ccm', 'updateconf', 'read_request_timeout_in_ms:[100](https://github.com/scylladb/scylla-ccm/actions/runs/4926279591/jobs/8803175292?pr=453#step:9:101)00', 'range_request_timeout_in_ms:10000', 'write_request_timeout_in_ms:10000', 'truncate_request_timeout_in_ms:10000', 'request_timeout_in_ms:10000']
2023-05-09 14:35:03 DEBUG    [stdout] 
2023-05-09 14:35:03 DEBUG    [stderr] 
2023-05-09 14:35:03 INFO     ['./ccm', 'start', '--wait-for-binary-proto']
2023-05-09 14:35:03 DEBUG    [stdout] [node1 ERROR] Unable to find java executable. Check JAVA_HOME and PATH environment variables.
2023-05-09 14:35:03 DEBUG    [stderr] Error starting nodes, see above for details (you can use --verbose for more information)
Error: -09 14:35:03 ERROR    [ERROR] Error starting nodes, see above for details (you can use --verbose for more information)
fruch commented 1 year ago

created #454 so we can undo this once the related issues on upstream (botocore and requests) are addressed.

Thanks, you are correct we shouldn't leave forever

fruch commented 1 year ago

the test failed:

2023-05-09 14:35:02 INFO     ['./ccm', 'create', 'CCMCluster-cassandra', '-v', '3.11.4', '-n', '1']
2023-05-09 14:35:02 DEBUG    [stdout] Current cluster is now: CCMCluster-cassandra
2023-05-09 14:35:02 DEBUG    [stderr] 
2023-05-09 14:35:02 INFO     ['./ccm', 'updateconf', 'read_request_timeout_in_ms:[100](https://github.com/scylladb/scylla-ccm/actions/runs/4926279591/jobs/8803175292?pr=453#step:9:101)00', 'range_request_timeout_in_ms:10000', 'write_request_timeout_in_ms:10000', 'truncate_request_timeout_in_ms:10000', 'request_timeout_in_ms:10000']
2023-05-09 14:35:03 DEBUG    [stdout] 
2023-05-09 14:35:03 DEBUG    [stderr] 
2023-05-09 14:35:03 INFO     ['./ccm', 'start', '--wait-for-binary-proto']
2023-05-09 14:35:03 DEBUG    [stdout] [node1 ERROR] Unable to find java executable. Check JAVA_HOME and PATH environment variables.
2023-05-09 14:35:03 DEBUG    [stderr] Error starting nodes, see above for details (you can use --verbose for more information)
Error: -09 14:35:03 ERROR    [ERROR] Error starting nodes, see above for details (you can use --verbose for more information)

Yeah, I didn't have time to look at it.

Doesn't seem to be related to the installation issue, maybe to the change of using java8, but still I would expect it to fail more than one test (or we are stopping at first failure 😭)

tchaikov commented 1 year ago

the test failed:

2023-05-09 14:35:02 INFO     ['./ccm', 'create', 'CCMCluster-cassandra', '-v', '3.11.4', '-n', '1']
2023-05-09 14:35:02 DEBUG    [stdout] Current cluster is now: CCMCluster-cassandra
2023-05-09 14:35:02 DEBUG    [stderr] 
2023-05-09 14:35:02 INFO     ['./ccm', 'updateconf', 'read_request_timeout_in_ms:[100](https://github.com/scylladb/scylla-ccm/actions/runs/4926279591/jobs/8803175292?pr=453#step:9:101)00', 'range_request_timeout_in_ms:10000', 'write_request_timeout_in_ms:10000', 'truncate_request_timeout_in_ms:10000', 'request_timeout_in_ms:10000']
2023-05-09 14:35:03 DEBUG    [stdout] 
2023-05-09 14:35:03 DEBUG    [stderr] 
2023-05-09 14:35:03 INFO     ['./ccm', 'start', '--wait-for-binary-proto']
2023-05-09 14:35:03 DEBUG    [stdout] [node1 ERROR] Unable to find java executable. Check JAVA_HOME and PATH environment variables.
2023-05-09 14:35:03 DEBUG    [stderr] Error starting nodes, see above for details (you can use --verbose for more information)
Error: -09 14:35:03 ERROR    [ERROR] Error starting nodes, see above for details (you can use --verbose for more information)

Yeah, I didn't have time to look at it.

Doesn't seem to be related to the installation issue, maybe to the change of using java8, but still I would expect it to fail more than one test (or we are stopping at first failure sob)

hi @fruch , is it possible that we fail to detect the java runtime installed by https://github.com/scylladb/scylla-ccm/blob/66e9b4c28f7e72a27acfec6928233cb767f461e5/.github/workflows/integration-tests.yml#L33 ? as in 66e9b4c28f7e72a27acfec6928233cb767f461e5, we only check for the well-known directories. which are the directories used by the debian / fedora packages. but the workflow of "setup-java" installs the runtime into somewhere else: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.362-9/x64. see https://github.com/scylladb/scylla-ccm/actions/runs/4926279591/jobs/8803175292#step:5:30 . shall we add this directory to the list of https://github.com/scylladb/scylla-ccm/blob/66e9b4c28f7e72a27acfec6928233cb767f461e5/ccmlib/common.py#L254 ?

fruch commented 1 year ago

the test failed:

2023-05-09 14:35:02 INFO     ['./ccm', 'create', 'CCMCluster-cassandra', '-v', '3.11.4', '-n', '1']
2023-05-09 14:35:02 DEBUG    [stdout] Current cluster is now: CCMCluster-cassandra
2023-05-09 14:35:02 DEBUG    [stderr] 
2023-05-09 14:35:02 INFO     ['./ccm', 'updateconf', 'read_request_timeout_in_ms:[100](https://github.com/scylladb/scylla-ccm/actions/runs/4926279591/jobs/8803175292?pr=453#step:9:101)00', 'range_request_timeout_in_ms:10000', 'write_request_timeout_in_ms:10000', 'truncate_request_timeout_in_ms:10000', 'request_timeout_in_ms:10000']
2023-05-09 14:35:03 DEBUG    [stdout] 
2023-05-09 14:35:03 DEBUG    [stderr] 
2023-05-09 14:35:03 INFO     ['./ccm', 'start', '--wait-for-binary-proto']
2023-05-09 14:35:03 DEBUG    [stdout] [node1 ERROR] Unable to find java executable. Check JAVA_HOME and PATH environment variables.
2023-05-09 14:35:03 DEBUG    [stderr] Error starting nodes, see above for details (you can use --verbose for more information)
Error: -09 14:35:03 ERROR    [ERROR] Error starting nodes, see above for details (you can use --verbose for more information)

Yeah, I didn't have time to look at it.

Doesn't seem to be related to the installation issue, maybe to the change of using java8, but still I would expect it to fail more than one test (or we are stopping at first failure sob)

hi @fruch , is it possible that we fail to detect the java runtime installed by https://github.com/scylladb/scylla-ccm/blob/66e9b4c28f7e72a27acfec6928233cb767f461e5/.github/workflows/integration-tests.yml#L33 ? as in 66e9b4c28f7e72a27acfec6928233cb767f461e5, we only check for the well-known directories. which are the directories used by the debian / fedora packages. but the workflow of "setup-java" installs the runtime into somewhere else: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.362-9/x64. see https://github.com/scylladb/scylla-ccm/actions/runs/4926279591/jobs/8803175292#step:5:30 . shall we add this directory to the list of https://github.com/scylladb/scylla-ccm/blob/66e9b4c28f7e72a27acfec6928233cb767f461e5/ccmlib/common.py#L254 ?

The failing case trys to run cassandra, not scylla

fruch commented 1 year ago

the test failed:

2023-05-09 14:35:02 INFO     ['./ccm', 'create', 'CCMCluster-cassandra', '-v', '3.11.4', '-n', '1']
2023-05-09 14:35:02 DEBUG    [stdout] Current cluster is now: CCMCluster-cassandra
2023-05-09 14:35:02 DEBUG    [stderr] 
2023-05-09 14:35:02 INFO     ['./ccm', 'updateconf', 'read_request_timeout_in_ms:[100](https://github.com/scylladb/scylla-ccm/actions/runs/4926279591/jobs/8803175292?pr=453#step:9:101)00', 'range_request_timeout_in_ms:10000', 'write_request_timeout_in_ms:10000', 'truncate_request_timeout_in_ms:10000', 'request_timeout_in_ms:10000']
2023-05-09 14:35:03 DEBUG    [stdout] 
2023-05-09 14:35:03 DEBUG    [stderr] 
2023-05-09 14:35:03 INFO     ['./ccm', 'start', '--wait-for-binary-proto']
2023-05-09 14:35:03 DEBUG    [stdout] [node1 ERROR] Unable to find java executable. Check JAVA_HOME and PATH environment variables.
2023-05-09 14:35:03 DEBUG    [stderr] Error starting nodes, see above for details (you can use --verbose for more information)
Error: -09 14:35:03 ERROR    [ERROR] Error starting nodes, see above for details (you can use --verbose for more information)

Yeah, I didn't have time to look at it.

Doesn't seem to be related to the installation issue, maybe to the change of using java8, but still I would expect it to fail more than one test (or we are stopping at first failure sob)

hi @fruch , is it possible that we fail to detect the java runtime installed by https://github.com/scylladb/scylla-ccm/blob/66e9b4c28f7e72a27acfec6928233cb767f461e5/.github/workflows/integration-tests.yml#L33 ? as in 66e9b4c28f7e72a27acfec6928233cb767f461e5, we only check for the well-known directories. which are the directories used by the debian / fedora packages. but the workflow of "setup-java" installs the runtime into somewhere else: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.362-9/x64. see https://github.com/scylladb/scylla-ccm/actions/runs/4926279591/jobs/8803175292#step:5:30 . shall we add this directory to the list of https://github.com/scylladb/scylla-ccm/blob/66e9b4c28f7e72a27acfec6928233cb767f461e5/ccmlib/common.py#L254 ?

The failing case trys to run cassandra, not scylla

And I've forgotten you convinced me to not use alternatives command line...

Yeah that is the root cause, and I didn't notice it.

fruch commented 1 year ago

I decided to merge this first, and then handle the CI breakage