search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.85k stars 3.77k forks source link

Example-ORMs pulls files from 3rd party vendors on every CI run, and fails if upstream is unavailable #52342

Open knz opened 4 years ago

knz commented 4 years ago

The Java tests in the Examples-ORMs pull their dependencies from repo.maven.apache.org upon every run. The Python tests pull their dependencies from pipy servers. And so on.

This is happening on every run. The files are not cached

This is causing unnecessary ingress costs on our side, and expensive egress cost on the side of 3rd party.

Lately we've had an uptick of failures due to the upstream server refusing our requests. (I wouldn't blame them personally, from their perspective we're an impolite leecher with a missing/incorrect proxy config.)

This is really a variation of #51543, with the problem focused on Examples-ORMs instead of the cdc roachtests.

Jira issue: CRDB-3960

knz commented 4 years ago

cc @rafiss @jlinder

rafiss commented 4 years ago

Example failure:

https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_ExampleORMs/2146876?buildTab=log&focusLine=3&linesState=378

     > Could not resolve dom4j:dom4j:1.6.1.
        > Could not get resource 'https://repo.maven.apache.org/maven2/dom4j/dom4j/1.6.1/dom4j-1.6.1.pom'.
           > Could not GET 'https://repo.maven.apache.org/maven2/dom4j/dom4j/1.6.1/dom4j-1.6.1.pom'.
              > Connect to repo.maven.apache.org:443 [repo.maven.apache.org/199.232.64.215] failed: connect timed out
jbowens commented 4 years ago

Here's another:

https://teamcity.cockroachdb.com/viewLog.html?tab=buildLog&logTab=tree&filter=debug&expand=all&buildId=2178687&_focus=573

[11:13:39][Step 2/3] :compileJava FAILED
[11:13:39][Step 2/3] 
[11:13:39][Step 2/3] FAILURE: Build failed with an exception.
[11:13:39][Step 2/3] 
[11:13:39][Step 2/3] * What went wrong:
[11:13:39][Step 2/3] Could not resolve all files for configuration ':compileClasspath'.
[11:13:39][Step 2/3] > Could not resolve com.fasterxml:classmate:1.3.0.
[11:13:39][Step 2/3]   Required by:
[11:13:39][Step 2/3]       project : > org.hibernate:hibernate-core:5.2.4.Final
[11:13:39][Step 2/3]    > Could not resolve com.fasterxml:classmate:1.3.0.
[11:13:39][Step 2/3]       > Could not get resource 'https://repo.maven.apache.org/maven2/com/fasterxml/classmate/1.3.0/classmate-1.3.0.pom'.
[11:13:39][Step 2/3]          > Could not GET 'https://repo.maven.apache.org/maven2/com/fasterxml/classmate/1.3.0/classmate-1.3.0.pom'.
[11:13:39][Step 2/3]             > Connect to repo.maven.apache.org:443 [repo.maven.apache.org/199.232.64.215] failed: connect timed out
rafiss commented 3 years ago

This could be resolved by the idea here: https://github.com/cockroachdb/cockroach/issues/59437

knz commented 3 years ago

Funnily that's what we already do for all the crdb tests in pkg/acceptance, there's an acceptance test image with a bunch of languages/drivers. Maybe we can add to that?