Unable to upgrade a connector from a private registry

philippeboyd commented 9 months ago

What method are you using to run Airbyte?

Docker

Platform Version or Helm Chart Version

0.50.34

What step the error happened?

Other

Revelant information

We used to be able to upgrade connectors from private registry as long as they are pulled on the machine prior. Now we're unable to upgrade a connector from private registry we get the error

Error upgrading <Custom Connector Nam> to version 1.2.3: Internal Server Error: Get Spec job failed.

When executing docker images my image is present on the machine.

Something either changed in io.airbyte.config.persistence.ActorDefinitionVersionResolver and/or io.airbyte.config.specs.RemoteDefinitionsProvider

Relevant log output

2023-11-29 18:49:03 ERROR i.a.c.p.ActorDefinitionVersionResolver(fetchRemoteActorDefinitionVersion):91 - Failed to fetch registry entry for us-docker.pkg.dev/<custom_connector_path>:1.2.3
2023-11-29 18:49:04 ERROR i.a.s.a.ApiHelper(execute):44 - Unexpected Exception
java.lang.IllegalStateException: Get Spec job failed.
    at com.google.common.base.Preconditions.checkState(Preconditions.java:502) ~[guava-31.1-jre.jar:?]
    at io.airbyte.commons.server.converters.SpecFetcher.getSpecFromJob(SpecFetcher.java:17) ~[io.airbyte-airbyte-commons-server-0.50.34.jar:?]
    at io.airbyte.commons.server.handlers.helpers.ActorDefinitionHandlerHelper.getSpecForImage(ActorDefinitionHandlerHelper.java:149) ~[io.airbyte-airbyte-commons-server-0.50.34.jar:?]
    at io.airbyte.commons.server.handlers.helpers.ActorDefinitionHandlerHelper.defaultDefinitionVersionFromUpdate(ActorDefinitionHandlerHelper.java:122) ~[io.airbyte-airbyte-commons-server-0.50.34.jar:?]
    at io.airbyte.commons.server.handlers.SourceDefinitionsHandler.updateSourceDefinition(SourceDefinitionsHandler.java:284) ~[io.airbyte-airbyte-commons-server-0.50.34.jar:?]
    at io.airbyte.server.apis.SourceDefinitionApiController.lambda$updateSourceDefinition$9(SourceDefinitionApiController.java:157) ~[io.airbyte-airbyte-server-0.50.34.jar:?]
    at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:27) ~[io.airbyte-airbyte-server-0.50.34.jar:?]
    at io.airbyte.server.apis.SourceDefinitionApiController.updateSourceDefinition(SourceDefinitionApiController.java:157) ~[io.airbyte-airbyte-server-0.50.34.jar:?]
    at io.airbyte.server.apis.$SourceDefinitionApiController$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-server-0.50.34.jar:?]
    at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:371) ~[micronaut-inject-3.10.1.jar:3.10.1]
    at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594) ~[micronaut-inject-3.10.1.jar:3.10.1]
    at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303) ~[micronaut-router-3.10.1.jar:3.10.1]
    at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111) ~[micronaut-router-3.10.1.jar:3.10.1]
    at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103) ~[micronaut-http-3.10.1.jar:3.10.1]
    at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659) ~[micronaut-http-server-3.10.1.jar:3.10.1]
    at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49) ~[reactor-core-3.5.5.jar:3.5.5]
    at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62) ~[reactor-core-3.5.5.jar:3.5.5]
    at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.5.5.jar:3.5.5]
    at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62) ~[micronaut-runtime-3.10.1.jar:3.10.1]
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.5.jar:3.5.5]
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.5.jar:3.5.5]
    at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53) ~[micronaut-context-3.10.1.jar:3.10.1]
    at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1589) ~[?:?]

mcornejo commented 9 months ago

I had the same issue when I was trying to build the docker image using an apple silicon (and airbyte was running in an amd64). I fixed it by building the docker image using buildx: docker buildx build --platform linux/amd64 -t my_tag .

philippeboyd commented 9 months ago

Hi @mcornejo I'm building on macOS Intel (x86_64) and Airbyte is also deployed on a x86_64

The error is due to the fact that Airbyte still http requests the image from the private registry Failed to fetch registry entry ...

oskarryn commented 7 months ago

Seems that fetching registry entry fails even when pulling not from a private registry: https://github.com/airbytehq/airbyte/issues/34198 I confirm to hit the same bug on 0.50.43 with private registry

airbytehq / airbyte