Unable to add connector to UI: Get Spec Job Failed

[aantn]

Environment

• Airbyte version: git master, commit d84df02f261d44b45c173c06dc3d3304cc8fb120
• OS Version / Instance: Kubernetes, GKE
• Deployment: Kubernetes
• Source Connector and version: Custom one I wrote
• Severity: Critical
• Step where error happened: Setup new connection

Current Behavior

Adding custom connectors to the UI is currently broken for me and at least one other person who requested support on Slack.

Examples of this occuring:

• Me: https://airbytehq.slack.com/archives/C01VDDEGL7M/p1649601487291009
• Rishika: https://airbytehq.slack.com/archives/C01MFR03D5W/p1649439800991579

Expected Behavior

When following the exact instructions in the docs, it should work.

https://docs.airbyte.com/connector-development/tutorials/cdk-tutorial-python-http/use-connector-in-airbyte

https://docs.airbyte.com/connector-development/

Logs

What logs do you need?

[aantn]

Also worth pointing out that the link in that dialog is broken and points to the docs homepage.

[yzislin]

I've had similar error on multiple occasions. It was caused by invalid JSON object of the spec file or spec file was in the wrong place. Try to use an online JSON validator to make sure your spec file is a valid JSON object.

[aantn]

The error happens so rapidly that I'm skeptical the Docker container has even been downloaded yet. Are there any logs I can fetch to verify this?

In any event, I ran the spec command and the json is valid.

{"type": "SPEC", "spec": {"documentationUrl": "https://docs.orbit.love/reference/about-the-orbit-api", "connectionSpecification": {"$schema": "http://json-schema.org/draft-07/schema#", "title": "Orbit Love Spec", "type": "object", "required": ["api_key", "workspace_slug"], "additionalProperties": false, "properties": {"api_key": {"type": "string", "description": "Orbit.love API key"}, "workspace_slug": {"type": "string", "description": "Orbit.love workspace slug"}}}}}

[yzislin]

My spec file looks a bit different. I dont have { "type": "SPEC", "spec": In the beginning. Not sure if it is needed.

For logs, I think it was the server docker container where you need to see the logs. Run docker ps and take node of the container ID. Then run docker logs ID -f. This will tail the logs. Then try to add the connector and see the logs.

[aantn]

I'm running on Kubernetes. So would that be the airbyte-server deployment?

On Thu, Apr 14, 2022 at 8:11 PM yzislin @.***> wrote:

My spec file looks a bit different. I dont have { "type": "SPEC", "spec": In the beginning. Not sure if it is needed.

For logs, I think it was the server docker container where you need to see the logs. Run docker ps and take node of the container ID. Then run docker logs ID -f. This will tail the logs. Then try to add the connector and see the logs.

— Reply to this email directly, view it on GitHub https://github.com/airbytehq/airbyte/issues/12030#issuecomment-1099424332, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADYUBY7GCZPCYTRQ24FSF3VFBGU3ANCNFSM5TOKVSHQ . You are receiving this because you authored the thread.Message ID: @.***>

[yzislin]

Yup. If I recall correctly spec errors show up there.

[aantn]

Here is the exception:

{"message":"Internal Server Error: Get Spec job failed.","exceptionClassName":"java.lang.IllegalStateException","exceptionStack":["java.lang.IllegalStateException: Get Spec job failed.","\tat com.google.common.base.Preconditions.checkState(Preconditions.java:502)","\tat io.airbyte.server.converters.SpecFetcher.getSpecFromJob(SpecFetcher.java:14)","\tat io.airbyte.server.handlers.SourceDefinitionsHandler.getSpecForImage(SourceDefinitionsHandler.java:283)","\tat io.airbyte.server.handlers.SourceDefinitionsHandler.sourceDefinitionFromCreate(SourceDefinitionsHandler.java:192)","\tat io.airbyte.server.handlers.SourceDefinitionsHandler.createPrivateSourceDefinition(SourceDefinitionsHandler.java:172)","\tat io.airbyte.server.apis.ConfigurationApi.lambda$createSourceDefinition$12(ConfigurationApi.java:354)","\tat io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:871)","\tat io.airbyte.server.apis.ConfigurationApi.createSourceDefinition(ConfigurationApi.java:354)","\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)","\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)","\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)","\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)","\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)","\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)","\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)","\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)","\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)","\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)","\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)","\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)","\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)","\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)","\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)","\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)","\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)","\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)","\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)","\tat org.eclipse.jetty.server.Server.handle(Server.java:501)","\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)","\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)","\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)","\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)","\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)","\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)","\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)","\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)","\tat java.base/java.lang.Thread.run(Thread.java:833)"],"rootCauseExceptionClassName":null,"rootCauseExceptionStack":[]}

Any idea what the problem is?

[EinavDanielDX]

I get the same results on Kubernetes deployment:

The errors shows too quickly, indicates it did not actually download the connector. And this is the log from the airbyte-server pod:

2022-04-28 12:28:25 ERROR i.a.s.e.UncaughtExceptionMapper(toResponse):22 - Uncaught exception
java.lang.IllegalStateException: Get Spec job failed.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:502) ~[guava-31.0.1-jre.jar:?]
        at io.airbyte.server.converters.SpecFetcher.getSpecFromJob(SpecFetcher.java:14) ~[io.airbyte-airbyte-server-0.36.4-alpha.jar:?]
        at io.airbyte.server.handlers.DestinationDefinitionsHandler.getSpecForImage(DestinationDefinitionsHandler.java:294) ~[io.airbyte-airbyte-server-0.36.4-alpha.jar:?]
        at io.airbyte.server.handlers.DestinationDefinitionsHandler.destinationDefinitionFromCreate(DestinationDefinitionsHandler.java:199) ~[io.airbyte-airbyte-server-0.36.4-alpha.jar:?]
        at io.airbyte.server.handlers.DestinationDefinitionsHandler.createPrivateDestinationDefinition(DestinationDefinitionsHandler.java:179) ~[io.airbyte-airbyte-server-0.36.4-alpha.jar:?]
        at io.airbyte.server.apis.ConfigurationApi.lambda$createDestinationDefinition$43(ConfigurationApi.java:550) ~[io.airbyte-airbyte-server-0.36.4-alpha.jar:?]
        at io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:871) ~[io.airbyte-airbyte-server-0.36.4-alpha.jar:?]
        at io.airbyte.server.apis.ConfigurationApi.createDestinationDefinition(ConfigurationApi.java:550) ~[io.airbyte-airbyte-server-0.36.4-alpha.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.31.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.31.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.31.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.31.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[jersey-server-2.31.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[jersey-server-2.31.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[jersey-server-2.31.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[jersey-server-2.31.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[jersey-server-2.31.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.31.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.31.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.31.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.31.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.31.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.31.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) ~[jersey-server-2.31.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[jersey-server-2.31.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.31.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.31.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[jersey-container-servlet-core-2.31.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[jersey-container-servlet-core-2.31.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.31.jar:?]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.Server.handle(Server.java:501) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
        at java.lang.Thread.run(Thread.java:833) [?:?]
2022-04-28 12:28:25 ERROR i.a.s.RequestLogger(filter):110 - REQ 10.2.1.225 POST 500 /api/v1/destination_definitions/create - {"name":"custom_clickhouse_connector","documentationUrl":"https://docs.mydomain.com","dockerImageTag":"1.2.3","dockerRepository":"public.ecr.aws/my_private_repo/custom_clickhouse_dst"}

running the spec command locally finishes successfully with valid JSON object docker run --rm airbyte/destination-clickhouse:dev spec

Any ideas?

[aantn]

Try using an image that doesn't contain the character _

[EinavDanielDX]

I get the same error with airbyte/destination-clickhouse:0.1.4

[aantn]

Don't know then. I'm just a fellow user and that is what fixed it for me.

[EinavDanielDX]

what airbyte version and chart version are you using?

[EinavDanielDX]

@aantn Trying another public repo did the trick, had to make sure to use format such as "repo/image:tag" and not "repo/subpath/image:tag"

[cgardens]

awesome!

[stepcha]

@cgardens same error here, how you solved it?

[mamontesp]

Same error here 🤚🏽

[cgardens]

What version of Airbyte are you running and what error are you seeing? Any logs would be helpful as well.

This was resolved a little while ago, so if you are seeing a new issue on a new version of airbyte, it is likely a regression.

[mamontesp]

Thank you for your response. I was using the wrong label in the docker image when adding the new connector. After the correction, it worked as expected 👍🏽

[jparkrr]

Just in case it's helpful for someone else in the future, I was seeing this error with a custom connector where I was trying to use the latest tag. I switched to using a real version number and it's working now.

[anchitagarwal]

I'm facing this issue again in Airbyte v0.50.7. There's no _ in the image name and it was working as expected before I upgraded my Airbyte instance.

[Nozima29]

same error! I tried all the solutions above. my airbyte version is 0.50.33 how to solve?

[Nozima29]

In my case Docker platform was the issue, as DOCKER_DEFAULT_PLATFORMS are different in Mac and Win.

[sigeek]

Hi, I'm facing this issue with airbyte version 0.59.0 locally spec, check, read and discover are running fine.

[mschendelmann]

I was facing the same issue and none of the solutions above worked. Turns out the version tag in the build command docker build . --build-arg version=0.0.1 -t sometag (from the faros community edition documentation) was causing the problems. It was not related to the docker image tag, but setting it to a "higher" version made airbyte accept the destination connector.