search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.45k stars 3.98k forks source link

Bootloader crashes when using image tags that don't follow semantic versioning #21006

Open SebastienN15 opened 1 year ago

SebastienN15 commented 1 year ago

Environment

Current Behavior

I’m trying to update Airbyte from 0.40.14 to 0.40.26, and the booloader fails because it tries to upgrade my custom connectors, which don’t follow semantic versioning for tag names. I know this issue already happened before, and was solved with this PR.

Expected Behavior

The bootloader shouldn't check for most recent versions of custom connector images, as explained in this PR I believe (but I could be wrong) that the PR that introduced this issue is https://github.com/airbytehq/airbyte/pull/18749

Logs

Bootloader logs ``` 2022-12-30 11:44:25,488 main INFO Loading mask data from '/seed/specs_secrets_mask.yaml 2022-12-30 11:44:25 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable CONFIG_DATABASE_USER: 'main' 2022-12-30 11:44:25 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: '*****' 2022-12-30 11:44:25 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable CONFIG_DATABASE_URL: 'jdbc:postgresql://airbyte-test-postgres-db.cwjsqgvfpeim.us-east-1.rds.amazonaws.com:5433/airbyte_test' 2022-12-30 11:44:25 INFO c.z.h.HikariDataSource():80 - HikariPool-1 - Starting... 2022-12-30 11:44:26 INFO c.z.h.HikariDataSource():82 - HikariPool-1 - Start completed. 2022-12-30 11:44:26 INFO c.z.h.HikariDataSource():80 - HikariPool-2 - Starting... 2022-12-30 11:44:26 INFO c.z.h.HikariDataSource():82 - HikariPool-2 - Start completed. 2022-12-30 11:44:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE' 2022-12-30 11:44:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE' 2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(load):194 - Initializing databases... 2022-12-30 11:44:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable CONFIGS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000' 2022-12-30 11:44:27 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available... 2022-12-30 11:44:27 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte configs database connection... 2022-12-30 11:44:27 INFO i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available. 2022-12-30 11:44:27 INFO i.a.d.i.DatabaseInitializer(initializeSchema):90 - The airbyte configs database is initialized 2022-12-30 11:44:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable JOBS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000' 2022-12-30 11:44:27 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available... 2022-12-30 11:44:27 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte jobs database connection... 2022-12-30 11:44:27 INFO i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available. 2022-12-30 11:44:27 INFO i.a.d.i.DatabaseInitializer(initializeSchema):90 - The airbyte jobs database is initialized 2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(load):200 - Databases initialized. 2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(load):202 - Setting up config database and default workspace... 2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(assertNonBreakingMigration):340 - Checking illegal upgrade.. 2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(isLegalUpgrade):384 - Current Airbyte version: AirbyteVersion{version='0.40.26', major='0', minor='40', patch='26'} 2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(isLegalUpgrade):385 - Future Airbyte version: AirbyteVersion{version='0.40.26', major='0', minor='40', patch='26'} 2022-12-30 11:44:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable AIRBYTE_PROTOCOL_VERSION_MIN: '0.0.0' 2022-12-30 11:44:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable AIRBYTE_PROTOCOL_VERSION_MAX: '0.3.0' 2022-12-30 11:44:27 INFO i.a.b.ProtocolVersionChecker(validate):77 - Using AirbyteProtocolVersion range [0.0.0:0.3.0] 2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(trackProtocolVersion):374 - AirbyteProtocol version support range [0.0.0:0.3.0] 2022-12-30 11:44:27 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate 2022-12-30 11:44:27 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://airbyte-test-postgres-db.cwjsqgvfpeim.us-east-1.rds.amazonaws.com:5433/airbyte_test (PostgreSQL 12.11) 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Schema history table "public"."airbyte_configs_migrations" already initialized with (0.29.0.001,Baseline from file-based migration v1). Skipping. 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://airbyte-test-postgres-db.cwjsqgvfpeim.us-east-1.rds.amazonaws.com:5433/airbyte_test (PostgreSQL 12.11) 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Schema history table "public"."airbyte_jobs_migrations" already initialized with (0.29.0.001,Baseline from file-based migration v1). Skipping. 2022-12-30 11:44:28 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable RUN_DATABASE_MIGRATION_ON_STARTUP: 'true' 2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(runFlywayMigration):396 - Migrating configs database 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Successfully validated 33 migrations (execution time 00:00.019s) 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Current version of schema "public": 0.40.23.002 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Schema "public" is up to date. No migration necessary. 2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(runFlywayMigration):398 - Migrating jobs database 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Successfully validated 11 migrations (execution time 00:00.007s) 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Current version of schema "public": 0.40.18.002 2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Schema "public" is up to date. No migration necessary. 2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):216 - Ran Flyway migrations. 2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(createWorkspaceIfNoneExists):317 - workspace already exists for the deployment. 2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):219 - Default workspace created. 2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(createDeploymentIfNoneExists):307 - running deployment: 5c1e816f-dada-4e28-bcf2-fd430ae877c1 2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):222 - Default deployment created. 2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):225 - Set version to AirbyteVersion{version='0.40.26', major='0', minor='40', patch='26'} 2022-12-30 11:44:28 INFO i.a.c.p.ActorDefinitionMigrator(updateConfigsFromSeed):72 - Updating connector definitions from the seed if necessary... 2022-12-30 11:44:28 INFO i.a.c.p.ActorDefinitionMigrator(updateConfigsFromSeed):75 - Connectors in use: [airbyte/destination-snowflake, 464622532012.dkr.ecr.us-east-1.amazonaws.com/dd-airbyte] Exception in thread "main" java.lang.IllegalArgumentException: Invalid version string: source-s3-0.1.18 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) at io.airbyte.commons.version.Version.(Version.java:37) at io.airbyte.commons.version.AirbyteVersion.(AirbyteVersion.java:15) at io.airbyte.config.persistence.ActorDefinitionMigrator.lambda$getConnectorRepositoryToInfoMap$3(ActorDefinitionMigrator.java:144) at java.base/java.util.HashMap.merge(HashMap.java:1391) at java.base/java.util.stream.Collectors.lambda$toMap$68(Collectors.java:1673) at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.base/java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:1003) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1921) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) at io.airbyte.config.persistence.ActorDefinitionMigrator.getConnectorRepositoryToInfoMap(ActorDefinitionMigrator.java:110) at io.airbyte.config.persistence.ActorDefinitionMigrator.updateConfigsFromSeed(ActorDefinitionMigrator.java:77) at io.airbyte.config.persistence.ActorDefinitionMigrator.lambda$migrate$0(ActorDefinitionMigrator.java:59) at io.airbyte.db.Database.lambda$transaction$0(Database.java:27) at org.jooq.impl.DefaultDSLContext.lambda$transactionResult0$0(DefaultDSLContext.java:564) at org.jooq.impl.Tools$12$1.block(Tools.java:4926) at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3744) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3689) at org.jooq.impl.Tools$12.get(Tools.java:4923) at org.jooq.impl.DefaultDSLContext.transactionResult0(DefaultDSLContext.java:616) at org.jooq.impl.DefaultDSLContext.transactionResult(DefaultDSLContext.java:533) at io.airbyte.db.Database.transaction(Database.java:27) at io.airbyte.db.ExceptionWrappingDatabase.transaction(ExceptionWrappingDatabase.java:31) at io.airbyte.config.persistence.ActorDefinitionMigrator.migrate(ActorDefinitionMigrator.java:57) at io.airbyte.config.persistence.ConfigRepository.seedActorDefinitions(ConfigRepository.java:1190) at io.airbyte.config.init.ApplyDefinitionsHelper.apply(ApplyDefinitionsHelper.java:71) at io.airbyte.config.init.ApplyDefinitionsHelper.apply(ApplyDefinitionsHelper.java:47) at io.airbyte.bootloader.BootloaderApp.lambda$new$0(BootloaderApp.java:178) at io.airbyte.bootloader.BootloaderApp.load(BootloaderApp.java:227) at io.airbyte.bootloader.BootloaderApp.main(BootloaderApp.java:300) 2022-12-30 11:44:28 INFO c.z.h.HikariDataSource(close):350 - HikariPool-1 - Shutdown initiated... ```

Steps to Reproduce

  1. Add a source definition with an image tag that don't follow semantic versioning (ie foo-1.2.3)
  2. Re-deploy / re-start your airbyte instance, this should trigger the bootloader
  3. The bootloader then fails with the error mentioned above Note : the behavior is the same in Docker and in k8s deployment

Are you willing to submit a PR?

Yes, but I would definitely need some guidance on what should be changed.

rach-r commented 1 year ago

I can confirm this is still happening in Airbyte latest version as of today (0.42.1). It indeed looks like a regression

marcosmarxm commented 1 year ago

@malikdiarra can you give your thoughts here?

SebastienN15 commented 1 year ago

Hey @rach-r @marcosmarxm, is there a planned ETA for this?

rach-r commented 1 year ago

@SebastienN15 I'm not from the Airbyte team so I can't give you an ETA, but I've encountered the same issue and here's a workaround until this gets fixed

SebastienN15 commented 1 year ago

Oh that's awesome, thank you! 🙇

marcosmarxm commented 1 year ago

@rach-r @SebastienN15 did you try to disable the connectors.versionOverridesEnabled flags.yaml https://github.com/airbytehq/airbyte-platform/blob/4f5ac8c328bc02e3238ed18c82599c029611c095/flags.yml#L8-L9

rach-r commented 1 year ago

@marcosmarxm tried it on my side with this line changed to false

- name: connectors.versionOverridesEnabled
  serve: false

but still not working unfortunately (exact same error)