Unrecognized field "version"

[alberttwong]

environment: docker compose with openjdk 11, minio, xtable, spark 3.4, hive 2.3.10, hadoop 2.10.2

root@spark:/opt/LakeView# ls
delta.yaml  LakeView-release-v0.10.0-all.jar
root@spark:/opt/LakeView# java -version
openjdk version "11.0.23" 2024-04-16
OpenJDK Runtime Environment Temurin-11.0.23+9 (build 11.0.23+9)
OpenJDK 64-Bit Server VM Temurin-11.0.23+9 (build 11.0.23+9, mixed mode, sharing)
root@spark:/opt/LakeView# java -jar LakeView-release-v0.10.0-all.jar -p '/opt/LakeView/delta.yaml' 
16:43:02.592 [main] INFO  com.onehouse.Main - Starting LakeView extractor service
Exception in thread "main" java.lang.RuntimeException: Failed to load config
        at com.onehouse.config.ConfigLoader.loadConfigFromConfigFile(ConfigLoader.java:31)
        at com.onehouse.Main.loadConfig(Main.java:92)
        at com.onehouse.Main.start(Main.java:56)
        at com.onehouse.Main.main(Main.java:41)
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "version" (class com.onehouse.config.models.common.OnehouseClientConfig$OnehouseClientConfigBuilder), not marked as ignorable (7 known properties: "projectId", "apiKey", "userId", "requestId", "file", "region", "apiSecret"])
 at [Source: (StringReader); line: 1, column: 12] (through reference chain: com.onehouse.config.models.common.OnehouseClientConfig$OnehouseClientConfigBuilder["version"])
        at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1127)
        at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2023)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1700)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1678)
        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.vanillaDeserialize(BuilderBasedDeserializer.java:298)
        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:217)
        at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3629)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3597)
        at com.onehouse.config.ConfigLoader.loadConfigFromJsonNode(ConfigLoader.java:53)
        at com.onehouse.config.ConfigLoader.loadConfigFromConfigFile(ConfigLoader.java:29)
        ... 3 more

[alberttwong]

root@spark:/opt/LakeView# cat delta.yaml 
version: V1

onehouseClientConfig:
    # can be obtained from the Onehouse console
    projectId: c3eb3868-6979-41cd-9018-952d29a43337
    apiKey: XXXXXX
    apiSecret: YYYYY
    userId: x2gblCN8xNSurvCsqDaGJ84zy913 
    file: delta.yaml

fileSystemConfiguration:
    # Provide either s3Config or gcsConfig
    s3Config:
        region: us-west-2
        accessKey: admin
        accessSecret: password

metadataExtractorConfig:
    jobRunMode: ONCE
    pathExclusionPatterns: 
    parserConfig:
        - lake: <lake1>
          databases:
            - name: people
              basePaths: ["s3://warehouse/people"]
        # Add additional lakes and databases as needed

[alberttwong]

If you remove version: V1 from yaml you get a different error

root@spark:/opt/LakeView# java -jar LakeView-release-v0.10.0-all.jar -p '/opt/LakeView/delta.yaml' 
16:43:42.637 [main] INFO  com.onehouse.Main - Starting LakeView extractor service
Exception in thread "main" java.lang.RuntimeException: Failed to load config
        at com.onehouse.config.ConfigLoader.loadConfigFromConfigFile(ConfigLoader.java:31)
        at com.onehouse.Main.loadConfig(Main.java:92)
        at com.onehouse.Main.start(Main.java:56)
        at com.onehouse.Main.main(Main.java:41)
Caused by: java.lang.NullPointerException
        at com.onehouse.config.ConfigLoader.loadConfigFromJsonNode(ConfigLoader.java:44)
        at com.onehouse.config.ConfigLoader.loadConfigFromConfigFile(ConfigLoader.java:29)
        ... 3 more

[alberttwong]

answer:

had to remove file:delta.yaml field

comments from dev:

the file option is to be used only when we dont want to pass the creds directly in the config.yaml file