search

dvankley / firefly-plaid-connector-2

Connector to pull Plaid financial data into the Firefly finance tool
GNU General Public License v3.0
94 stars 14 forks source link

FAIL_ON_UNKNOWN_PROPERTIES ... is an enum a property? #104

Closed bmschwa closed 1 month ago

bmschwa commented 2 months ago

Hey @dvankley,

Continuing the conversation we were having over in a pull request here instead, because I think its different enough to warrant a new thread.

For background, I'm now running v 1.1.2 (via docker) and I'm getting this error.

Caused by: com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type net.djvk.fireflyPlaidConnector2.api.plaid.models.Products from String "statements": not one of the values accepted for Enum class: [investments, assets, auth, income, transfer, employment, income_verification, transactions, recurring_transactions, standing_orders, credit_details, identity, signal, liabilities, deposit_switch, identity_verification, balance, payment_initiation]

The root of the error is that in the openapi auto generated models used in the project, are based on an old version of the openapi spec plaid makes available. It would be nice to easily update the models based on the latest open api spec automagically (what I think PR #96 will ultimately facilitate). Ignoring differences and handling only those fields, objects, enums, etc explicitly defined would work for me just as well. I think that was the intention of commit 3165f7f you mentioned in the other thread.

The FAIL_ON_UNKNOWN_PROPERTIES parameter has the following behavior:

Used to control whether encountering of unknown properties (one for which there is no setter; and there is no fallback "any setter" method defined using @JsonAnySetter annotation) should result in a JsonMappingException (when enabled), or just quietly ignored (when disabled)

The error thrown in my case, InvalidFormatException, is derived from JsonMappingException (through MismatchedInputException, since v 2.9.

Rather than an unknown jackson 'property', I think we need to specify behavior when encountering an unknown enum... either READ_UNKNOWN_ENUM_VALUES_AS_NULL or READ_UNKNOWN_ENUM_VALUES_USING_DEFAULT_VALUE

(Cudos to this on Stack Overflow)

dvankley commented 2 months ago

Thanks for digging into this. Looks like the right approach.

Can you post your whole stack trace? I'm wondering what the downstream effect of this field being null would be (hopefully nothing).

bmschwa commented 2 months ago

No problem. Happy to have made some small contribution to this project... its really valuable for me.

Here's the stack trace for when the bug is encountered:

java.lang.reflect.UndeclaredThrowableException: Failed to invoke event listener method HandlerMethod details: Bean [net.djvk.fireflyPlaidConnector2.FireflyPlaidConnector2Application$$SpringCGLIB$$0] Method [public void net.djvk.fireflyPlaidConnector2.FireflyPlaidConnector2Application.appReady()] Resolved arguments: 

at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:382) ~[spring-context-6.1.8.jar:6.1.8]
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:237) ~[spring-context-6.1.8.jar:6.1.8]
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:168) ~[spring-context-6.1.8.jar:6.1.8]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) ~[spring-context-6.1.8.jar:6.1.8]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.1.8.jar:6.1.8]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) ~[spring-context-6.1.8.jar:6.1.8]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451) ~[spring-context-6.1.8.jar:6.1.8]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384) ~[spring-context-6.1.8.jar:6.1.8]
at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109) ~[spring-boot-3.3.0.jar:3.3.0]
at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80) ~[spring-boot-3.3.0.jar:3.3.0]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118) ~[spring-boot-3.3.0.jar:3.3.0]
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112) ~[spring-boot-3.3.0.jar:3.3.0]
at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80) ~[spring-boot-3.3.0.jar:3.3.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:349) ~[spring-boot-3.3.0.jar:3.3.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1363) ~[spring-boot-3.3.0.jar:3.3.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1352) ~[spring-boot-3.3.0.jar:3.3.0]
at net.djvk.fireflyPlaidConnector2.FireflyPlaidConnector2ApplicationKt.main(FireflyPlaidConnector2Application.kt:31) ~[main/:na]

Caused by: io.ktor.serialization.JsonConvertException: Cannot deserialize value of type net.djvk.fireflyPlaidConnector2.api.plaid.models.Products from String "statements": not one of the values accepted for Enum class: [investments, assets, auth, income, transfer, employment, income_verification, transactions, recurring_transactions, standing_orders, credit_details, identity, signal, liabilities, deposit_switch, identity_verification, balance, payment_initiation] at [Source: REDACTED (StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION disabled); line: 41, column: 7] (through reference chain: java.util.ArrayList[5]) at _COROUTINE.BOUNDARY.(CoroutineDebugging.kt:42) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:na] at io.ktor.client.plugins.HttpCallValidator$Companion$install$2.invokeSuspend(HttpCallValidator.kt:142) ~[ktor-client-core-jvm-2.3.11.jar:2.3.11] at io.ktor.client.call.HttpClientCall.bodyNullable(HttpClientCall.kt:89) ~[ktor-client-core-jvm-2.3.11.jar:2.3.11] at io.ktor.client.call.HttpClientCall.body(HttpClientCall.kt:115) ~[ktor-client-core-jvm-2.3.11.jar:2.3.11] at net.djvk.fireflyPlaidConnector2.api.plaid.infrastructure.TypedBodyProvider.body(HttpResponse.kt:35) ~[main/:na] at net.djvk.fireflyPlaidConnector2.sync.BatchSyncRunner$run$1.invokeSuspend(BatchSyncRunner.kt:98) ~[main/:na] Caused by: io.ktor.serialization.JsonConvertException: Cannot deserialize value of type net.djvk.fireflyPlaidConnector2.api.plaid.models.Products from String "statements": not one of the values accepted for Enum class: [investments, assets, auth, income, transfer, employment, income_verification, transactions, recurring_transactions, standing_orders, credit_details, identity, signal, liabilities, deposit_switch, identity_verification, balance, payment_initiation] at [Source: REDACTED (StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION disabled); line: 41, column: 7] (through reference chain: java.util.ArrayList[5]) ... 6 common frames omitted Caused by: io.ktor.serialization.JsonConvertException: Cannot deserialize value of type net.djvk.fireflyPlaidConnector2.api.plaid.models.Products from String "statements": not one of the values accepted for Enum class: [investments, assets, auth, income, transfer, employment, income_verification, transactions, recurring_transactions, standing_orders, credit_details, identity, signal, liabilities, deposit_switch, identity_verification, balance, payment_initiation] at [Source: REDACTED (StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION disabled); line: 41, column: 7] (through reference chain: java.util.ArrayList[5]) at io.ktor.serialization.jackson.JacksonConverter.deserialize(JacksonConverter.kt:67) ~[ktor-serialization-jackson-jvm-2.1.1.jar:2.1.1] at io.ktor.serialization.jackson.JacksonConverter$deserialize$1.invokeSuspend(JacksonConverter.kt) ~[ktor-serialization-jackson-jvm-2.1.1.jar:2.1.1] at _COROUTINE.BOUNDARY.(CoroutineDebugging.kt:42) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:na] at io.ktor.client.HttpClient$4.invokeSuspend(HttpClient.kt:177) ~[ktor-client-core-jvm-2.3.11.jar:2.3.11] ... 5 common frames omitted Caused by: com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type net.djvk.fireflyPlaidConnector2.api.plaid.models.Products from String "statements": not one of the values accepted for Enum class: [investments, assets, auth, income, transfer, employment, income_verification, transactions, recurring_transactions, standing_orders, credit_details, identity, signal, liabilities, deposit_switch, identity_verification, balance, payment_initiation] at [Source: REDACTED (StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION disabled); line: 41, column: 7] (through reference chain: java.util.ArrayList[5]) at com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:67) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:1958) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.DeserializationContext.handleWeirdStringValue(DeserializationContext.java:1245) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.EnumDeserializer._deserializeAltString(EnumDeserializer.java:447) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.EnumDeserializer._fromString(EnumDeserializer.java:304) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.EnumDeserializer.deserialize(EnumDeserializer.java:273) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:361) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:246) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:30) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:545) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.MapDeserializer._deserializeUsingCreator(MapDeserializer.java:675) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:432) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:545) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.MapDeserializer._deserializeUsingCreator(MapDeserializer.java:675) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:432) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4905) ~[jackson-databind-2.17.1.jar:2.17.1] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3877) ~[jackson-databind-2.17.1.jar:2.17.1] at io.ktor.serialization.jackson.JacksonConverter$deserialize$2.invokeSuspend(JacksonConverter.kt:64) ~[ktor-serialization-jackson-jvm-2.1.1.jar:2.1.1] at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.9.22.jar:1.9.22-release-704] at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:na] at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:na] at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:na] at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:na] at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:811) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:na] at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:715) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:na] at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:702) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:na] Caused by: io.ktor.serialization.JsonConvertException: Cannot deserialize value of type net.djvk.fireflyPlaidConnector2.api.plaid.models.Products from String "statements": not one of the values accepted for Enum class: [investments, assets, auth, income, transfer, employment, income_verification, transactions, recurring_transactions, standing_orders, credit_details, identity, signal, liabilities, deposit_switch, identity_verification, balance, payment_initiation]

dvankley commented 2 months ago

Ah I see, Products is used through: https://github.com/dvankley/firefly-plaid-connector-2/blob/7cdfcab141939dcd1f3b448ebf6bb10e2542a30a/src/main/kotlin/net/djvk/fireflyPlaidConnector2/api/plaid/models/TransactionsGetResponse.kt#L50 https://github.com/dvankley/firefly-plaid-connector-2/blob/7cdfcab141939dcd1f3b448ebf6bb10e2542a30a/src/main/kotlin/net/djvk/fireflyPlaidConnector2/api/plaid/models/Item.kt#L73-L79 and the connector doesn't use anything in that response aside from https://github.com/dvankley/firefly-plaid-connector-2/blob/7cdfcab141939dcd1f3b448ebf6bb10e2542a30a/src/main/kotlin/net/djvk/fireflyPlaidConnector2/api/plaid/models/TransactionsGetResponse.kt#L43, so I think this should be safe.

dvankley commented 1 month ago

Fixed in https://github.com/dvankley/firefly-plaid-connector-2/pull/105