error in feast_batch-serving and feast_online-serving container when built using docker-compose

[napitupulus]

I am installing feast using docker-compose and following the step by step guide (batch and online serving), but feast_batch-serving and feast_online-serving container always restarting all the time. I have checked the logs of both containers, and the last logs are here:

Caused by: java.lang.RuntimeException: Unable to read store config at /etc/feast/store.yml at feast.serving.service.CachedSpecService.readConfig(CachedSpecService.java:181) at feast.serving.service.CachedSpecService.(CachedSpecService.java:74) at feast.serving.configuration.SpecServiceConfig.specService(SpecServiceConfig.java:65) at feast.serving.configuration.SpecServiceConfig$$EnhancerBySpringCGLIB$$ac250397.CGLIB$specService$1() at feast.serving.configuration.SpecServiceConfig$$EnhancerBySpringCGLIB$$ac250397$$FastClassBySpringCGLIB$$5efcdb08.invoke() at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:365) at feast.serving.configuration.SpecServiceConfig$$EnhancerBySpringCGLIB$$ac250397.specService() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ... 54 more

Caused by: com.google.protobuf.InvalidProtocolBufferException: Cannot find field: project in message feast.core.Store.Subscription at com.google.protobuf.util.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1500) at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1460) at com.google.protobuf.util.JsonFormat$ParserImpl.parseFieldValue(JsonFormat.java:1993) at com.google.protobuf.util.JsonFormat$ParserImpl.mergeRepeatedField(JsonFormat.java:1728) at com.google.protobuf.util.JsonFormat$ParserImpl.mergeField(JsonFormat.java:1661) at com.google.protobuf.util.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1502) at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1460) at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1342) at com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:477) at feast.serving.util.mappers.YamlToProtoMapper.yamlToStoreProto(YamlToProtoMapper.java:34) at feast.serving.service.CachedSpecService.readConfig(CachedSpecService.java:178) ... 66 more

The error said that problem is when reading store config at /etc/feast/store.yml. I hypothesize that error is in yaml format of redis-store.yml file, i have modified that but i doesn't work. Please help me :)

[khorshuheng]

@frankynapit We will look into this. For the time being, can you use version 0.3.4 instead of latest? (modifiable from .env file)

[khorshuheng]

@frankynapit Found the issue. In redis-store.yml and bq-store.yml, could you change the subscription field to:

subscriptions: - name: "*"    version: ">0"

Let us know if you still face difficulty. This should work with the latest image version as well.

[napitupulus]

hi @khorshuheng, i have tried replacing the subscription as this. but when i run this sample code i got error in applying feature set, it said that:

ValueError: No project found in feature set customer_transactions. Please set the project within the feature set or within your Feast Client.

i have set the project id in bigquery_config bq-store.yml. And when i tried to set project name in feature set, i got this error:

Traceback (most recent call last): File "main.py", line 43, in client.apply(customer_fs) File "/Users/mapan/Documents/franky/gojek_feast/feast/sdk/python/feast/client.py", line 311, in apply self._apply_feature_set(feature_set) File "/Users/mapan/Documents/franky/gojek_feast/feast/sdk/python/feast/client.py", line 345, in _apply_feature_set raise grpc.RpcError(e.details()) grpc.RpcError

The container is stopped restarting, but when i looking for logging of feast_core_1 container i got this:

08:09:36 [grpc-default-executor-9] ERROR io.grpc.internal.SerializingExecutor - Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable@423b774d io.grpc.StatusRuntimeException: INTERNAL: Invalid protobuf byte sequence at io.grpc.Status.asRuntimeException(Status.java:521) at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:218) at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:118) at io.grpc.MethodDescriptor.parseRequest(MethodDescriptor.java:289) at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263) at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:686) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8. at com.google.protobuf.InvalidProtocolBufferException.invalidUtf8(InvalidProtocolBufferException.java:141) at com.google.protobuf.Utf8$DecodeUtil.handleTwoBytes(Utf8.java:1909) at com.google.protobuf.Utf8$DecodeUtil.access$700(Utf8.java:1883) at com.google.protobuf.Utf8$UnsafeProcessor.decodeUtf8(Utf8.java:1411) at com.google.protobuf.Utf8.decodeUtf8(Utf8.java:340) at com.google.protobuf.CodedInputStream$ArrayDecoder.readStringRequireUtf8(CodedInputStream.java:809) at feast.core.FeatureSetProto$FeatureSetSpec.(FeatureSetProto.java:267) at feast.core.FeatureSetProto$FeatureSetSpec.(FeatureSetProto.java:220) at feast.core.FeatureSetProto$FeatureSetSpec$1.parsePartialFrom(FeatureSetProto.java:2297) at feast.core.FeatureSetProto$FeatureSetSpec$1.parsePartialFrom(FeatureSetProto.java:2291) at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:888) at feast.core.CoreServiceProto$ApplyFeatureSetRequest.(CoreServiceProto.java:5693) at feast.core.CoreServiceProto$ApplyFeatureSetRequest.(CoreServiceProto.java:5646) at feast.core.CoreServiceProto$ApplyFeatureSetRequest$1.parsePartialFrom(CoreServiceProto.java:6270) at feast.core.CoreServiceProto$ApplyFeatureSetRequest$1.parsePartialFrom(CoreServiceProto.java:6264) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:86) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48) at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parseFrom(ProtoLiteUtils.java:223) at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:215) ... 9 common frames omitted

I even used the version 0.3.4 of feast, but it gives the same results. hope you can help me for this :)

[khorshuheng]

Project name is still in active development, and hence, it's only on the master branch but not on any of the stable released branch. This is why the sample code doesn't work when you use the official release images, as the release images is not based on the latest master branch code, but the latest released version (0.3.x in this case). On the other hand, the sample code is meant for master branch only. We planned to release version 0.4.0 soon, which will include project name.

If you don't care about project name functionality, then instead of the sample code, you can use the Jupyter notebook included in the docker compose file to explore Feast's functionality instead: http://localhost:8888/notebooks/feast-notebooks

Alternatively, you can also build the feast images based on the master branch, and use the local image instead of latest official release image. This might be time consuming if you are not familiar with Feast, so we will update the documentation in near future to include steps to build images based on the master branch.

[napitupulus]

hi @khorshuheng, every time i retrieve batch features using feast it always automatically add some new tables in bigquery. I have some concerned with this in terms of the cost. Is it possible to not creating new table every time we want to retrieve batch features? Thankyou

[khorshuheng]

The temporary tables should be expired and removed automatically after one day. Currently the expiry period is fixed and not configurable, though we can enable that. Are the tables not deleted after a day? If so, could you please raise this in a separate issue so that we can trace this better?

[napitupulus]

ok, thanks @khorshuheng. I will take a look in one day and then raise new issue if it’ll getting a problem. By the way, do you have documentation of feast’s functionality expected behaviour? so I would know whether i explore it on the right track haha. If you have, could you share it please? thanks

[khorshuheng]

Which version of Feast? The latest master branch (version 0.4), or version 0.3?

We are aware that documentation is lacking right now. We plan to have a proper Jupyter notebook that capture all the expected behaviour of Feast. It will be helpful if you can comment here, https://github.com/gojek/feast/issues/415, on the areas which you would like us to focus more on.

[woop]

ok, thanks @khorshuheng. I will take a look in one day and then raise new issue if it’ll getting a problem. By the way, do you have documentation of feast’s functionality expected behaviour? so I would know whether i explore it on the right track haha. If you have, could you share it please? thanks

Hi @napitupulus. Thank you again for the interest in the project. Just wanted to add that these temporary tables are being used to store the intermediate state of tables during joins. If we don't use temporary tables then BigQuery quickly runs out of memory when joining either large data sets or many small data sets.