client.start_offline_to_online_ingestion job is failing in minimal_ride_hailing example

[Adedolapo-Akin]

Expected Behavior

{'driver_id': [100651, 561506, 304334, 997026, 420223, 546179, 308686, 888397, 731657, 684814], 'driver_statistics:avg_daily_trips': [614, 71, 801, 341, 158, 747, 509, 568, 844, 315]}

Current Behavior

{'driver_id': [507133, 891268, 940489, 750166, 830970, 815675, 85423, 56280, 230535, 846488], 'driver_statistics:acc_rate': [None, None, None, None, None, None, None, None, None, None]}

kubectl logs feast-release-feast-jupyter-86444477b7-vxptg

Exception in thread "main" redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to localhost:6379 Caused by: java.net.ConnectException: Connection refused (Connection refused)

Steps to reproduce

features = client.get_online_features( feature_refs=["driver_statistics:avg_daily_trips"], entity_rows=entities_sample).to_dict()

features

Specifications

• Version: helm repo add feast-charts https://feast-charts.storage.googleapis.com helm repo update

  app: feast-core app.kubernetes.io/managed-by: Helm chart: feast-core-0.8.2 component: core heritage: Helm release: feast-release managedFields:

• Platform:

• Subsystem:

Possible Solution

[woop]

Thanks for raising this issue @Adedolapo-Akin. I'll try to reproduce this problem today.

[woop]

Quick question @Adedolapo-Akin, did you just run kubectl logs feast-release-feast-jupyter-86444477b7-vxptg to get the logs? What command did you run to get the below logs, did you perhaps get it from the serving service?

Exception in thread "main" redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to localhost:6379
Caused by: java.net.ConnectException: Connection refused (Connection refused)

Are you sure that Redis doesn't have any security enabled?

[Adedolapo-Akin]

Quick question @Adedolapo-Akin, did you just run kubectl logs feast-release-feast-jupyter-86444477b7-vxptg to get the logs? yes, this was the command I used.

Are you sure that Redis doesn't have any security enabled? I followed the installation guide on the feast website which used helm for the installation (helm repo add feast-charts https://feast-charts.storage.googleapis.com) . I did not change any settings for Redis as this was not specified in the installation guide.

[woop]

@Adedolapo-Akin would you mind providing a bit more information. Are you running on GCP?

[Adedolapo-Akin]

Yes...I am using GKE on GCP

[woop]

@Adedolapo-Akin what isn't clear to me is where the logs came from. kubectl logs feast-release-feast-jupyter-86444477b7-vxptg shouldn't be giving you any Redis connection logs. Did the logs come from the online serving API?

[Adedolapo-Akin]

I have attached a copy of the log from feast-release-feast-jupyter-86444477b7-vxptg so that you can verify for yourself jupyter.log

[woop]

Ok, so you are using Spark from within the notebook.

Starting with config IngestionJobConfig(Online,FeatureTable(driver_trips,default,List(Field(driver_id,INT64)),List(Field(trips_today,INT32)),None,Map()),KafkaSource(10.52.10.236:9092,driver_trips,AvroFormat({"type": "record", "name": "DriverTrips", "fields": [{"name": "driver_id", "type": "long"}, {"name": "trips_today", "type": "int"}, {"name": "datetime", "type": {"type": "long", "logicalType": "timestamp-micros"}}]}),Map(),datetime,Some(datetime),Some()),2021-02-02T12:18:00.400Z,2021-02-02T12:18:00.465Z,RedisConfig(localhost,6379,false),None,None,None,0,None,true)

Can you try to set Client(redis_host="your redis address"), the address should be something like feast-redis-headless.namespace.svc

[Adedolapo-Akin]

@woop Thanks . Setting the redis host solved my problem.

[woop]

Great!

[theofpa]

I had the same problem using the helm installation method in an on-premise k8s cluster, resolved using:

client = Client(redis_host=os.environ["FEAST_RELEASE_REDIS_MASTER_SERVICE_HOST"])

[woop]

Thanks @theofpa !